Difference between revisions of "URLTeam"
m (add tracker https link)
m (→301Work cooperation: Fixed Typo)
|Line 18:||Line 18:|
== 301Work cooperation ==
== 301Work cooperation ==
The fine folks at archive.org have
The fine folks at archive.org have us with upload permissions to the 301Works archive: [http://www.archive.org/details/301utm http://www.archive.org/details/301utm]. They unfortunately do not want to make them downloadable, but the same data is in our torrents too, just in a different format (we use pipe-delimited, xz-compressed files while 301works uses comma-delimited uncompressed files).
== Tools ==
== Tools ==
Revision as of 19:33, 9 February 2015
url shortening was a fucking awful idea
|Archiving status||In progress...|
|Project source||Old: urlteam-stuff tinyback tinyarchive|
|Project tracker||http://tracker.archiveteam.org:1337/ (HTTPS)|
|IRC channel||(on EFnet)|
TinyURL, bit.ly and other similar services allow long URLs to be converted to smaller ones on their specific service; the small URL is visited by a consumer and their web browser is redirected to the long URL.
Such services are a ticking timebomb. If they go away, get hacked or sell out millions of links will be lost (see Wikipedia: Link Rot). Archive.org/301Works is acting as an escrow for URL shortener databases, but they rely on URL shorteners to actually give them their databases. Even 301Works founding member bit.ly does not actually share their databases and most other big shorteners don't share theirs either.
The fine folks at archive.org have provided us with upload permissions to the 301Works archive: http://www.archive.org/details/301utm. They unfortunately do not want to make them downloadable, but the same data is in our torrents too, just in a different format (we use pipe-delimited, xz-compressed files while 301works uses comma-delimited uncompressed files).
- fetcher.pl: Perl-based scraper by User:Chronomex
- TinyBack: Python 2.x-based, distributed scraper (formerly used)
- Terror of Tiny Town: currently used by ArchiveTeam
Terror of Tiny Town
The easiest way to help with scraping is to run the Warrior and select the URLTeam 2 project. You can also run ToTT outside the warrior; to do so, follow the instructions at https://github.com/ArchiveTeam/terroroftinytown-client-grab.
The new table includes shorteners we have already started to scrape.
|Name||Est. number of shorturls||Scraping done by||Status||Comments|
done: sequential to zzzzzz
|current: non-sequential, 7 characters|
done: non-sequential 6 characters
|current: non-sequential, 6 characters|
|Goo.gl||?||User:Scumola||started (2011-03-04)||goo.gl throttles pulls|
|is.gd||934,134,706 (2013-05-20)||Warrior||In progress
done: sequential up to ZZZZZ
|new shorturls: non-sequential, 6 characters|
|ff.im||?||User:Chronomex||only used by FriendFeed, no interface to shorten new URLs|
|4url.cc||1279 (2009-08-14)||User:Chronomex||dead (2011-02-15)|
|litturl.com||17096 (2010-04-15)||User:Chronomex||dead (2010-11-18)|
|xs.md||3084 (2009-08-15)||User:Chronomex||done||dead (2010-11-18)|
|url.0daymeme.com||14867 (2009-08-14)||User:Chronomex||done||dead (2010-11-18)|
|tr.im (old)||1990425||-||got what we could||dead (2011-12-31)|
|tr.im (new)||?||Warrior||done: sequential to 42pzz||dead (2014-JUL-17)|
|visibli (hex)||16777216||User:Chfoo||Done. 15104865 301MB||Using links.sharedby.co/links/ as URL prefix. These results are already in the torrent.|
scraping: sequential, done up to dzzzz
|new shorturls: sequential|
scraping: sequential, done up to lyZZZ
|new shorturls: sequential|
scraping: sequential, done up to 271~~~~
|new shorturls: sequential, starting from 20wa5rt|
|vbly.us (formerly vb.ly)||?||Warrior||TODO
scraping: sequential, done up to 2hba
|new shorturls: sequential|
scraping: sequential, done up to 4fv3
|new shorturls: sequential|
|zapd.co||326592||User:Chfoo||Done. 144093 1.7M||xxxx.zapd.co. Uploaded to IA|
|Bre.ad||120932351||User:Chfoo||Incomplete (59771889 examined). 54506 1.2MB||de.ad (2013-11-18). Uploaded to IA
Got what I can without overloading their EC2 instance.
|2jump.info||Warrior||Done up to ?||Banned|
|burl.se||Warrior||Done up to ?|
|kcy.me||Warrior||Done up to ?|
|nig.gr||Warrior||Done up to ?|
|pub.vitrue.com||Warrior||Done up to ?|
|shrt.st||Warrior||Done up to ?|
|srtn.us||Warrior||Done up to ?|
|tighturl.com||Warrior||Done up to ?|
|tr.im (gravity 4)||Warrior||In progress|
|u4.hu||Warrior||Done up to ?|
|waa.ai||Warrior||Done up to ?|
|x.co||Warrior||Done up to ?|
|yahoo.it||Warrior||In progress (5, 6, 8 len)|
|yoolink.to||Warrior||Done up to ?|
|Name||Number of shorturls||Scraping done by||Status||Comments|
For the latest TinyTown updates, please see chfoo's spreadsheet.
Last verified 2014-12-07. Original list last updated 2009-08-14.
- adf.ly - Ex: http://adf.ly/bnpYL
- ar.gy - Argyle Social, main page 404s, existing urls still work
- ask.fm - Ex: ask.fm/a/40k05kgp
- awe.sm - main page redirects, doesn't allow for new urls to be publicly shortened, existing urls still work
- budurl.com - Appears non-incremental
- buff.ly - Buffer App
- burl.se - Incremental. Ex: http://burl.se/428
- cf.ly (CashFly.com)
- cli.gs - Appears non-incremental
- cl.ly - CloudApp
- cmt.com - Country Music Television
- cur.lv (CoinURL.com)
- decenturl.com - Not at all easy to scrape.
- del.ly - sprinklr
- df4.us - daringfireball.net
- dld.bz - "private URL shortening service"
- doiop.com - Appears non-incremental
- easyurl.net - Appears non-incremental. Ex: http://easyurl.net/afd2f
- fav.me - Used by DeviantArt. Ex: http://fav.me/d31sfml
- flip.it - Flipboard
- flpbd.it - Flipboard
- fnd.us (See offical shorteners)
- htl.li - Alias of ow.ly
- ht.ly - Alias of ow.ly
- jdem.cz - Incremental with random (?) last digit - Ex: http://jdem.cz/bw388
- kics.it – Restricted access to shourturl creation
- ln.is - linkis.com
- metamark.net / xrl.us - no longer allowing new urls to be shortened, existing urls still work (Ex. http://xrl.us/bfabog)
- mgnet.me - for torrent magnet URIs.
- msft.it - Sprinklr
- my.dot.tk/tweak - Appears non-incremental
- nohref.hu – Allows custom shorturl
- notlong.com - Appears to be alpha-only - Ex: http://yeitoo.notlong.com/
- nutshellurl.com - Appears incremental. 301s to a redirector script, which then 301s you to the destination.
- pear.ly - Used by pearltrees.com. Ex: http://pear.ly/6J1H
- ph.ly - Related to the pond called Philadelphia, where links are born and raised, doesn't allow for new urls to be publicly shortened, existing urls still work
- pnut.co - see nutshellurl.com Ex: http://pnut.co/3a
- prsm.tc - getprismatic.com
- sdai.ly – Allows custom shorturl
- sharedby.co - See vsb.li. Double redirects via USERNAME.sharedby.co/share/XXXXXX
- shar.es (See offical shorteners). Still resolves URLs, but the site is 404
- shorl.com - Doesn't appear guessable - Ex: http://shorl.com/tisikestibahu
- shorturl.com - Probably sequential/loweralpha - Ex: http://alturl.com/wqok
- shrd.by - see sharedby.co
- shrinkurl.us - Still resolves, but does not allow creating new URLs ("The URL you entered was not valid or did not exist.")
- shrt.st - doesn't allow new urls to be shortened, existing urls still work. Appears incremental - Ex: http://shrt.st/vpz
- smarturl.eu / joturl.com - Doesn't appear guessable, HTML redirect.
- smarturl.it - smartURL
- snipr.com / snipurl.com / snurl.com - Appears incremental - Ex: http://snipr.com/27nvst http://snipr.com/27nvtt. snipr.com and snipurl.com work but appear infected with malware.
- soa.li - Gigya inc.
- soc.li - Gigya inc.
- spne.ws - Silicon Prairie News
- spnsr.tw - sponsoredtweets.com
- srtn.us - still resolves URLs, but site just shows blank page
- surl.co.uk - Many shortening options.
- t7.hu – Doesn't make any more shorturls
- techme.me - Techmeme
- tighturl.com - Appears incremental: http://tighturl.com/30xu http://tighturl.com/30xv
- tinyarrows.com / ta.gd / ri.ms / ➡.ws / ➨.ws / ➯.ws / ➔.ws / ➞.ws / ➽.ws / ➹.ws / ✩.ws / ✿.ws / ❥.ws / ›.ws / ⌘.ws / ‽.ws / ☁.ws - Appears non-incremental: uses user-defined words for URLs (e.g. http://➡.ws/URLTEAM)
- tiny.cc - Appears non-incremental
- tm.to - Twtmore
- totesz.hu/x – Allows custom shorturl
- tr.im - Appears incremental - Ex: http://tr.im/44tn2 http://tr.im/44tn4
- usite.hu/link.php – Numeric incremental, public database
- vitrue.com - Now part of Oracle
- x.co - Appears incremental - Ex: http://x.co/1IxUV http://x.co/1IxUW; but custom ones also exist (up to 10 characters)
- xrl.us - see metamark.net
- y2u.be - meant for YouTube videos
- yatuc.com - Not accepting new urls.
- yoolink.to - Yoolink
- abcn.ws - ABC News
- bull.hn - Bullhorn Reach (format: bull.hn/l/19JQE/)
- CokeURL.com - Coca-Cola
- db.tt - Dropbox
- di.sn - Disney
- fb.me - Facebook
- flic.kr - Flickr
- fnd.us - Fundrazr.com
- fxn.ws - Fox News
- g.co - Google (used for Google products and services)
- getpocket.com/s/ - Pocket
- goo.gl - Google
- go.usa.gov - USA Government (and since they control the Internets, it doesn't get much more official than this)
- git.io - GitHub only URLs
- gty.im - Getty Images (format: gty.im/488068439; links by editorial number)
- gu.com - The Guardian (weird format - https://gu.com/p/3f7ca )
- hub.me - HubPages
- ift.tt - IFTTT
- igg.me - Indiegogo
- lnkd.in - LinkedIn
- mfi.re - MediaFire
- msft.it - Microsoft
- mysp.ac - Myspace
- nydn.us - New York Daily News
- off365.ms - Office 365
- pocket.co - Pocket
- post.ly - Posterous
- redd.it - Reddit
- reut.rs - Reuters
- rsg.ms - Rockstar Games
- shar.es - ShareThis - 404 on homepage, otherwise ok
- skfb.ly - Sketchfab
- spoti.fi - Spotify
- stanford.io - Stanford University
- su.pr - StumbleUpon
- sx3.se - swedishstartupspace.se
- t.co - Twitter
- ti.me - Time Magazine
- tmblr.co - Tumblr
- uoft.me - University of Toronto
- upl.nu - Ung Pirat (Youth Pirate Party, Sweden)
- vstphl.ly - Visit Philly
- wapo.st - Washington Post
- wh.gov - White House (format: wh.gov/i3lXR)
- wp.me - Wordpress.com
- y.ahoo.it - Yahoo (shutting down)
- youtu.be - YouTube
A bit.ly alias works just like a bit.ly URL. The shortcode is the same, it sets the same bit.ly cookie, and DNS resolving the address shows the IP addresses are the same as bit.ly. The homepage may be different however.
- 1.usa.gov - USA Government
- 4sq.com - Foursquare
- aje.me - Aljazeera
- amzn.to - Amazon
- atfp.co - Foreign Policy
- bbc.in - BBC
- binged.it - Bing (bonus points for being longer than bing.com)
- bnkrpt.am - Bankrupting America
- bzfd.it - Buzzfeed
- cb.com - Career Builder
- chzb.gr - Cheezeburger
- cmplx.it - Complex Magazine
- cnet.co - CNET
- cnnmon.ie - CNN Money
- conta.cc - Constant Contact Inc.
- cpurl.net - Current Photographer.com
- curbed.cc - Curbed.com
- dennysd.in - Denny's Restaurants
- dtoid.it - Destructoid
- econ.st - The Economist
- emarketee.rs - Emarketeers
- engri.sh - Engrish.com
- eonli.ne - E! Online
- es.pn - ESPN
- fakes.pn - The Fake ESPN (at lockerdome.com)
- fanpa.ge - Fanpage.it
- feedly.com/k/ - redirect, see below for their own
- gaw.kr - Gawker
- geekiss.im - Geekismo
- grd.to - The Grid TO
- grn.bz - GreenBiz
- gtg.lu - GetGlue
- hoblu.es - House of Blues
- hub.am - HubSpot
- huff.to - Huffington Post
- ift.tt - IFTTT
- j.mp - bit.ly
- jrnl.to - thejournal.ie
- kck.st - Kickstarter
- marsdd.it - MaRS Discovery District
- mbist.ro - MediaBistro
- mojo.ly - Mother Jones
- muo.fm - MakeUseOf
- mwne.ws - MarketWired News
- nie.mn - Neiman Journalism Lab
- nokia.ly - Nokia
- nyti.ms - New York Times
- onforb.es - Forbes
- onion.com - The Onion
- pops.ci - Popular Science
- popu.pe - Pop-Up Pantry
- propub.ca - ProPublica
- read.bi - Business Insider
- rseo.co - realseo
- s831.us - Studio831 - whatever that is
- sbn.to - sbnation
- skygrid.me - SkyGrid
- slackers.co - slackers.com
- squid.us - Laughing Squid
- s.shr.lc - shareaholic - Naive, redirects any shortcode to bit.ly
- stjo.es - St. Joseph Media
- tcrn.ch - Techcrunch
- theatln.tc - The Atlantic
- tnw.co - The Next Web
- tom.hn - Tom Hillenbrand
- toms.sh - TOMS Shoes
- tvt.ag - tvtag.com
- txpr.de - TexasStore
- unr.ly - Unruly media
- usat.ly - USA Today Newspaper
- vrge.co - The Verge
- yhoo.it - Yahoo! (not to be confused with yahoo.it, their non-bitly public url shortener)
- zite.to - Zite
Dead or Broken
- 1link.in - Website dead
- 6url.com - HTML redirect, Error 500
- ad.vu - mirror of adjix.com, application not found
- biglnk.com - dead, replaced with unrelated blog
- bwtm.co - DNS fails to resolve.
- calyp.co - Server error. 403 - Forbidden: Access is denied.
- canurl.com - Website dead
- chod.sk - Appears non-incremental, not resolving
- come.to - Related to various .to shorteners. Started in 1997, killed in 2013 after parent company died.
- da.co - Parked.
- digg.com - discontinued - 
- dwarfurl.com - Website dead/Numeric, appears incremental: http://dwarfurl.com/08041
- easy.tc - DNS not resolving.
- easyuri.com - Website dead/Appears hex incremental with last digit random/checksum: http://easyuri.com/1339f , http://easyuri.com/133a3
- eqent.me - Improper redirect to bitly.
- feedzil.la - Domain parked.
- go2cut.com - Website dead
- gob.li - Golbin Ridge Limited. Timed out
- gonext.org - not resolving
- go.to - sold its domains on Sedo apparently.
- go2.me - everything 404s
- hashonomy.com - Timed out
- htcdev.net - DNS not resolving.
- iawtp.me - DNS not resolving
- icymi.me - DNS not resolving
- ilix.in - domain parked
- imfy.us - requires a recaptcha to get to the linked site, and avast goes nuts. DNS fails to resolve.
- inspr.in - Inspired Beta. Can't find server
- ix.it - Not resolving
- jijr.com - Doesn't appear to be a shortener, now parked
- jump.to - dead as of February 1, 2013
- kissa.be - "Kissa.be url shortener service is shutdown"
- kl.am - "kl.am Closes its Shell"
- kuijt.nu - replaced with unrelated site
- kurl.us - Parked.
- lnkurl.com - Website dead
- marv.ly - DNS fails to resolve.
- mash.to - Cannot connect.
- memurl.com - Pronounceable. Broken.
- me.lt - Connection refused.
- mens.hm - Not responding (timeout)
- miklos.dk - Doesn't appear guessable: http://miklos.dk/!z7bA6a - "Vi arbejder på sagen..."
- minilien.com - Doesn't appear guessable: http://minilien.com/?9nyvwnA0gh - Website dead
- minim.in - Times out
- minurl.org - Presently in ERROR 404
- ms.me - Parked.
- msplinks.com - Used by Myspace
- mtw.tl - everything 403s
- muhlink.com - Not resolving
- mytinyurl.com - redirects to an unrelated image
- myurl.us - cpanel frontend
- myv.bz - Not resolving
- nyturl.com - NY Times (bonus points for being longer than nyt.com, which they own). Taken by squatters
- onvzi.com - DNS fails to resolve.
- otf.me - Empty WordPress site
- ping.fm - Fails to resolve.
- pln.so - Not working.
- plzretwt.me - Fails to resolve.
- pnt.me - Doesn't appear guessable, too big a space to bruteforce: http://pnt.me/FzAblc
- pulsene.ws - Expired. Parked by GoDaddy.
- re.ad - Fails to resolve.
- redirx.com - Lowercase alpha only, appears sequential or guessable - Ex: http://redirx.com/?wyok. Website still online but does not resolve existing URLs nor does it allow creating new ones (responds with the message: blame the spammers)
- see.sc - Fails to resolve.
- s.me - Domain parked.
- say.ly - redirects to unrelated site
- s3nt.com - Probably sequential. http://s3nt.com/aa goes somewhere different from /ab . Domain parked.
- shortlinks.co.uk - Working again. Maybe not.
- short.to - Domain is parked - Probably sequential/loweralpha: http://short.to/msmp
- shrinklink.co.uk - Doesn't appear sequential: http://www.shrinklink.co.uk/45bmx , www.shrinklink.co.uk/npk6xp . Domain parked.
- shrtn.us - myshorturls.appspot.com. 404, does not resolve
- simurl.com - Doesn't appear guessable - Ex: http://simurl.com/panpes. Website is blank; does not resolve URLs ("This SimURL is now inactive")
- smf.is - DNS not resolving.
- sns.mx - SNS Analytics, domain parked
- sq.com - Now redirects to Singapore Airlines.
- tiny.ly - DNS not resolving.
- to.gg - Global Giving, everything 503s
- traceurl.com - DNS fails to resolve.
- tr.im (1st generation) - "Be back soon!"
- tweetburner.com / twurl.nl - Appears incremental, everything 404s
- twixar.com - "Estamos fora do ar por algum tempo, mas estamos trabalhando para voltar a oferecer o serviço para encurtar URLs longa em breve!"
- twthpr.co - DNS not resolving.
- twitpwr.com - Domain parked.
- u.mavrev.com - Stopped accepting new urls. Now times out
- u.nu - "The shortest URLs. period." Website dead since at least 1st of october 2010 (http://web.archive.org/web/20100104023208/http://u.nu/)
- url9.com - Sequential, alphanumeric. Leading 0s are significant. "The site is working correctly."
- urlborg.com - 404 Not Found.
- urlcover.com - Domain parked.
- urlhawk.com - Domain parked.
- url-press.com - Suspended by web host.
- urlsinn.com - DNS not resolving.
- urlsmash.com - DNS not resolving.
- urltea.com - Dreamhost's coming soon page.
- urlvi.be - Domain parked.
- urlx.org - Owner has agreed to share his database
- uxp.in -
still resolves URLs, but site just shows blank page. Domain parked.
- vibemag.co - Vibe Magazine. Times out
- vsb.li / links.visibli.com/links/ - The latter uses truncated md5 hex string. See sharedby.co.
- w3t.org - 403 Forbidden.
- wlink.us - Domain parked.
- wl.tl - DNS not resolving.
- xaddr.com - Domain parked.
- xil.in - Under construction.
- x.se - Cannot resolve, but www.x.se works.
- xym.kr - Gibberish (?) Korean text blog.
- yweb.com - Suspicious iframe with long url and fake loading gif image.
- zi.ma - DNS not resolving.
- zip.sm - was a redirect to joturl.com. Now times out
- urlbrief.com - co-operates with 301Works.org
- feedly.com/e/ - realized that URL shorteners were bad 
- adjix.com -
Still resolves URLs, but site does not work: "The requested application was not found on this server."- Is static host on AWS service.
Check out Audit2014 and help audit the archives. In particular, the stuff not on Internet Archive needs to be uploaded.
- See the latest torrent release for URLs before Tinytown. A copy is available at URLTeamTorrentRelease2013July
- Tinytown results are uploaded to the Internet Archive. They are incremental, so you will need to download them all to get all URLs.
Common URL shortening software
Ha-ha! Please don't run a URL shortening service.