URLTeam

Urlteam
url shortening was a fucking awful idea
URL	http://urlte.am
Status	Online!
Archiving status	In progress...
Archiving type	Unknown
Project source	https://github.com/ArchiveTeam/urlteam-stuff
Project tracker	http://urlteam.terrywri.st/
IRC channel	#urlteam (on hackint)

TinyURL, bit.ly and other similar services allow long URLs to be converted to smaller ones on their specific service; the small URL is visited by a consumer and their web browser is redirected to the long URL.

Such services are a ticking timebomb. If they go away, get hacked or sell out millions of links will be lost (see Wikipedia: Link Rot). Archive.org/301Works is acting as an escrow for URL shortener databases, but they rely on URL shorteners to actually give them their databases. Even 301Works founding member bit.ly does not actually share their databases and most other big shorteners don't share theirs either.

301Work cooperation

The fine folks at archive.org have provides us with upload permissions to the 301Works archive: http://www.archive.org/details/301utm. They unfortunately do not want to make them downloadable, but the same data is in our torrents too, just in a different format (we use pipe-delimited, xz-compressed files while 301works uses comma-delimited uncompressed files).

Tools

• fetcher.pl: Perl-based scraper by User:Chronomex
• TinyBack: Python 2.x-based, distributed scraper (also works with the Warrior)

TinyBack

The easiest way to help with scraping is to run the Warrior and select the URLTeam project. You can also run TinyBack outside the warrior, thought Python 2.6 or newer is required:

 git clone https://github.com/ArchiveTeam/tinyback
 cd tinyback
 # Use ./run.py --help for more information on command-line options
 ./run.py --tracker=http://urlteam.terrywri.st/ --num-threads=3 --sleep=180

URL shorteners

New table

The new table includes shorteners we have already started to scrape.

Name	Est. number of shorturls	Scraping done by	Status	Comments
Tinyurl.com	10,000,000,000	Warrior	scraping: sequential, done up to azzzzz	new shorturls: non-sequential, 7 characters
Bit.ly	50,000,000,000	Warrior	scraping: non-sequential, 6 characters	new shorturls: non-sequential, 6 characters
Goo.gl	?	User:Scumola	started (2011-03-04)	goo.gl throttles pulls
is.gd	934,134,706 (2013-05-20)	Warrior	scraping: sequential, done up to kZZZZ	new shorturls: non-sequential, 6 characters
ff.im	?	User:Chronomex		only used by FriendFeed, no interface to shorten new URLs
4url.cc	1279 (2009-08-14)	User:Chronomex		dead (2011-02-15)
litturl.com	17096 (2010-04-15)	User:Chronomex		dead (2010-11-18)
xs.md	3084 (2009-08-15)	User:Chronomex	done	dead (2010-11-18)
url.0daymeme.com	14867 (2009-08-14)	User:Chronomex	done	dead (2010-11-18)
Old tr.im	1990425	-	got what we could	dead (2011-12-31)
New tr.im	?	Warrior	scraping: sequential, done up to 42pzz	new shorturls: sequential
visibli (hex)	16777216	User:Chfoo	Done. 15104865 301MB	Using links.sharedby.co/links/ as URL prefix.
ur1.ca	?	Warrior	scraping: sequential, done up to dzzzz	new shorturls: sequential
ow.ly	?	Warrior	scraping: sequential, done up to lyZZZ	new shorturls: sequential
snipurl.com	?	Warrior	scraping: sequential, done up to 271~~~~	new shorturls: sequential, starting from 20wa5rt
post.ly (Posterous)	?	Warrior/EC2	done	dead
vbly.us (formerly vb.ly)	?	Warrior	scraping: sequential, done up to 2hba	new shorturls: sequential
arseh.at	?	Warrior	scraping: sequential, done up to 4fv3	new shorturls: sequential
zapd.co	326592	User:Chfoo	Done. 144093 1.7M	xxxx.zapd.co
Bre.ad	120932351	User:Chfoo	In progress (6766852 examined). 21463 517KB
Name	Number of shorturls	Scraping done by	Status	Comments

Alive

Last verified 2013-04-17. Original list last updated 2009-08-14 .

• adf.ly
• adjix.com
• ar.gy - Argyle Social
• ask.fm - ask.fm/a/40k05kgp
• awe.sm
• biglnk.com
• budurl.com - Appears non-incremental
• buff.ly - Buffer App
• burl.se
• calyp.co
• cli.gs - Appears non-incremental
• cl.ly - CloudApp
• cmt.com - Country Music Television
• decenturl.com - Not at all easy to scrape.
• del.ly - sprinklr
• df4.us - daringfireball.net
• dld.bz - "private URL shortening service"
• dlvr.it
• doiop.com - Appears non-incremental
• easyurl.net - Appears non-incremental: http://easyurl.net/afd2f
• flip.it - Flipboard
• flpbd.it - Flipboard
• fnd.us (See offical shorteners)
• fwdurl.net
• gob.li - Golbin Ridge Limited
• go.to
• go2.me - Appears incremental: http://u.go2.me/6YK http://u.go2.me/6YL
• hashonomy.com
• htl.li
• ht.ly
• ilix.in - HTML redirect
• inspr.in - Inspired Beta
• jdem.cz - Incremental with random (?) last digit: http://jdem.cz/bw388
• korta.nu
• kcy.me
• kuijt.nu
• lnq.me
• metamark.net / xrl.us - ? http://xrl.us/bfabog
• mens.hm
• migre.me
• mindless.co
• minim.in
• msft.it - Sprinklr
• msnbc.com
• msplinks.com
• mtw.tl
• myloc.me
• my.dot.tk/tweak - Appears non-incremental
• mytinyurl.com
• myurl.in - http://myurl.in/xtP5H / http://urlgator.com/xtP5H /http://ug4.me/xtP5H / http://link-ed.in/xtP5H - HTML redirect
• nblo.gs
• news.me
• nig.gr
• notlong.com - Appears to be alpha-only: http://yeitoo.notlong.com/
• nutshellurl.com - Appears incremental. 301s to a redirector script, which then 301s you to the destination.
• owl.li
• pear.ly
• ph.ly Related to the pond called Philadelphia, where links are born and raised
• pocket.co
• po.st
• prsm.tc - getprismatic.com
• r.ebay.com
• rod.gs
• redirx.com - Lowercase alpha only, appears sequential or guessable: http://redirx.com/?wyok
• say.ly
• sharedby.co - See vsb.li. Double redirects via USERNAME.sharedby.co/share/XXXXXX
• shar.es (See offical shorteners)
• shorl.com - Doesn't appear guessable: http://shorl.com/tisikestibahu
• shorturl.com - Probably sequential/loweralpha: http://alturl.com/wqok
• shrinkurl.us - Alway telling URL is malformed
• shrd.by - see sharedby.co
• shrt.st - Appears incremental: http://shrt.st/vpz
• shrtn.us - myshorturls.appspot.com
• simurl.com - Doesn't appear guessable: http://simurl.com/panpes
• smarturl.eu / joturl.com / zip.sm - Doesn't appear guessable, HTML redirect.
• smarturl.it - smartURL
• snipr.com / snipurl.com / snurl.com - Appears incremental: http://snipr.com/27nvst http://snipr.com/27nvtt
• sns.mx - SNS Analytics
• soa.li - Gigya inc.
• soc.li - Gigya inc.
• spne.ws - Silicon Prairie News
• spnsr.tw - sponsoredtweets.com
• srtn.us
• surl.co.uk - Many shortening options.
• techme.me - Techmeme
• tighturl.com - Appears incremental: http://tighturl.com/30xu http://tighturl.com/30xv
• tiny.cc - Appears non-incremental
• tinyarrows.com / ta.gd / ri.ms / ➡.ws / ➨.ws / ➯.ws / ➔.ws / ➞.ws / ➽.ws / ➹.ws / ✩.ws / ✿.ws / ❥.ws / ›.ws / ⌘.ws / ‽.ws / ☁.ws - Appears non-incremental: uses user-defined words for URLs (e.g. http://➡.ws/URLTEAM)
• tm.to - Twtmore
• to.gg - Global Giving
• trap.it
• trib.al
• tr.im - Appears incremental: http://tr.im/44tn2 http://tr.im/44tn4
• tweetburner.com / twurl.nl - Appears incremental
• twitthis.com
• u.mavrev.com - Not accepting new urls.
• urlcut.com
• uxp.in
• vibemag.co - Vibe Magazine
• vimeo.com
• vitrue.com - Now part of Oracle
• vk.cc
• waa.ai
• x.co - Appears incremental: http://x.co/1IxUV http://x.co/1IxUW
• xrl.us - see metamark.net
• yatuc.com - Not accepting new urls.
• yep.it
• yoolink.to - Yoolink

"Official" shorteners

• upl.nu - Ung Pirat (Youth Pirate Party, Sweden)
• bull.hn - Bullhorn Reach (format: bull.hn/l/19JQE/)
• CokeURL.com - Coca-Cola
• db.tt - DropBox
• di.sn - Disney
• fb.me - Facebook
• flic.kr - Flickr
• fnd.us - Fundrazr.com
• goo.gl - Google
• go.usa.gov - USA Government (and since they control the Internets, it doesn't get much more official than this)
• gu.com - The Guardian (weird format - https://gu.com/p/3f7ca )
• hub.me - HubPages
• igg.me - Indiegogo
• lnkd.in - LinkedIn
• post.ly - Posterous
• shar.es - ShareThis - 404 on homepage, otherwise ok
• skfb.ly - Sketchfab
• spoti.fi - Spotify
• stanford.io - Stanford University
• su.pr - StumbleUpon
• t.co - Twitter
• tmblr.co - Tumblr
• wapo.st - Washington Post
• wp.me - Wordpress.com
• y.ahoo.it - Yahoo
• youtu.be - YouTube

bit.ly aliases

• 1.usa.gov - USA Government
• 4sq.com - Foursquare
• aje.me - Aljazeera
• amzn.to - Amazon
• atfp.co - Foreign Policy
• bbc.in - BBC
• bbnew.be
• bbybgrl.com
• binged.it - Bing (bonus points for being longer than bing.com)
• bnkrpt.am - Bankrupting America
• bzfd.it - Buzzfeed
• cb.com - Career Builder
• chzb.gr - Cheezeburger
• cnet.co - CNET
• cnnmon.ie - CNN Money
• cmplx.it - Complex Magazine
• conta.cc - Constant Contact Inc.
• cot.ag
• cpurl.net - Current Photographer.com
• curbed.cc - Curbed.com
• dag.gy -
• dennysd.in - Denny's Restaurants
• dtoid.it - Destructoid
• econ.st - The Economist
• emarketee.rs
• engri.sh - Engrish.com
• eonli.ne
• es.pn - ESPN
• fakes.pn
• fanpa.ge - Fanpage.it
• gaw.kr - Gawker
• geekiss.im
• grd.to - The Grid TO
• grn.bz
• gtg.lu - GetGlue
• hoblu.es
• hub.am
• huff.to - Huffington Post
• ift.tt
• j.mp - bit.ly
• jrnl.to - thejournal.ie
• kck.st - Kickstarter
• marsdd.it - MaRS Discovery District
• mbist.ro
• mwne.ws
• ncl.uz
• nyti.ms - New York Times
• onforb.es - Forbes
• onion.com - The Onion
• pops.ci - Popular Science
• popu.pe
• psxs.us
• read.bi - Business Insider
• rseo.co - realseo
• s831.us - Studio831 - whatever that is
• sbn.to - sbnation
• skygrid.me - SkyGrid
• slackers.co - slackers.com
• shr.li
• s.shr.lc - shareaholic - Naive, redirects any shortcode to bit.ly
• stay.am
• stjo.es - St. Joseph Media
• squid.us - Laughing Squid
• tag.my
• tcrn.ch - Techcrunch
• theatln.tc - The Atlantic
• tnw.co - The Next Web
• tom.hn
• toms.sh - TOMS Shoes
• txpr.de - TexasStore
• unr.ly - Unruly media
• usat.ly - USA Today Newspaper
• vrge.co - The Verge
• zite.to - Zite

Dead or Broken

• 1link.in - Website dead
• 6url.com - HTML redirect, Error 500
• ad.vu - mirror of adjix.com, application not found
• bwtm.co - DNS fails to resolve.
• canurl.com - Website dead
• chod.sk - Appears non-incremental, not resolving
• da.co - Parked.
• digg.com - discontinued - [1]
• dwarfurl.com - Website dead/Numeric, appears incremental: http://dwarfurl.com/08041
• easy.tc - DNS not resolving.
• easyuri.com - Website dead/Appears hex incremental with last digit random/checksum: http://easyuri.com/1339f , http://easyuri.com/133a3
• eqent.me - Improper redirect to bitly.
• feedzil.la - Domain parked.
• go2cut.com - Website dead
• gonext.org - not resolving
• htcdev.net - DNS not resolving.
• iawtp.me - DNS not resolving
• icymi.me - DNS not resolving
• imfy.us - requires a recaptcha to get to the linked site, and avast goes nuts. DNS fails to resolve.
• ix.it - Not resolving
• jijr.com - Doesn't appear to be a shortener, now parked
• jump.to - dead as of February 1, 2013
• kissa.be - "Kissa.be url shortener service is shutdown"
• kl.am - "kl.am Closes its Shell"
• kurl.us - Parked.
• lk.to
• lnkurl.com - Website dead
• marv.ly - DNS fails to resolve.
• memurl.com - Pronounceable. Broken.
• me.lt - Connection refused.
• miklos.dk - Doesn't appear guessable: http://miklos.dk/!z7bA6a - "Vi arbejder på sagen..."
• minilien.com - Doesn't appear guessable: http://minilien.com/?9nyvwnA0gh - Website dead
• minurl.org - Presently in ERROR 404
• ms.me - Parked.
• muhlink.com - Not resolving
• myurl.us - cpanel frontend
• myv.bz - Not resolving
• nyturl.com - NY Times (bonus points for being longer than nyt.com, which they own). Taken by squatters
• onvzi.com - DNS fails to resolve.
• otf.me - Empty WordPress site
• ping.fm - Fails to resolve.
• pln.so - Not working.
• plzretwt.me - Fails to resolve.
• pnt.me - Doesn't appear guessable, too big a space to bruteforce: http://pnt.me/FzAblc
• pulsene.ws - Expired. Parked by GoDaddy.
• qurlyq.com - Javascript redirect. Appears sequential: http://qurlyq.com/5nf. Domain parked.
• re.ad - Fails to resolve.
• see.sc - Fails to resolve.
• s.me - Domain parked.
• s3nt.com - Probably sequential. http://s3nt.com/aa goes somewhere different from /ab . Domain parked.
• shortlinks.co.uk - Working again. Maybe not.
• short.to - Domain is parked - Probably sequential/loweralpha: http://short.to/msmp
• shrinklink.co.uk - Doesn't appear sequential: http://www.shrinklink.co.uk/45bmx , www.shrinklink.co.uk/npk6xp . Domain parked.
• smf.is - DNS not resolving.
• sq.com - Now redirects to Singapore Airlines.
• tiny.ly - DNS not resolving.
• traceurl.com - DNS fails to resolve.
• tr.im (1st generation) - "Be back soon!"
• twixar.com - "Estamos fora do ar por algum tempo, mas estamos trabalhando para voltar a oferecer o serviço para encurtar URLs longa em breve!"
• twthpr.co - DNS not resolving.
• twitpwr.com - Domain parked.
• u.nu - "The shortest URLs. period." Website dead since at least 1st of october 2010 (http://web.archive.org/web/20100104023208/http://u.nu/)
• url9.com - Sequential, alphanumeric. Leading 0s are significant. "The site is working correctly."
• urlborg.com - 404 Not Found.
• urlcover.com - Domain parked.
• urlhawk.com - Domain parked.
• url-press.com - Suspended by web host.
• urlsinn.com - DNS not resolving.
• urlsmash.com - DNS not resolving.
• urltea.com - Dreamhost's coming soon page.
• urlvi.be - Domain parked.
• urlx.org - Owner has agreed to share his database
• vsb.li / links.visibli.com/links/ - The latter uses truncated md5 hex string. See sharedby.co.
• w3t.org - 403 Forbidden.
• wlink.us - Domain parked.
• wl.tl - DNS not resolving.
• xaddr.com - Domain parked.
• xil.in - Under construction.
• x.se - Cannot resolve, but www.x.se works.
• xym.kr - Gibberish (?) Korean text blog.
• yweb.com - Suspicious iframe with long url and fake loading gif image.
• zi.ma - DNS not resolving.

Discontinued

• urlbrief.com - co-operates with 301Works.org

Hueg list

[2]

Weblinks

• urlte.am
• 301works.org
• Why URL shortening services and shortURLs are bad