Difference between revisions of "URLTeam"

From Archiveteam
Jump to navigation Jump to search
Line 152: Line 152:
=== Old list<ref>http://blog.go2.me/2009/01/exhausting-review-of-link-shorteners.html</ref> ===
=== Old list<ref>http://blog.go2.me/2009/01/exhausting-review-of-link-shorteners.html</ref> ===
List last updated 2009-08-14.
List last updated 2009-08-14.
* 1link.in - Website dead
* 6url.com - HTML redirect
* 6url.com - HTML redirect
* ad.vu - mirror of adjix.com
* ad.vu - mirror of adjix.com
* budurl.com - Appears non-incremental
* budurl.com - Appears non-incremental
* canurl.com - Website dead
* cli.gs - Appears non-incremental
* cli.gs - Appears non-incremental
* decenturl.com - Not at all easy to scrape.
* decenturl.com - Not at all easy to scrape.
* dlvr.it
* dlvr.it
* doiop.com - Appears non-incremental
* doiop.com - Appears non-incremental
* dwarfurl.com - Website dead/Numeric, appears incremental: http://dwarfurl.com/08041
* easyuri.com - Website dead/Appears hex incremental with last digit random/checksum: http://easyuri.com/1339f , http://easyuri.com/133a3
* easyurl.net - Appears non-incremental: http://easyurl.net/afd2f
* easyurl.net - Appears non-incremental: http://easyurl.net/afd2f
* go2cut.com - Website dead
* ilix.in - HTML redirect
* ilix.in - HTML redirect
* imfy.us - requires a recaptcha to get to the linked site, and avast goes nuts.
* imfy.us - requires a recaptcha to get to the linked site, and avast goes nuts.
* jdem.cz - Incremental with random (?) last digit: http://jdem.cz/bw388
* jdem.cz - Incremental with random (?) last digit: http://jdem.cz/bw388
* lnkurl.com - Website dead
* memurl.com - Pronounceable.  Broken.
* metamark.net / xrl.us - ? http://xrl.us/bfabog
* metamark.net / xrl.us - ? http://xrl.us/bfabog
* minilien.com - Doesn't appear guessable: http://minilien.com/?9nyvwnA0gh - Website dead
* myurl.in - http://myurl.in/xtP5H / http://urlgator.com/xtP5H /http://ug4.me/xtP5H / http://link-ed.in/xtP5H - HTML redirect
* myurl.in - http://myurl.in/xtP5H / http://urlgator.com/xtP5H /http://ug4.me/xtP5H / http://link-ed.in/xtP5H - HTML redirect
* lnkurl.com
* memurl.com - Pronounceable.  Broken.
* metamark.net / xrl.us - ? http://xrl.us/bfabog
* minilien.com - Doesn't appear guessable: http://minilien.com/?9nyvwnA0gh
* minilien.com - Doesn't appear guessable: http://minilien.com/?9nyvwnA0gh
* myurl.in - Doesn't appear guessable, is probably bruteforceable: http://myurl.in/lT7z5
* notlong.com - Appears to be alpha-only: http://yeitoo.notlong.com/
* notlong.com - Appears to be alpha-only: http://yeitoo.notlong.com/
* nutshellurl.com - Appears incremental.  301s to a redirector script, which then 301s you to the destination.
* nutshellurl.com - Appears incremental.  301s to a redirector script, which then 301s you to the destination.
* ow.ly - I can't get it to work.
* ow.ly - I can't get it to work.
* plexp.com - Parked.
* pnt.me - Doesn't appear guessable, too big a space to bruteforce: http://pnt.me/FzAblc
* pnt.me - Doesn't appear guessable, too big a space to bruteforce: http://pnt.me/FzAblc
* poprl.com - Not resolving
* qurlyq.com - Javascript redirect.  Appears sequential: http://qurlyq.com/5nf
* qurlyq.com - Javascript redirect.  Appears sequential: http://qurlyq.com/5nf
* redirx.com - Lowercase alpha only, appears sequential or guessable: http://redirx.com/?wyok
* redirx.com - Lowercase alpha only, appears sequential or guessable: http://redirx.com/?wyok
* s3nt.com - Probably sequential.  http://s3nt.com/aa goes somewhere different from /ab
* s3nt.com - Probably sequential.  http://s3nt.com/aa goes somewhere different from /ab
* shorterlink.com - Parked.
* shortlinks.co.uk - Working again.
* shortlinks.co.uk - Not resolving
* short.to - Domain is parked - Probably sequential/loweralpha: http://short.to/msmp
* short.to - Domain is parked - Probably sequential/loweralpha: http://short.to/msmp
* shorturl.com - Probably sequential/loweralpha: http://alturl.com/wqok
* shorturl.com - Probably sequential/loweralpha: http://alturl.com/wqok
* shrinklink.co.uk - Doesn't appear sequential: http://www.shrinklink.co.uk/45bmx , www.shrinklink.co.uk/npk6xp
* shrinklink.co.uk - Doesn't appear sequential: http://www.shrinklink.co.uk/45bmx , www.shrinklink.co.uk/npk6xp
* shrinkurl.us
* shrinkurl.us - Alway telling URL is malformed
* shrt.st
* shrt.st - Appears incremental: http://shrt.st/vpz
* shurl.net
* simurl.com - Doesn't appear guessable: http://simurl.com/panpes
* simurl.com
* shorl.com - Doesn't appear guessable: http://shorl.com/tisikestibahu
* shorl.com
* smarturl.eu / joturl.com / zip.sm - Doesn't appear guessable, HTML redirect.
* smarturl.eu
* snipr.com - Appears incremental: http://snipr.com/27nvst http://snipr.com/27nvtt
* snipr.com
* snipurl.com - See above ^
* snipurl.com
* snurl.com - See above above ^^
* snurl.com
* surl.co.uk - Many shortening options.
* sn.vc
* tighturl.com - Appears incremental: http://tighturl.com/30xu http://tighturl.com/30xv
* starturl.com
* tiny.cc - Appears non-incremental
* surl.co.uk
* tighturl.com
* timesurl.at
* tiny123.com
* tiny.cc
* tinylink.com
* tobtr.com
* traceurl.com
* traceurl.com
* tr.im
* tr.im
* tweetburner.com
* tweetburner.com / twurl.nl - Appears incremental
* twitpwr.com
* twitpwr.com
* twitthis.com
* twitthis.com

Revision as of 20:52, 19 March 2011

Urlteam
url shortening was a fucking awful idea
url shortening was a fucking awful idea
URL http://urlte.am
Status Online!
Archiving status In progress...
Archiving type Unknown
IRC channel #archiveteam-bs (on hackint)

TinyURL and other similar services allow long URLs to be converted to smaller ones on their specific service; the small URL is visited by a consumer and their web browser is redirected to the long URL.

Services like TinyURL are a ticking timebomb. If they go away, get hacked or sell out millions of links will be lost (see Wikipedia: Link Rot). 301Works.org claims to archive shorturls, but so far they have not released a single backup file for download. As always, the Archiveteam is here to help with their Urlteam subcommitee.

Who did this?

You can join us in our IRC channel: #urlteam on EFNet

Tools

  • TinyBack (written in ruby by User:Soult)
  • User:Chronomex wrote his own Perl-based scraper: [1]
  • The Monkeyshines algorithmic scraper has a tool to efficiently scrape URL shortening services -- see the examples/shorturls directory. It scales efficiently to tens of millions of saved URLs. It uses a read-through cache to prevent re-requesting urls, it allows multiple scrapers to run on different machines while sharing the same lookup cache. You can either feed it a list of bare urls, or have it randomly try either base-36 or base-62 URLs. With it, User:Mrflip gathered about 6M valid URLs pulled from twitter messages so far.

Or just ask!

Here's a template that worked for me at least once. Well, data is pending but the site owner is gung-ho.

Try sending an email to the website owner:

Hello!

I'm working with Jason Scott of textfiles.org and other members of the
Archive Team.

Since the recent scare involving http://tr.im/'s announced (and then
retracted) imminent demise, we've been working to archive all the
links from URL shorteners around the Internet.

If I'm not mistaken, you operate urlx.org.  Would you be so kind as to
share with us a copy of your URL database?  We'll do our best to
preserve this data forever in a useful way.

We are already very far along in scraping links from tr.im, but it's
faster (and friendlier!) to contact site owners asking for a copy of
their data than it is to scrape.

We've got a domain registered, urlte.am, and all links will be
available for redirect in the format:

http://urlx.org.urlte.am/av3

If you could help us, that would be excellent!

Thank you,

URL shorteners

New table

The new table includes shorteners we have already started to scrape.

Name Number of shorturls Scraping done by Status Comments
TinyURL 1,000,000,000 User:Soult 5-letter codes done, on halt due to being banned (2010-12-20) non-sequential, bans IP for requesting too many non-existing shorturls
bit.ly 4,000,000,000 User:Soult about 1/3 (2011-02-15) non-sequential
goo.gl ?? User:Scumola started (2011-03-04) goo.gl throttles pulls
is.gd 354,527,352 User:Chronomex/User:Soult probably got about 95% before switch to non-sequential now non-sequential, new software version added crappy rate limiting
ff.im ? User:Chronomex only used by FriendFeed, no interface to shorten new URLs
4url.cc 1279 (2009-08-14)[1] User:Chronomex done (2009-08-14) dead (2011-02-15)
litturl.com 17096[2] User:Chronomex done dead (2010-11-18)
xs.md 3084 (2009-08-15)[3] User:Chronomex done dead (2010-11-18)
url.0daymeme.com 14867 (2009-08-14)[4] User:Chronomex done dead (2010-11-18)
tr.im ? User:Soult 5-letter codes finished, 6-letter codes in progress resolving still works despite planning to completely shut down at the end of 2010 (2011-02-15), whoever owns that thing is a major pain in the ass
adjix.com ? User:Jeroenz0r Already done: 00-zz, 000-zzz, 0000-izzz. case-insensitive, incremental
rod.gs ? User:Jeroenz0r Done: 00-ZZ, 000-2Qc case-sensitive, incremental, server can't keep up with all the requests.
biglnk.com ? User:Jeroenz0r Done: 0-Z, 00-ZZ, 000-ZZZ case-sensitive, incremental
Name Number of shorturls Scraping done by Status Comments

Old list[5]

List last updated 2009-08-14.

"Official" shorteners

  • goo.gl - Google
  • fb.me - Facebook
  • amzn.to - Amazon
  • binged.it - Bing (bonus points for being longer than bing.com)
  • y.ahoo.it - Yahoo
  • youtu.be - YouTube
  • t.co? - Twitter
  • post.ly - Posterous
  • wp.me - Wordpress.com
  • flic.kr - Flickr
  • lnkd.in - LinkedIn
  • su.pr - StumbleUpon
  • go.usa.gov - USA Government (and since they control the Internets, it doesn't get much more official than this)
  • nyturl.com - NY Times (bonus points for being longer than nyt.com, which they own). Not a shortener anymore.
  • tcrn.ch - Techcrunch
  • digg.com - discontinued - [2]

Dead or Broken Shorteners

  • chod.sk - Appears non-incremental, not resolving
  • gonext.org - not resolving
  • ix.it - Not resolving
  • jijr.com - Doesn't appear to be a shortener, now parked
  • kissa.be - "Kissa.be url shortener service is shutdown"
  • kurl.us - Parked.
  • miklos.dk - Doesn't appear guessable: http://miklos.dk/!z7bA6a - "Vi arbejder på sagen..."
  • minurl.org - Presently in ERROR 404
  • muhlink.com - Not resolving
  • myurl.us - cpanel frontend

Hueg list

[3]


References

Weblinks