Difference between revisions of "TwitPic"

From Archiveteam
Jump to navigation Jump to search
(update empty zip file)
(Update export section)
Line 159: Line 159:
Twitpic's export tool is buggy, handing out seemingly empty zip files<ref>https://news.ycombinator.com/item?id=8473393</ref> and 503 errors.<ref>https://twitter.com/textfiles/status/522837349676236801</ref> The empty zip file problem can sometimes be fixed:
Twitpic's export tool is buggy, handing out seemingly empty zip files<ref>https://news.ycombinator.com/item?id=8473393</ref> and 503 errors.<ref>https://twitter.com/textfiles/status/522837349676236801</ref> The empty zip file problem can sometimes be fixed:
The problem is twofold.  If the problem is on a non-Windows computer, it is probably a corrupted download (which happens way too often).  On Windows, the built in zip file handler is not able to reliably handle zip files.  7z seems to have the most success with the zip file but others have worked as well.
The problem is twofold.  If the problem is on a non-Windows computer, it is probably a corrupted download (which happens way too often).  On Windows, the built in zip file handler is not able to reliably handle zip files.  [[http://www.7-zip.org/]] seems to have the most success with the zip file but others have worked as well.
General process to follow:
<li>Download and install 7z</li>
<li>Download the zip file and rename it something short.</li>
<li>Open a command prompt.</li>
<li>run the command 7z t zipfilename.zip</li>
<li>If it tests successfully run 7z x zipfilename.zip</li>
<li>Browse to the photo directory.</li>
<li>Pictures should be visible and a text file with the metadata.</li>
== Downloaders ==
== Downloaders ==

Revision as of 15:24, 18 October 2014

TwitPic logo
TwitPic mainpage in 2011-01-12
TwitPic mainpage in 2011-01-12
URL http://twitpic.com
Project status Closing
Archiving status In progress...
Project source twitpic-discovery, twitpic-grab, twitpic-items, twitpic-cloudfront-grab
Project tracker twitpicdisco, twitpic, twitpic-cloudfront
IRC channel #quitpic (on EFnet)
Project lead Unknown

TwitPic is an image hosting service. The service is designed mainly for Twitter users - the images uploaded on the service are given short URLs for usage in Twitter posts. Twitter carries a 140-character post limit, the average Twitpic URL is 25/26 characters long.

On September 4, 2014 TwitPic announced they were shutting down on September 25. On September 18, 2014, TwitPic announced that they'd been acquired and would "live on". However, on October 16, 2014, Twitpic announced that "agreeable terms could not be met" and that the service would be shutting down on October 25th.


Twitpic's erratic demise

Posted on September 4, 2014 by Noah Everett on blog.twitpic.com:

"Twitpic will be shutting down September 25th. You will be able to export all your photos and videos. We’ll let everyone know when this feature is live in the next few days.

This is an unexpected and hard announcement for us to make and we want to lay out what led us to this decision.

A few weeks ago Twitter contacted our legal demanding that we abandon our trademark application or risk losing access to their API. This came as a shock to us since Twitpic has been around since early 2008, and our trademark application has been in the USPTO since 2009.

Here is some backstory on the history of our trademark:

We originally filed for our trademark in 2009 and our first use in commerce dates back to February 2008 when we launched. We encountered several hurdles and difficulties in getting our trademark approved even though our first use in commerce predated other applications, but we worked through each challenge and in fact had just recently finished the last one. During the “published for opposition” phase of the trademark is when Twitter reached out to our counsel and implied we could be denied access to their API if we did not give up our mark.

Unfortunately we do not have the resources to fend off a large company like Twitter to maintain our mark which we believe whole heartedly is rightfully ours. Therefore, we have decided to shut down Twitpic.

On a personal note I (@noaheverett) want to thank you for letting us be a part of your life and helping you share your experiences over the past 6 years, it’s truly been an honor. I have learned so much through running Twitpic over the years. Through the many mistakes I’ve made and lessons learned, to the bad days and the great days. Thank you again everyone…I will miss and cherish the days of Twitpic we had together."

Won't (?) shut down

Twitpic writes on Twitter, on September 18, 2014:

"We're happy to announce we've been acquired and Twitpic will live on! We will post more details as we can disclose them"

However, ArchiveTeam goes on downloading TwitPic, for safety.


UPDATE on blog.twitpic.com:

"It’s with a heavy heart that I announce again that Twitpic will be shutting down on October 25th. We worked through a handful of potential acquirers and exhausted all potential options. We were almost certain we had found a new home for Twitpic (hence our previous tweet), but agreeable terms could not be met. Normally we wouldn’t announce something like that prematurely but we were hoping to let our users know as soon as possible that Twitpic was living on.
I’m sincerely sorry (and embarrassed) for the circumstances leading up to this, from our initial shutdown announcement to an acquisition false alarm.
You can export your data and photos at: http://twitpic.com/account/settings "

But wait! There's more!

On October 17th, 2014, Twitpic began blocking public access to images,[1] replacing them with a shutdown notice. Comments are still available for now, but the images are not.

Site structure

Image page urls:

http://twitpic.com/****** http://twitpic.com/***** http://twitpic.com/**** http://twitpic.com/*** http://twitpic.com/** http://twitpic.com/* where * = 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t, u, v, w, x, y, z

where ****** consists of up to 6 alphanumeric characters. Leading zeros are irrelevant, e.g.: /000joe = /0joe = /joe. Like incremental numbers in base-36 numeral system.


Phase 1: content discovery

From September 5 to 6, until ArchiveTeam got banned, ~41 million of the possible ~900 million urls were discovered. The discovery was suspended.

On September 6th, someone claiming to be Noah Everett showed up in #quitpic[2]:

[16:21:14] <n00b957> hey guys
[16:21:16] <n00b957> Noah Everett here
[16:21:26] <n00b957> noticed the site was really bogging down due to ArchiveTeam requersts
[16:21:30] <n00b957> *requests
[16:21:55] <n00b957> didn't know what it was at first so we blocked it to continue normal site operations and users can get their data easily
[16:22:27] <n00b957> just wanted to give a heads up so you don't think we are trying to be malicious
[16:23:00] <n00b957> we're working on getting our export tool out the door right now
[16:23:14] <n00b957> I'd like to let our users get their data off the site via that first as quickly as possible

Unfortunately, he left #quitpic shortly afterwards and has not returned any of Archive Team's repeated inquiries about archiving Twitpic.

Phase 2: content grab

After some testing, actual content grab began on September 14. Its progress can be followed on the tracker. (One item contains 36 images and/or other elements of the image pages.)

How can I help?

Important notice: TwitPic staff may ban ArchiveTeam members' access to their site through AT tools, or completely (IP address ban), and for a long time. If you want to use TwitPic outside ArchiveTeam tools (e.g. if you have an account there and you want to access it), consider running the Warrior/script with low concurrency, or, if you're paranoid, not running it at all.

Running a Warrior

You can start up a Warrior and there select TwitPic Phase 2. (If you don't really care what you are archiving, select ArchiveTeam's Choice instead, as at some points ArchiveTeam may priorize another project.)

If you see "Project code is out of date", simply restart the warrior.

Running the script manually

If you use Linux and you're a bit familiar with it, you can try running the script directly.

The instructions can be found at twitpic-grab.

Don't forget to replace YOURNICKHERE with your nickname.

The number after --concurrent determines how many threads run at the same time. You can increase this number if your resources (RAM, CPU, bandwidth) are sufficient. However, if you constantly see messages about rate limiting, there is no need to increase the concurrency. Note: the higher the concurrency is, the more the chance is to be banned by TwitPic staff.

If you want to stop the script, please do it gracefully if possible. To do so, create an empty file named STOP in the folder of the script (terminal command: touch STOP). The script finishes the current item(s) and stops only after that. (If you kill the script immediately, the items get broken, and they will need to be reassigned to another user.) – Before starting the script again, don't forget to remove the STOP file.

If you see "Project code is out of date", kill the script, go to its folder (cd twitpic-grab) and issue git pull https://github.com/ArchiveTeam/twitpic-grab. After the updating has finished, re-launch the script.

For both Warrior and script

If you see 403 error codes in the output of the script about files not on twitpic.com (e.g. on twimg.com or else), don't worry. That is normal and the script handles the problem. However, if the script receives 403s from twitpic.com, your script (or even your IP address) has possibly got blocked. You can retry later, but you may be banned for a long time.

Joining us on IRC

Either you run the Warrior or the script, you should join our IRC channel #quitpic to catch the latest news about the project and its progress, and there you can also put questions if something doesn't work. You can use the web interface at http://chat.efnet.org:9090, or if you use a standalone IRC client, connect to irc://irc.efnet.org.

Spread the word!

Time is short and we must grab a lot of stuff. Furthermore, it seems that many users running a few threads is better than one user running a lot. So, please try to get some more people work on this project! Speak/write about it, tell your friends. This is definitely an urgent and important project.

Contents saved from TwitPic will be given to and stored by the Internet Archive. The amount of the data is tens of terabytes. This will cost thousands of dollars to store in the long run, so if you an afford, please donate to the Internet Archive so that contents of TwitPic can be available forever. http://archive.org/donate


As TwitPic is probably not shutting down, it seems to be needless to store the downloaded data (more than 100 terabytes). However, TwitPic cannot be considered a reliable service anymore. So, archives should be stored somewhere. But the Internet Archive will probably not be willing to ingest this amount of data also present on the internet.

To solve this problem, to decide where to store the data grabbed from TwitPic (and from sites in similar situation), Project Valhalla has been established. Read more about it on the linked wiki page.

Archives of TwitPic will be stored by the Internet Archive. Please donate if you can so that the costs of storing can be covered in the long run.

Download Your Data

User elise81 writes on Reddit:

"Log into Twitpic.com, click settings, scroll to bottom and click the request your data button. It takes a little while, but you'll eventually get a .zip file of all of your data."

"You can export your data and photos at: http://twitpic.com/account/settings"

When it's about your content, don't rely on ArchiveTeam's archives, as they may be incomplete, and are not made in a way that a single user's content can be extracted from them. Use the export-tool!

Export Tool Bugs

Twitpic's export tool is buggy, handing out seemingly empty zip files[3] and 503 errors.[4] The empty zip file problem can sometimes be fixed:

The problem is twofold. If the problem is on a non-Windows computer, it is probably a corrupted download (which happens way too often). On Windows, the built in zip file handler is not able to reliably handle zip files. [[1]] seems to have the most success with the zip file but others have worked as well.

General process to follow:

  1. Download and install 7z
  2. Download the zip file and rename it something short.
  3. Open a command prompt.
  4. run the command 7z t zipfilename.zip
  5. If it tests successfully run 7z x zipfilename.zip
  6. Browse to the photo directory.
  7. Pictures should be visible and a text file with the metadata.


  • Downloader by tag (it saves the full resolution image and metadata: uploader, date and description)


External links

v · t · e         Archive Team
Current events

Alive... OR ARE THEY · Deathwatch · Projects

Archiving projects

APKMirror · Archive.is · BetaArchive · Government Backup (#datarefuge · ftp-gov· Gmane · Internet Archive · It Died · Megalodon.jp · OldApps.com · OldVersion.com · OSBetaArchive · TEXTFILES.COM · The Dead, the Dying & The Damned · The Mail Archive · UK Web Archive · WebCite · Vaporwave.me


Blog.pl · Blogger · Blogster · Blogter.hu · Freeblog.hu · Fuelmyblog · Jux · LiveJournal · My Opera · Nolblog.hu · Open Diary · ownlog.com · Posterous · Powerblogs · Proust · Roon · Splinder · Tumblr · Vox · Weblog.nl · Windows Live Spaces · Wordpress.com · Xanga · Yahoo! Blog · Zapd

Cloud hosting/file sharing

aDrive · AnyHub · Box · Dropbox · Docstoc · Fast.io · Google Drive · Google Groups Files · iCloud · Fileplanet · LayerVault · MediaCrush · MediaFire · Mega · MegaUpload · MobileMe · OneDrive · Pomf.se · RapidShare · Ubuntu One · Yahoo! Briefcase


Apple · IBM · Google · Loblaw · Lycos Europe · Microsoft · Yahoo!


Arab Spring · Great Ape-Snake War · Spanish Revolution

Font Repos

DaFont · Google Web Fonts · GNU FreeFont · Fontspace

Forums/Message boards

4chan · Captain Luffy Forums · College Confidential · DSLReports · ESPN Forums · Facepunch Forums · forums.starwars.com · HeavenGames · JamiiForums · Invisionfree · NeoGAF · Textream · The Classic Horror Film Board · Yahoo! Messages · Yahoo! Neighbors · Yuku.com · Zetaboards


Atomicgamer · Bazaar.tf · City of Heroes · Club Nintendo · Clutch · Counter-Strike: Global Offensive · CS:GO Lounge · Desura · Dota 2 · Dota 2 Lounge · Emulation Zone · ESEA · GameBanana · GameMaker Sandbox · GameTrailers · Halo · HLTV.org · HQ Trivia · Infinite Crisis · joinDOTA · League of Legends · Liquipedia · Minecraft.net · Player.me · Playfire · Raptr · SingStar · Steam · SteamDB · SteamGridDB · Team Fortress 2 · TF2 Outpost · Warhammer · Xfire

Image hosting

500px · AOL Pictures · Blipfoto · Blingee · Canv.as · Camera+ · Cameroid · DailyBooth · Degree Confluence Project · DeviantART · Demotivalo.net · Flickr · Fotoalbum.hu · Fotolog.com · Fotopedia · Frontback · Geograph Britain and Ireland · Giphy · GTF Képhost · ImageShack · Imgh.us · Imgur · Inkblazers · Instagram · Kepfeltoltes.hu · Kephost.com · Kephost.hu · Kepkezelo.com · Keptarad.hu · Madden GIFERATOR · MLKSHK · Microsoft Clip Art · Microsoft Photosynth · Nokia Memories · noob.hu · Odysee · Panoramio · Photobucket · Picasa · Picplz · Pixiv · Portalgraphics.net · PSharing · Ptch · puu.sh · Rawporter · Relay.im · ScreenshotsDatabase.com · Sketch · Smack Jeeves · Snapjoy · Streetfiles · Tabblo · Tinypic · Trovebox · TwitPic · Wallbase · Wallhaven · Webshots · Wikimedia Commons


arXiv · Citizendium · Clipboard.com · Deletionpedia · EditThis · Encyclopedia Dramatica · Etherpad · Everything2 · infoAnarchy · GeoNames · GNUPedia · Google Books (Google Books Ngram· Horror Movie Database · Insurgency Wiki · Knol · Lost Media Wiki · Neoseeker.com · Notepad.cc · Nupedia · OpenCourseWare · OpenStreetMap · Orain · Pastebin · Patch.com · Project Gutenberg · Puella Magi · Referata · Resedagboken · SongMeanings · ShoutWiki · The Internet Movie Database · TropicalWikis · Uncyclopedia · Urban Dictionary · Urban Exploration Resource · Webmonkey · Wikia · Wikidot · WikiHow · Wikkii · WikiLeaks · Wikipedia (Simple English Wikipedia· Wikispaces · Wikispot · Wik.is · Wiki-Site · WikiTravel · Word Count Journal


Cyberpunkreview.com · Game Developer Magazine · Gigaom · Hardware Canucks · Helium · JPG Magazine · Make Magazine · The Escapist · Polygamia.pl · San Fransisco Bay Guardian · Scoop · Regretsy · Yahoo! Voices


Heello · Identi.ca · Jaiku · Mommo.hu · Plurk · Sina Weibo · Tencent Weibo · Twitter · TwitLonger


8tracks · AOL Music · Audimated.com · Cinch · digCCmixter · Dogmazic.net · Earbits · exfm · Free Music Archive · Gogoyoko · Indaba Music · Instacast · Instaudio · Jamendo · Last.fm · Music Unlimited · MOG · PureVolume · Reverbnation · ShareTheMusic · SoundCloud · Soundpedia · Spotify · This Is My Jam · TuneWiki · Twaud.io · WinAmp


Aaron Swartz · Michael S. Hart · Steve Jobs · Mark Pilgrim · Dennis Ritchie · Len Sassaman Project


FTP · Gopher · IRC · Usenet · World Wide Web
BitTorrent DHT


Askville · Answerbag · Answers.com · Ask.com · Askalo · Baidu Knows · Blurtit · ChaCha · Experts Exchange · Formspring · GirlsAskGuys · Google Answers · Google Baraza · JustAnswer · MetaFilter · Quora · Retrospring · StackExchange · The AnswerBank · The Internet Oracle · Uclue · WikiAnswers · Yahoo! Answers


Allrecipes · Epicurious · Food.com · Foodily · Food Network · Punchfork · ZipList

Social bookmarking

Addinto · Backflip · Balatarin · BibSonomy · Bkmrx · Blinklist · BlogMarks · BookmarkSync · CiteULike · Connotea · Delicious · Designer News · Digg · Diigo · Dir.eccion.es · Evernote · Excite Bookmark · Faves · Favilous · folkd · Freelish · Getboo · GiveALink.org · Gnolia · Google Bookmarks · Hacker News · HeyStaks · IndianPad · Kippt · Knowledge Plaza · Licorize · Linkwad · Menéame · Microsoft Developer Network · myVIP · Mister Wong · My Web · Mylink Vault · Newsvine · Oneview · Pearltrees · Pinboard · Pocket · Propeller.com · Reddit · sabros.us · Scloog · Scuttle · Simpy · SiteBar · Slashdot · Squidoo · StumbleUpon · Twine · Voat · Vizited · Yummymarks · Xmarks · Yahoo! Buzz · Zootool · Zotero

Social networks

Bebo · BlackPlanet · Classmates.com · Cyworld · Dogster · Dopplr · douban · Ello · Facebook · Flixster · FriendFeed · Friendster · Friends Reunited · Gaia Online · Google+ · Habbo · hi5 · Hyves · iWiW · LinkedIn · Miiverse · mixi · MyHeritage · MyLife · Myspace · myVIP · Netlog · Odnoklassniki · Orkut · Plaxo · Qzone · Renren · Skyrock · Sonico.com · Storylane · Tagged · tvtag · Upcoming · Viadeo · Vine · Vkontakte · WeeWorld · Weibo · Wretch · Yahoo! Groups · Yahoo! Stars India · Yahoo! Upcoming · more sites...


Alibaba · AliExpress · Amazon · Apple Store · Barnes & Noble · DirectCanada · eBay · Kmart · NCIX · Printfection · RadioShack · Sears · Sears Canada · Target · The Book Depository · ThinkGeek · Toys "R" Us · Walmart

Software/code hosting

Android Development · Alioth · Assembla · BerliOS · Betavine · Bitbucket · BountySource · Codecademy · CodePlex · Freepository · Free Software Foundation · GNU Savannah · GitHost  · GitHub · GitHub Downloads · Gitorious · Gna! · Google Code · ibiblio · java.net · JavaForge · KnowledgeForge · Launchpad · LuaForge · Maemo · mozdev · OSOR.eu · OW2 Consortium · Openmoko · OpenSolaris · Ourproject.org · Ovi Store · Project Kenai · RubyForge · SEUL.org · SourceForge · Stypi · TestFlight · tigris.org · Transifex · TuxFamily · Yahoo! Downloads


ABC · Austin City Limits · BBC · CBC · CBS · Computer Chronicles · CTV · Fox · G4 · Global TV · Jeopardy! · NBC · NHK · PBS · Penn & Teller: Bullshit! · The Howard Stern Show · TV News Archive (Understanding 9/11)


ExtraTorrent · EZTV · isoHunt · KickassTorrents · The Pirate Bay · Torrentz · Library Genesis

Video hosting

Academic Earth · Bambuser · Blip.tv · Epic · Freshlive · Google Video · Justin.tv · Mixer · Niconico · Nokia Trailers · Oddshot.tv · Periscope · Plays.tv · Qwiki · Skillfeed · Stickam · TED Talks · Ticker.tv · Twitch.tv · Ustream · Videoplayer.hu · Viddler · Viddy · Vidme · Vimeo · Vine · Vstreamers · Yahoo! Video · YouTube · Famous Internet videos (Me at the zoo)

Web hosting

Angelfire · Brace.io · BT Internet · CableAmerica Personal Web Space · Claranet Netherlands Personal Web Pages · Comcast Personal Web Pages · Extra.hu · FortuneCity · Free ProHosting · GeoCities (patch· Google Business Sitebuilder · Google Sites · Internet Centrum · MBinternet · MSN TV · Nifty · Nwnyet · Parodius Networking · Prodigy.net · Saunalahti Iso G · Swipnet · Telenor · Tripod · University of Michigan personal webpages · Verizon Mysite · Verizon Personal Web Space · Webs · Webzdarma · Virgin Media

Web applications

Mailman · MediaWiki · phpBB · Simple Machines Forum · vBulletin


A Million Ways to Die on the Web · Backup Tips · Cheap storage · Collecting items randomly · Data compression algorithms and tools · Dev · Discovery Data · DOS Floppies · Fortress of Solitude · Keywords · Naughty List · Nightmare Projects · Rescuing floppy disks · Rescuing optical media · Site exploration · The WARC Ecosystem · Working with ARCHIVE.ORG


ArchiveCorps · Audit2014 · Emularity · Faceoff · FlickrFckr · Froogle · INTERNETARCHIVE.BAK (Internet Archive Census· IRC Quotes · JSMESS · JSVLC · Just Solve the Problem · NewsGrabber · Project Newsletter · Valhalla · Web Roasting (ISP Hosting · University Web Hosting· Woohoo


ArchiveBot · ArchiveTeam Warrior (Tracker· Google Takeout · HTTrack · Video downloaders · Wget (Lua · WARC)


Bibliotheca Anonoma · LibreTeam · URLTeam · Yahoo Video Warroom · WikiTeam


800notes · AOL · Akoha · Ancestry.com · April Fools' Day · Amplicate · AutoAdmit · Bre.ad · Circavie · Cobook · Co.mments · Countdown · Discourse · Distill · Dmoz · Easel · Eircode · Electronic Frontier Foundation · FanFiction.Net · Feedly · Ficlets · Forrst · FunnyExam.com · FurAffinity · Google Helpouts · Google Moderator · Google Poly · Google Reader · ICQmail · IFTTT · Jajah · JuniorNet · Lulu Poetry · Mobile Phone Applications · Mochi Media · Mozilla Firefox · MyBlogLog · NBII · Newgrounds · Neopets · Quantcast · Quizilla · Salon Table Talk · Shutdownify · Slidecast · Stack Overflow · SOPA blackout pages · starwars.yahoo.com · TechNet · Toshiba Support · USA-Gov · Volán · Widgetbox · Windows Technical Preview · Wunderlist · YTMND · Zoocasa

About Archive Team

Introduction · Philosophy · Who We Are · Our stance on robots.txt · Why Back Up? · Software · Formats · Storage Media · Recommended Reading · Films and documentaries about archiving · Talks · In The Media · FAQ