Frequently Asked Questions

From Archiveteam
Jump to navigation Jump to search

How can I help?

See Who We Are, Deathwatch, and Category:Projects_status. These pages describe our projects and the things you can do to help.

Is the Archive Team affiliated with the Internet Archive (archive.org)?

No. A few members are affiliated, but majority of Archive Team members are volunteers who help while not busy at work or school.

Why is ArchiveTeam crawling my site / disrespecting robots.txt?

A detailed manifesto is located at Robots.txt. Please read it first and contact us through IRC (described in an answer below) before making harsh actions. We cooperate!

If you notice the crawler's user-agent is "ArchiveBot", please see ArchiveBot.

How should I go about backing things up?

What would you like to back up? If you want to mirror/backup a website, the de facto tool is Wget (but there's lots more, see Software!). WARC files are highly recommended as they can be ingested by the Wayback Machine.

If you want to back up your personal files, "List of backup software" at Wikipedia is an extensive list of backup software. See Backup Tips as well!

Where do all the saved files go?

Files are ultimately uploaded to Internet Archive on the archiveteam collection. Archive Team relies on Internet Archive for storing the files.

How do I access the stuff you archived?

Usually, the content we archived is available in the Wayback Machine, and this is generally the recommended way of accessing it. However, in some cases, this will not work as you might expect. If the obvious plug a URL into the WBM doesn't work, check whether the wiki page for the specific project has more information.

What are these WARC files in the Internet Archive? How do I extract files from a WARC file?

WARC files are de facto medium of digital preservation of the web. These WARC files are ingested by the Wayback Machine. WARC files are not simple zip files; they're designed to record metadata.

There is a growing number of tools that can manipulate WARC files in The WARC Ecosystem.

halp pls halp

I uploaded a WARC file but why doesn't it show up in Wayback Machine?

To ensure content integrity, items with WARC files must have the mediatype set to "web" and be uploaded by a whitelisted Internet Archive account for it to be ingested by the Wayback Machine.

I think there is a web site that's going to shut down / sun set / end its incredible journey. Can you save it?

Yes, do tell us a soon as possible! Don't just Tweet about it; do something. Get things done. Talk to us on Archiveteam:IRC to let us know. There is also a /r/shutdown sub-Reddit.

For small websites, see ArchiveBot. For large websites, the Warrior may need to be deployed. Please take a look at Dev for learning how the Warrior works.

I saved/archived some stuff from a website / ftp server. Do you want it? Where should I upload it?

Great, yes, and https://archive.org! Tag your items with the subject keyword "archiveteam" and let us know so we can move it under the ArchiveTeam collection.

P.S. Creating an account on Internet Archive is free and should be the first thing that comes to mind for archive files. File hosting sites like Putfile, Megaupload, etc. are not suitable for hosting archives!

I lost my stuff from Geocities/Tabblo/Posterous/some web host! Where can I get it back?

Try searching the wiki for a page about the specific website to find out more about what happened. Typically, there are several ways of recovering files from the Internet Archive:

  • A specially crafted username lookup page created by Archive Team
    • Allows you to search by your username and will present the relevant materials. Only a small set of projects have this feature.
  • The Internet Archive's Wayback Machine
    • This method is the easiest for most users but some web pages take months to show up in the Wayback Machine.
  • Individual WARC Files uploaded to the Internet Archive
    • This method is the most accurate but requires power user skills with working with WARC files. As well, WARC files produced by the Internet Archive are not publicly available (but the ones by Archive Team are always available).

For details, see Restoring.

I need help running the Warrior or scripts. I think it's broken.

See the FAQ in the Warrior wiki page.

We Are Not The Internet Archive

How do I upload something to the Internet Archive (archive.org)?

You can either use the HTML5/Flash uploader or use an alternative method.

Can someone remove or fix something on the Internet Archive?

Possibly. Keep in mind that majority of Archive Team are volunteers who are not affiliated with the Internet Archive and requests should go to staff instead. If strenuous circumstances arise, please see the question about contacting Archive Team below.

How do I get the original file on the Wayback Machine?

Add id_ after the date in the URL. For example: web.archive.org/web/20090119040418id_/http://www.archiveteam.org/index.php?title=Main_Page

How do I search the contents of the Wayback Machine?

You can't unfortunately. However, the Internet Archive provides API access (designed for programmers and power users) to the Wayback Machine and to the CDX database.

Why does the Wayback Machine follow robots.txt in a way that I don't like?

Because it makes the lawyers go away. We're not the Internet Archive. Don't ask us, ask them.

Can I upload $COPYRIGHTED_THING to the Internet Archive?

Although the Internet Archive prefers freely-redistributable content, they also accept still-in-copyright things. If there's a valid complaint/DMCA takedown request, they'll simply make the item private, but they will not delete the data. Having said that, the Internet Archive is not The Pirate Bay, so please don't treat it as such.

Can I save big files with the Save Now feature?

No, files larger than 200 MB will not be saved correctly.

How redundant is Archiveteam?

Is there a backup of the data on the archiveteam.org website? If so where can I download it?

Two sets of backups of this wiki are available. There are backups done by the hosting provider (several, going back days and weeks as well as hours), which use the storage capability of the shared hosting to keep them automatically (no tape or disk backups being done as most people would think of them). There are similarly copies of the database kept going back months.

Additionally, an XML dump of the Mediawiki database (which can be imported into any MediaWiki software) is accessible at https://www.archiveteam.org/dumps. New backups are currently pushed out once a week (and will be increased if changes on the site require it). Our entire images directory is available at https://www.archiveteam.org/images.

Dumps of the ArchiveTeam Wiki are generated with WikiTeam tools and uploaded to the Internet Archive quite regularly. You can find them here and older ones here.

Is there a mirror of the archiveteam.org website?

There are no mirrors we know of, although we encourage our more paranoid or protective readers to maintain one based on the above dumps.

Who are y'all?

Does Archive Team have any social media accounts?

Follow us on Twitter: @archiveteam, @at_warrior, /r/archiveteam and like us on Facebook. These accounts are run by selected volunteers and may not be monitored for questions.

(There is a Google ArchiveTeam group but it is not used.)

Who's the administrator?

For a list of administrators, see Tracker#People which has a table at the bottom of the page.

I went through the wiki and I still have a question! How do I contact the Archive Team?

Join us on IRC! For general inquiries, visit #archiveteam on hackint. Email can be sent to archiveteam@archiveteam.org.

This wiki is not monitored for questions.

If a FAQ should appear here, please add it.

Notes



v · t · e         Archive Team
Current events

Alive... OR ARE THEY · Deathwatch · Projects

Archiveteam.jpg
Archiving projects

APKMirror · Archive.is · BetaArchive · Government Backup (#datarefuge · ftp-gov· Gmane · Internet Archive · It Died · Megalodon.jp · OldApps.com · OldVersion.com · OSBetaArchive · TEXTFILES.COM · The Dead, the Dying & The Damned · The Mail Archive · UK Web Archive · WebCite · Vaporwave.me

Blogging

Blog.pl · Blogger · Blogster · Blogter.hu · Freeblog.hu · Fuelmyblog · Jux · LINE BLOG · LiveJournal · My Opera · Nolblog.hu · Open Diary · ownlog.com · Posterous · Powerblogs · Proust · Roon · Splinder · Tumblr · Vox · Weblog.nl · Windows Live Spaces · Wordpress.com · Xanga · Yahoo! Blog · Zapd

Cloud hosting/file sharing

aDrive · AnyHub · Box · Dropbox · Docstoc · Fast.io · Google Drive · Google Groups Files · iCloud · Fileplanet · LayerVault · MediaCrush · MediaFire · Mega · MegaUpload · MobileMe · OneDrive · Pomf.se · RapidShare · Ubuntu One · Yahoo! Briefcase

Corporations

Apple · IBM · Google · Loblaw · Lycos Europe · Microsoft · Yahoo!

Events

Arab Spring · Great Ape-Snake War · Spanish Revolution

Font Repos

DaFont · Google Web Fonts · GNU FreeFont · Fontspace

Forums/Message boards

4chan · Captain Luffy Forums · College Confidential · Discourse · DSLReports · ESPN Forums · Facepunch Forums · forums.starwars.com · HeavenGames · JamiiForums · Invisionfree · NeoGAF · Textream · The Classic Horror Film Board · Yahoo! Messages · Yahoo! Neighbors · Yuku.com · Zetaboards

Gaming

Atomicgamer · Bazaar.tf · City of Heroes · Club Nintendo · Clutch · Counter-Strike: Global Offensive · CS:GO Lounge · Desura · Dota 2 · Dota 2 Lounge · Emulation Zone · ESEA · GameBanana · GameMaker Sandbox · GameTrailers · Halo · Heroes of Newerth · HLTV.org · HQ Trivia · Infinite Crisis · joinDOTA · League of Legends · Liquipedia · Minecraft.net · Player.me · Playfire · Raptr · SingStar · Steam · SteamDB · SteamGridDB · Team Fortress 2 · TF2 Outpost · Warhammer · Xfire

Image hosting

500px · AOL Pictures · Blipfoto · Blingee · Canv.as · Camera+ · Cameroid · DailyBooth · Degree Confluence Project · DeviantART · Demotivalo.net · Flickr · Fotoalbum.hu · Fotolog.com · Fotopedia · Frontback · Geograph Britain and Ireland · Giphy · GTF Képhost · ImageShack · Imgh.us · Imgur · Inkblazers · Instagram · Kepfeltoltes.hu · Kephost.com · Kephost.hu · Kepkezelo.com · Keptarad.hu · Madden GIFERATOR · MLKSHK · Microsoft Clip Art · Microsoft Photosynth · Nokia Memories · noob.hu · Odysee · Panoramio · Photobucket · Picasa · Picplz · Pixiv · Portalgraphics.net · PSharing · Ptch · puu.sh · Rawporter · Relay.im · ScreenshotsDatabase.com · Sketch · Smack Jeeves · Snapjoy · Streetfiles · Tabblo · Tinypic · Trovebox · TwitPic · Wallbase · Wallhaven · Webshots · Wikimedia Commons

Knowledge/Wikis

arXiv · Citizendium · Clipboard.com · Deletionpedia · EditThis · Encyclopedia Dramatica · Etherpad · Everything2 · infoAnarchy · GeoNames · GNUPedia · Google Books (Google Books Ngram· Horror Movie Database · Insurgency Wiki · Knol · Lost Media Wiki · Neoseeker.com · Notepad.cc · Nupedia · OpenCourseWare · OpenStreetMap · Orain · Pastebin · Patch.com · Project Gutenberg · Puella Magi · Referata · Resedagboken · SongMeanings · ShoutWiki · The Internet Movie Database · TropicalWikis · Uncyclopedia · Urban Dictionary · Urban Exploration Resource · Webmonkey · Wikia · Wikidot · WikiHow · Wikkii · WikiLeaks · Wikipedia (Simple English Wikipedia· Wikispaces · Wikispot · Wik.is · Wiki-Site · WikiTravel · Word Count Journal

Magazines/Blogs/News

Cyberpunkreview.com · Game Developer Magazine · Gigaom · Hardware Canucks · Helium · JPG Magazine · Make Magazine · The Escapist · Polygamia.pl · San Fransisco Bay Guardian · Scoop · Regretsy · Yahoo! Voices

Microblogging

Heello · Identi.ca · Jaiku · Mommo.hu · Plurk · Sina Weibo · Tencent Weibo · Twitter · TwitLonger

Music/Audio

8tracks · AOL Music · Audimated.com · Cinch · digCCmixter · Dogmazic.net · Earbits · exfm · Free Music Archive · Gogoyoko · Indaba Music · Instacast · Instaudio · Jamendo · Last.fm · Music Unlimited · MOG · PureVolume · Reverbnation · ShareTheMusic · SoundCloud · Soundpedia · Spotify · This Is My Jam · TuneWiki · Twaud.io · WinAmp

People

Aaron Swartz · Michael S. Hart · Steve Jobs · Mark Pilgrim · Dennis Ritchie · Len Sassaman Project

Protocols/Infrastructure

FTP · Gopher · IRC · Usenet · World Wide Web
BitTorrent DHT

Q&A

Askville · Answerbag · Answers.com · Ask.com · Askalo · Baidu Knows · Blurtit · ChaCha · Experts Exchange · Formspring · GirlsAskGuys · Google Answers · Google Baraza · JustAnswer · MetaFilter · Quora · Retrospring · StackExchange · The AnswerBank · The Internet Oracle · Uclue · WikiAnswers · Yahoo! Answers

Recipes/Food

Allrecipes · Epicurious · Food.com · Foodily · Food Network · Punchfork · ZipList

Social bookmarking

Addinto · Backflip · Balatarin · BibSonomy · Bkmrx · Blinklist · BlogMarks · BookmarkSync · CiteULike · Connotea · Delicious · Designer News · Digg · Diigo · Dir.eccion.es · Evernote · Excite Bookmark · Faves · Favilous · folkd · Freelish · Getboo · GiveALink.org · Gnolia · Google Bookmarks · Hacker News · HeyStaks · IndianPad · Kippt · Knowledge Plaza · Licorize · Linkwad · Menéame · Microsoft Developer Network · myVIP · Mister Wong · My Web · Mylink Vault · Newsvine · Oneview · Pearltrees · Pinboard · Pocket · Propeller.com · Reddit · sabros.us · Scloog · Scuttle · Simpy · SiteBar · Slashdot · Squidoo · StumbleUpon · Twine · Voat · Vizited · Yummymarks · Xmarks · Yahoo! Buzz · Zootool · Zotero

Social networks

Bebo · BlackPlanet · Classmates.com · Cyworld · Dogster · Dopplr · douban · Ello · Facebook · Flixster · FriendFeed · Friendster · Friends Reunited · Gaia Online · Google+ · Habbo · hi5 · Hyves · iWiW · LinkedIn · Miiverse · mixi · MyHeritage · MyLife · Myspace · myVIP · Netlog · Odnoklassniki · Orkut · Plaxo · Qzone · Renren · Skyrock · Sonico.com · Storylane · Tagged · tvtag · Upcoming · Viadeo · Vine · VK · WeeWorld · Weibo · Wretch · Xuite · Yahoo! Groups · Yahoo! Stars India · Yahoo! Upcoming · more sites...

Shopping/Retail

Alibaba · AliExpress · Amazon · Apple Store · Barnes & Noble · DirectCanada · eBay · Kmart · NCIX · Printfection · RadioShack · Sears · Sears Canada · Target · The Book Depository · ThinkGeek · Toys "R" Us · Walmart

Software/code hosting

Android Development · Alioth · Assembla · BerliOS · Betavine · Bitbucket · BountySource · Codecademy · CodePlex · Freepository · Free Software Foundation · GNU Savannah · GitHost  · GitHub · GitHub Downloads · Gitorious · Gna! · Google Code · ibiblio · java.net · JavaForge · KnowledgeForge · Launchpad · LuaForge · Maemo · mozdev · OSOR.eu · OW2 Consortium · Openmoko · OpenSolaris · Ourproject.org · Ovi Store · Project Kenai · RubyForge · SEUL.org · SourceForge · Stypi · TestFlight · tigris.org · Transifex · TuxFamily · Yahoo! Downloads

Television/Radio

ABC · Austin City Limits · BBC · CBC · CBS · Computer Chronicles · CTV · Fox · G4 · Global TV · Jeopardy! · NBC · NHK · PBS · Penn & Teller: Bullshit! · The Howard Stern Show · TV News Archive (Understanding 9/11)

Torrenting/Piracy

ExtraTorrent · EZTV · isoHunt · KickassTorrents · The Pirate Bay · Torrentz · Library Genesis

Video hosting

Academic Earth · Bambuser · Blip.tv · Epic · Freshlive · Google Video · Justin.tv · Mixer · Niconico · Nokia Trailers · Oddshot.tv · Periscope · Plays.tv · Qwiki · Skillfeed · Stickam · TED Talks · Ticker.tv · Twitch.tv · Ustream · Videoplayer.hu · Viddler · Viddy · Vidme · Vimeo · Vine · Vstreamers · Yahoo! Video · YouTube · Famous Internet videos (Me at the zoo)

Web hosting

Angelfire · Brace.io · BT Internet · CableAmerica Personal Web Space · Claranet Netherlands Personal Web Pages · Comcast Personal Web Pages · Extra.hu · FortuneCity · Free ProHosting · GeoCities (patch· Google Business Sitebuilder · Google Sites · Internet Centrum · MBinternet · MSN TV · Nifty · Nwnyet · Parodius Networking · Prodigy.net · Saunalahti Iso G · Swipnet · Telenor · Tripod · University of Michigan personal webpages · Verizon Mysite · Verizon Personal Web Space · Webs · Webzdarma · Virgin Media

Web applications

Mailman · MediaWiki · phpBB · Simple Machines Forum · vBulletin

Information

A Million Ways to Die on the Web · Backup Tips · Cheap storage · Collecting items randomly · Data compression algorithms and tools · Dev · Discovery Data · DOS Floppies · Fortress of Solitude · Keywords · Naughty List · Nightmare Projects · Rescuing floppy disks · Rescuing optical media · Site exploration · The WARC Ecosystem · Working with ARCHIVE.ORG

Projects

ArchiveCorps · Audit2014 · Emularity · Faceoff · FlickrFckr · Froogle · INTERNETARCHIVE.BAK (Internet Archive Census· IRC Quotes · JSMESS · JSVLC · Just Solve the Problem · NewsGrabber · Project Newsletter · Valhalla · Web Roasting (ISP Hosting · University Web Hosting· Woohoo

Tools

ArchiveBot · ArchiveTeam Warrior (Tracker· Google Takeout · HTTrack · Video downloaders · Wget (Lua · WARC)

Teams

Bibliotheca Anonoma · LibreTeam · URLTeam · Yahoo Video Warroom · WikiTeam

Other

800notes · AOL · Akoha · Ancestry.com · April Fools' Day · Amplicate · AutoAdmit · Bre.ad · Circavie · Cobook · Co.mments · Countdown · Discourse · Distill · Dmoz · Easel · Eircode · Electronic Frontier Foundation · FanFiction.Net · Feedly · Ficlets · Forrst · FunnyExam.com · FurAffinity · Google Helpouts · Google Moderator · Google Poly · Google Reader · ICQmail · IFTTT · Jajah · JuniorNet · Lulu Poetry · Mobile Phone Applications · Mochi Media · Mozilla Firefox · MyBlogLog · NBII · Newgrounds · Neopets · Quantcast · Quizilla · Salon Table Talk · Shutdownify · Slidecast · Stack Overflow · SOPA blackout pages · starwars.yahoo.com · TechNet · Toshiba Support · USA-Gov · Volán · Widgetbox · Windows Technical Preview · Wunderlist · YTMND · Zoocasa

About Archive Team

Introduction · Philosophy · Who We Are · Our stance on robots.txt · Why Back Up? · Software · Formats · Storage Media · Recommended Reading · Films and documentaries about archiving · Talks · In The Media · FAQ