Frequently Asked Questions
I think there is a web site that's going to shut down / sun set / end its incredible journey. Can you save it?
Yes, do tell us a soon as possible! Talk to us on our IRC chat to let us know.
For small websites, we (which can include you if you stick around!) will use ArchiveBot. For large websites, the Warrior may need to be deployed. Please take a look at Dev for learning how the Warrior works.
How can I help?
See Who We Are, Deathwatch, Category:Projects_status, and Donate. These pages describe our projects and the things you can do to help.
Is the Archive Team affiliated with the Internet Archive (archive.org)?
No. A few members are affiliated, but majority of Archive Team members are volunteers who help while not busy at work or school.
Why is ArchiveTeam crawling my site / disrespecting robots.txt?
A detailed manifesto is located at Robots.txt. Please read it first and contact us through IRC (described in an answer below) before making harsh actions. We cooperate!
If you notice the crawler's user-agent is "ArchiveBot", please see ArchiveBot.
How should I go about backing things up?
What would you like to back up? If you want to mirror/backup a website, the de facto tool is Wget (but there's lots more, see Software!), which produces WARC (Web ARChive) files. If the website makes sense as a public archive/in the Wayback Machine please contact us so that we can do so; a WARC you produce yourself will likely not end up in the Wayback Machine.
If you want to back up your personal files, "List of backup software" at Wikipedia is an extensive list of backup software. See Backup Tips as well!
Where do all the saved files go?
Files are ultimately uploaded to Internet Archive on the archiveteam collection. Archive Team relies on Internet Archive for storing the files.
How do I access the stuff you archived?
Usually, the content we archived is available in the Wayback Machine, and this is generally the recommended way of accessing it. However, in some cases, this will not work as you might expect. If the obvious plug a URL into the WBM doesn't work, check whether the wiki page for the specific project has more information.
I'm looking on ArchiveTeam's uploads to the Internet Archive, and I see a lot of files. Which ones do I need?
As stated above, if you just want an individual page, it is usually available in the Wayback Machine. However, if you do want to look through the archived data manually, the two formats you will likely want are the warc.gz or warc.zst files (see the next section for how to use them), which contain the actual page content, and the cdx or cdx.gz files, which contain the list of archived URLs and some information about them. (If it ends in gz, you will have to decompress it with gzip first.)
What are these WARC files in the Internet Archive? How do I extract files from a WARC file?
WARC files are de facto medium of digital preservation of the web. These WARC files are ingested by the Wayback Machine. WARC files are not simple zip files; they're designed to record metadata.
There is a growing number of tools that can manipulate WARC files in The WARC Ecosystem.
Why is my warrior uploading less than it's downloading?
Data is compressed before it is uploaded. The ratio depends on the project, but the upload should virtually always be less than the download, sometimes significantly so.
Media heavy sites will typically be more equal, while text is compressed efficiently.
halp pls halp
I uploaded a WARC file but why doesn't it show up in Wayback Machine?
To ensure content integrity, items with WARC files must have the mediatype set to "web" and be in a whitelisted collection to appear in the Wayback Machine. In general, unless you are already known to the Internet Archive before you create the WARC, you cannot upload to a whitelisted collection.
I saved/archived some stuff from a website / ftp server. Do you want it? Where should I upload it?
Great, yes, and https://archive.org! Tag your items with the subject keyword "archiveteam" and let us know.
P.S. Creating an account on Internet Archive is free and should be the first thing that comes to mind for archive files. File hosting sites like Putfile, Megaupload, etc. are not suitable for hosting archives!
I lost my stuff from Geocities/Tabblo/Posterous/some web host! Where can I get it back?
Try searching the wiki for a page about the specific website to find out more about what happened. Typically, there are several ways of recovering files from the Internet Archive:
- A specially crafted username lookup page created by Archive Team
- Allows you to search by your username and will present the relevant materials. Only a small set of projects have this feature.
- The Internet Archive's Wayback Machine
- This method is the easiest for most users but some web pages take months to show up in the Wayback Machine.
- Individual WARC Files uploaded to the Internet Archive
- This method is the most accurate but requires power user skills with working with WARC files. As well, WARC files produced by the Internet Archive are not publicly available (but the ones by Archive Team are always available).
For details, see Restoring.
I need help running the Warrior or scripts. I think it's broken.
See the FAQ in the Warrior wiki page.
We Are Not The Internet Archive
How do I upload something to the Internet Archive (archive.org)?
You can either use the HTML5/Flash uploader or use an alternative method.
Can someone remove or fix something on the Internet Archive?
Possibly. Keep in mind that majority of Archive Team are volunteers who are not affiliated with the Internet Archive and requests should go to staff instead. If strenuous circumstances arise, please see the question about contacting Archive Team below.
How do I get the original file on the Wayback Machine?
Add id_ after the date in the URL. For example: web.archive.org/web/20090119040418id_/http://www.archiveteam.org/index.php?title=Main_Page
How do I search the contents of the Wayback Machine?
You can't unfortunately. However, the Internet Archive provides API access (designed for programmers and power users) to the Wayback Machine and to the CDX database.
Why does the Wayback Machine follow robots.txt in a way that I don't like?
Because it makes the lawyers go away. We're not the Internet Archive. Don't ask us, ask them.
Can I upload $COPYRIGHTED_THING to the Internet Archive?
Although the Internet Archive prefers freely-redistributable content, they also accept still-in-copyright things. If there's a valid complaint/DMCA takedown request, they'll simply make the item private, but they will not delete the data. Having said that, the Internet Archive is not The Pirate Bay, so please don't treat it as such.
Can I save big files with the Save Now feature?
No, as of 2024 files larger than 2 GiB will not be saved correctly.
How redundant is Archiveteam?
Is there a backup of the data on the archiveteam.org website? If so where can I download it?
Two sets of backups of this wiki are available. There are backups done by the hosting provider (several, going back days and weeks as well as hours), which use the storage capability of the shared hosting to keep them automatically (no tape or disk backups being done as most people would think of them). There are similarly copies of the database kept going back months.
Additionally, an XML dump of the Mediawiki database (which can be imported into any MediaWiki software) is accessible at https://www.archiveteam.org/dumps. New backups are currently pushed out once a week (and will be increased if changes on the site require it). Our entire images directory is available at https://www.archiveteam.org/images.
Dumps of the ArchiveTeam Wiki are generated with WikiTeam tools and uploaded to the Internet Archive quite regularly. You can find them here and older ones here.
Is there a mirror of the archiveteam.org website?
There are no mirrors we know of, although we encourage our more paranoid or protective readers to maintain one based on the above dumps.
Who are y'all?
Does Archive Team have any social media accounts?
Follow us on Twitter: @archiveteam, @at_warrior, /r/archiveteam and like us on Facebook. These accounts are run by selected volunteers and may not be monitored for questions.
(There is a Google ArchiveTeam group but it is not used.)
Who's the administrator?
There are several, depending what exact infrastructure you're talking about. See the next section if you need to get in touch.
I went through the wiki and I still have a question! How do I contact the Archive Team?
Join us on IRC! For general inquiries, visit #archiveteam-bs (on hackint). Email can be sent to archiveteam@archiveteam.org.
This wiki is not monitored for questions.
If a FAQ should appear here, please add it.
Notes