The Archive Team infrastructure is a distributed web processing system used for distributed preservation of service attacks.
|1||Website in Danger|
Website in Danger
The website in danger is typically a website exhibiting combinations of
- mass layoffs
- neglect, decay, unhealthy, or owners missing in action
- political and legal issues
- robots.txt exclusion file that forbids crawling by Wayback Machine (whether intentionally or unintentionally)
- cultural significance
The Warrior is client code run by volunteers that grabs/scrapes the content of the website in danger.
Websites often implement throttling systems to protect themselves for various reasons such as spam or server load. Typical systems use IP address bans. As such, many Warriors, running on many IP addresses, are needed.
Content is usually grabbed and saved in WARC files.
The Tracker is server code run by "core" Archive Team volunteers. The Tracker assigns what the Warrior should download and provides a leaderboard.
Staging servers are typically servers running Rsync often run by "core" volunteers. Warriors upload WARC files to these hosts. The hosts queue and package up the WARC files into large WARC files (Megawarcs). Then, the Megawarcs are uploaded to the Internet Archive under the Archive Team collection.
The Internet Archive is a digital library and archive. It is different from other hosting services because they are not a distribution platform. If there is an legal issue, items are "darked" (made inaccessible to the general public) instead of deleted.
Items are ingested by the Wayback Machine if it
- has warc.gz files,
- has a "web" media type,
- and is under the Archive Team collection.
Since around 2015-2016 (I'm not 100% sure of the timeline), to prevent abuse, the Wayback Machine only ingests WARCs from whitelisted users. Official Archive Team crawls/tools almost always go into the Wayback Machine.