Dev/Infrastructure
The Archive Team infrastructure is a distributed web processing system used for distributed preservation of service attacks.
Component Overview
| Figure | Description |
|---|---|
| 1 | Website in Danger |
| 2 | Warrior |
| 3 | Tracker |
| 4 | Upload Target |
| 5 | Internet Archive |
Website in Danger
The website in danger is typically a website exhibiting combinations of
- acquihire
- mass layoffs
- neglect, decay, unhealthy, or owners missing in action
- political and legal issues
- robots.txt exclusion file that forbids crawling by Wayback Machine (whether intentionally or unintentionally)
- cultural significance
Warrior
The Warrior is client code run by volunteers that grabs/scrapes the content of the website in danger.
Websites often implement throttling systems to protect themselves for various reasons such as spam or server load. Typical systems use IP address bans. As such, many Warriors, running on many IP addresses, are needed.
Content is usually grabbed and saved in WARC files.
Tracker
The Tracker is server code run by "core" Archive Team volunteers. The Tracker assigns what the Warrior should download and provides a leaderboard.
Upload Targets
Targets (sometimes called "staging servers") are typically servers running Rsync run by "core" volunteers. Warriors upload WARC files to these hosts. The hosts queue and package up the WARC files into large WARC files (Megawarcs). Then, the Megawarcs are uploaded to the Internet Archive under the archiveteam collection.
Internet Archive
The Internet Archive is a digital library and archive. It is different from other hosting services because they are not a distribution platform. If there is an legal issue, items are "darked" (made inaccessible to the general public) instead of deleted.
An item is ingested by the Wayback Machine if it
- has .warc, .warc.gz. or .warc.zst files,
- has a "web" media type,
- and is under the Archive Team collection.