Running Archive Team Projects with Docker
You can run Archive Team scripts in Docker containers to help with our archiving efforts. It will download sites and upload them to our archive — and it’s really easy to do!
The scripts run in a Docker container, so there is no risk to your computer. The container will only use your bandwidth and some of your disk space. It will get tasks from and report progress to the Tracker.
Docker runs on Windows, macOS, and Linux, and is a free download.
Instructions for using Docker CLI on Windows, macOS, or Linux
- Download and install Docker from the link above.
- Open your terminal. On Windows, you can use either Command Prompt (CMD) or PowerShell, on macOS and Linux you can use Terminal (Bash).
- First, we will start the Watchtower container. Watchtower automatically checks for updates to Docker containers every five minutes, and if an update is found, it will gracefully shutdown your container, update it, and restart it.
Use the following command:
docker run -d --name watchtower -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --label-enable.
Why a Docker container in the first place?
A Docker container is a quick, safe, and easy way for newcomers to help us out. It offers many features:
- Optional self-updating software infrastructure
- Allows for unattended use
- In case of software faults, your machine is not ruined
- Restarts itself in case of runaway programs
- Runs on Windows, macOS, and Linux painlessly
- Ensures consistency in the archived data regardless of your machine's quirks
If you have suggestions for improving this system, please talk to us as described below.
Can I use whatever internet access for running scripts?
No. We need "clean" connections. Please ensure the following:
- No OpenDNS. No ISP DNS that redirects to a search page. Use non-captive DNS servers.
- No ISP connections that inject advertisements into web pages.
- No proxies. Proxies can return bad data. The original HTTP headers and IP address are needed for the WARC file.
- No content-filtering firewalls.
- No censorship. If you believe your country implements censorship, do not run a warrior.
- No Tor. The server may return an error page instead of content if they ban exit nodes.
- No free cafe wifi. Archiving your cafe's wifi service agreement repeatedly is not helpful.
- No VPNs. Data integrity is a very high priority for the Archive Team so use of VPNs with the official crawler is discouraged.
- We prefer connections from many public IP addresses if possible. (For example, if your apartment building uses a single IP address, we don't want your apartment banned.)
I turned my Docker container off. Will those tasks be lost?
If you've killed your Docker instance, then the work your container did has been lost. However, the tasks will be returned to the pool after a period of time, and others may claim them. If you want, you can alert the admins via IRC of what's happened and they can clear the claims your username may have made. but this isn't very important on most projects.
How much disk space will the Docker container use?
Short answer: it depends on the project.
Long answer: because each project defines items differently, sizes may vary. A single task may be a small file or a whole subsection of a website.
How can I run the Docker container headlessly (without leaving a window open)?
(add startup and shutdown instructions)
How can I run tons of containers easily?
We assume you've checked with the current ArchiveTeam project what concurrency and resources are needed or useful!
ArchiveTeam volunteers have successfully used a variety of hosting providers and tools (including free trials on AWS and GCE), often just by building their own flavour of virtual server and then repeating it with simple cloud-init scripts (to install and launch docker as above) or whatever tool the hosting provides. If you desire full automation, the archiveteam-infra repository by diggan helps with Terraform on DigitalOcean.
Some custom monitoring scripts also exist, for instance watcher.py.
I'd like to help write code or I want to tweak the scripts to run to my liking. Where can I find more info? Where is the source code and repository?
Check out the Dev documentation for details on the infrastructure and details of the source code layout.
I still have a question!
I see a message that no item was received.
This means that there is no work available. This can happen for several reasons:
- The project has just finished and someone is inspecting the work done. If a problem is discovered, items may be re-queued and more work will become available.
- You have checked out/claimed too many items. Reduce your concurrency and let others do some of the work too.
- In a rare case, you have been banned by a tracker administrator because there was a problem with your work: you were requesting too much, you were tampering with the scripts, a malfunction has occurred, or your internet connection is "unclean" (see above).
I see a message about rate limiting.
Don't worry. Keep in mind that although downloading the internet for fun and digital preservation are the primary goals of all Archive Team activities, serious stress on the target's server may occur. The rate limit is imposed by a tracker administrator and should not be subverted.
(In other words, we don't want to DDoS the servers.)
If you like, you can switch to another project with less load.
I see a message about code being out of date.
Don't worry. There is a new update ready. You do not need to do anything about this if you are running the container with Watchtower; Watchtower will update its code every five minutes. If you are impatient, please (insert manual update instructions) and it will download the latest code and resume work.
I'm running the scripts manually and I see a message about code being out of date.
This happens when a bug in the scripts is discovered. Bugs are unavoidable, especially when the server is out of our control.
I see messages about rsync errors.
Uh-oh! Something is not right. Please notify us immediately in the appropriate IRC channel.
The container is eating all my bandwidth!
(insert bandwidth limit instructions)
The item I'm working on is downloading thousands of URLs and it's taking hours.
Please notify us in the appropriate IRC channel. You may need to restart the container.
The instructions to run the software/scripts are awful and they are difficult to set up.
Well, excuuuuse me, princess!
Where can I file a bug, suggestion, or a feature request?
Are you a coder?
Like our scripts? Interested in how it works under the hood? Got software skills? Help us improve it!