ArchiveBot is an IRC bot designed to automate the archival of smaller websites (e.g. up to a few hundred thousand URLs). You give it a URL to start at, and it grabs all content under that URL, records it in a WARC file, and then uploads that WARC to ArchiveTeam servers for eventual injection into the Internet Archive's Wayback Machine (or other archive sites).
To use ArchiveBot, drop by the IRC channel #archivebot on EFNet. To interact with ArchiveBot, you issue commands by typing it into the channel. Note you will need channel operator (
@) or voice (
+) permissions in order to issue archiving jobs; please ask for assistance or leave a message describing the website you want to archive.
The dashboard publicly shows the sites being downloaded currently. The pipeline monitor station shows the status of deployed instances of crawlers. The viewer assists in browsing and searching archives.
- The bot listens for commands in the IRC channel and then reports back status on the IRC channel. You can ask it to archive a whole website or single webpage, check whether the URL has been saved, change the delay time between requests, or add some ignore rules to avoid crawling certain web cruft. This IRC interface is collaborative, meaning anyone with permission can adjust the parameter of jobs. Note that the bot isn't a chat bot so it will ignore you if it doesn't understand a command.
- The ArchiveBot dashboard is a web-based front-end displaying the URLs being downloaded by the various web crawls. Each URL line in the dashboard is categorized by its HTTP code into successes, warnings, and errors. It will be highlighted in yellow or red. the dashboard also provides RSS feeds.
- The backend contains the database of all jobs and several maintenance tasks such as trimming logs and posting Tweets on Twitter. The backend is the centralized portion of ArchiveBot.
- The crawler will download and spider the website into WARC files. The crawler is the distributed portion of ArchiveBot. Volunteers run pipeline nodes connected to the backend. The backend will tell the nodes/pipelines what jobs to run. Once the crawl job has finished, the pipeline reports back to the backend and uploads the WARC files to the staging server. This process is handled by a supervisor script called a pipeline.
- The staging server, known as FOS (Fortress of Solitude), is the place where all the WARC files are temporarily uploaded. Once the current batch has been approved, the files will be uploaded to the Internet Archive for consumption by the Wayback Machine.
The main server that controls the IRC bot, pipeline manager backend, and web dashboard is operated by yipdw, although a few other ArchiveTeam members were given SSH access in late 2017. The staging server FOS (Fortress of Solitude), where the data sits for final checks before being moved over to the Internet Archive serves, is operated by SketchCow. The pipelines are operated by various volunteers around the world. Each pipeline typically runs two or three web crawl jobs at any given time.
Volunteer to run a Pipeline
As of November 2017, ArchiveBot has again started accepting applications from volunteers who want to set up new pipelines. You'll need to have a machine with:
- lots of disk space (40 GB minimum / 200 GB recommended / 500 GB atypical)
- 512 MB RAM (2 GB recommended, 2 GB swap recommended)
- 10 mbps upload/download speeds (100 mbps recommended)
- long-term availability (2 months minimum)
- always-on unrestricted internet access (absolutely no firewall/proxies/censorship/ISP-injected-ads/DNS-redirection/free-cafe-wifi)
Suggestion: the $40/month Digital Ocean droplets (4 GB memory/2 CPU/60 GB hard drive) running Ubuntu work pretty well.
If you have a suitable server available and would like to volunteer, please review the Pipeline Install instructions. Then contact ArchiveTeam members Asparagirl, astrid, JAA, yipdw, or other ArchiveTeam members hanging out in #archivebot, and we can hook you up, adding your machine to the list of approved pipelines, so that it will start processing incoming ArchiveBot jobs.
As of August 2018, there are a few things you need to be aware of when operating an ArchiveBot pipeline:
- Please give access to the pipeline for maintenance work when you're away (e.g. holidays, busy IRL) to someone who's around frequently. This is to avoid situations where jobs or pipelines are stuck for weeks or months without anyone being able to intervene.
- Jobs that crash with an error need to be killed manually using
- The log files of jobs that are aborted or crash are not uploaded to the Internet Archive. Please keep the temporary
tmp-wpull-*.log.gzfiles in the pipeline directory, rename them so the filename follows the same format as the JSON file (with extension
.json), and upload them to FOS manually.
- You can find the job ID for these files in the second line.
- Finding the correct filename can be a bit tricky. You can use the viewer or the archivebot-archives repository. Keep in mind that the timestamp in the filename should approximately match the one at the beginning of the log file, though there is usually a difference between the two of at least a few seconds (the log file timestamps being later than the filename timestamp).
- Be careful with the filename if there were multiple jobs for the same URL (i.e. the same job ID).
- Here is a public gist on GitHub explaining step by step how to find the proper log file for your crashed or killed job, how to properly rename it, and how to rsync it up to FOS: []
- Contact User:JustAnotherArchivist if you need help with this.
- Due to a bug somewhere deep in the network stack, connections get stuck from time to time. This causes jobs to slow down or halt entirely.
- Also due to a bug suspected to be in the network stack, wpull processes sometimes use a lot of RAM (and CPU). If a process uses more than 300 MB continuously, that's likely the case. kill-wpull-connections seems to "fix" this issue, though it takes a while (minutes, rarely even an hour or more) from running the script until the usage actually drops down.
- Make sure that you don't have any
/etc/resolv.conf. We've grabbed a number of copies of the websites of OVH and Online.net as a result of such lines and broken
http://www/links... (Cf this issue on GitHub)
Installing the ArchiveBot can be difficult. The Pipeline Install instructions are online, but are tricky.
Since it's good enough for testing... it's good enough for installation, right? There must be a way to convert it into an installer script.
- Everything is provided on a best-effort basis; nothing is guaranteed to work. (We're volunteers, not a support team.)
- We can decide to stop a job or ban a user if a job is deemed unnecessary. (We don't want to run up operator bandwidth bills and waste Internet Archive donations on costs.)
- We're not Internet Archive. (We do what we want.)
- We're not the Wayback Machine. Specifically, we are not
archive.org_bot. (We don't run crawlers on behalf of other crawlers.)
Occasionally, we had to ban blocks of IP addresses from the channel. If you think a ban does not apply to you but cannot join the #archivebot channel, please join the main #archiveteam channel instead.
If you are a website operator and you notice ArchiveBot misbehaving, please contact us on #archivebot or #archiveteam on EFnet (see top of page for links).
ArchiveBot understands robots.txt (please read the article) but does not match any directives. It uses it for discovering more links such as sitemaps however.
Also, please remember that we are not the Internet Archive.
- Formerly known as @ATArchiveBot