From Archiveteam
< Dev
Revision as of 17:26, 16 September 2013 by Chfoo (talk | contribs) (→‎The Megawarc Factory: fix relative paths)
Jump to navigation Jump to search

This article describes how to set up your own tracker and projects.

You don't need to be a computer scientist to set up your own tracker, but you will need to be comfortable with:

  • Environment:
    • Ubuntu/Debian
    • running scripts
    • compiling programs
  • Languages:
    • Python
    • Ruby
    • JavaScript
    • Lua
    • Shell scripting like Bash
  • Web:
    • Nginx
    • Phusion Passenger
    • Rails
    • Redis
    • Node.js
  • Tools:
    • Screen
    • Rsync
    • Git
    • Wget
    • regular expressions

There are 3 components to the tracker and project system:

  1. The Seesaw client
  2. The Rsync server
  3. The Tracker server

Writing a Seesaw Client

The Seesaw client is a specific set of tasks that must be done within an item. Think of it as a template of instructions. Typically, the file is called The pipeline file uses the Seesaw Library.

The pipeline file will typically use Wget with Lua scripting. The Lua script provided as an argument to Wget within the pipeline file. It controls fine grain operations within Wget such as rejecting unneeded URLs or adding more URLs as they are discovered.

Take a look at the grab scripts in recent Archive Team repositories for examples of clients.


You will need:

  • Python 2.7
  • Lua
  • Wget with Lua hooks

Typically, you can install these by running:

sudo apt-get install build-essential lua5.1 liblua5.1-0-dev python python-setuptools python-dev openssl libssl-dev python-pip make
sudo pip install seesaw

You will also need Wget with Lua. Look into recent repositories for the following script and run it:


The pipeline file

The pipeline file typically includes:

  • Copy and pasted monkey patches
  • A routine to find Wget Lua
  • A version number in the form of YYYYMMDD.NN
  • Tracker hostname
  • Custom Tasks:
    • PrepareDirectories
    • MoveFiles
  • Project information saved into the project variable
  • Instructions on how to deal with the item saved into the pipeline variable
  • An undeclared downloader variable which will be filled in by the Seesaw library

It is important to remember that each Task is a template on how to deal with each Item. Specific item variables should not be stored on a Task, but rather, it should be saved onto the item.

To run a pipeline file, run the command:

run-pipeline YOUR_NICKNAME

Setup the Rsync target

The Rsync target consists of disk space, Rsync, and WARC packing scripts in a dedicated user account.

Create the system user account dedicated for the Rsync target:

sudo adduser --system --group --shell /bin/bash archiveteam

Log in as archiveteam:

sudo -u archiveteam -i

Create a place to store the uploads:

mkdir -p PROJECT_NAME/incoming-uploads/

You may log out of archiveteam at this point.


You will need to install Rsync:

sudo apt-get install rsync

Once rsync is installed, you will need to edit the rsync configuration file. If no rsyncd.conf exists in /etc, copy it from /usr/share/doc/rsync/examples/rsyncd.conf

Rsync uses a concept of "modules" which can be considered as namespaces. If you have copied the example file, you can modify the example ftp module to fit your new project. Perhaps you may call the module after the project name.

You will also need to include:

  • path = /home/archiveteam/PROJECT_NAME/incoming-uploads/
  • read only = no
  • uid = archiveteam
  • gid = archiveteam

Make Rsync start up as daemon on boot up by editing /etc/default/rsync. Ensure it reads


Start up Rsync deamon:

sudo invoke-rc.d rsync start

The Megawarc Factory

The Megawarc Factory are scripts that package and bundle up all the uploaded WARC files that is received.

If Git, Curl, or Screen is not yet installed, install it now:

sudo apt-get install git curl screen

Log in as archiveteam and download the scripts needed:

git clone
cd archiveteam-megawarc-factory/
git clone

Let's begin to populate the configuration file:

cp archiveteam-megawarc-factory/ PROJECT_NAME/

Going through the

  • MEGABYTES_PER_CHUNK denotes how big the mega WARC files. Typically it should be set at 50GB, but if you really don't have the space, you can use smaller files like 10GB.
  • IA_AUTH is your Internet Archive S3-like API authentication keys.
  • IA_COLLECTION, IA_ITEM_TITLE, IA_ITEM_PREFIX, FILE_PREFIX all should have the todos replaced with the project name.
  • FS1_BASE_DIR should be set to /home/archiveteam/PROJECT_NAME/
  • FS2_BASE_DIR should be set to same as above or another location.
  • COMPLETED_DIR should be left empty (i.e., "") if the uploaded file is to be deleted.

Bother or ask politely someone about getting permission to upload your files to the collection archiveteam_PROJECT_NAME. You can ask on #archiveteam on EFNet.

Let's run the Megawarc Factory. First, create a sentinel file:

touch RUN

You can run the Megawarc Factory in Screen. The 3 scripts will on separate command shells within one Screen session:

ionice -c 2 -n 6 nice -n 19 ../archiveteam-megawarc-factory/pack-multiple

Here's a few Screen pointers:

  • screen -r will resume an existing screen session
  • CTRL+A c creates a new command window
  • CTRL+A SPACE switches to the next window
  • CTRL+A " shows you a list of windows
  • CTRL+A d leaves, or detaches, the screen session

To stop the Megawarc Factory, remove the sentinel file:

rm RUN

You can log out of the archiveteam account now.

The Tracker

The Tracker manages what items are claimed by users that run the Seesaw client. It also shows a pretty leaderboard.

Let's create a dedicated account to run the web server and tracker:

sudo adduser --system --group --shell /bin/bash tracker


Redis is database stored in memory. So, engineer your item names so you do not have to load many items at once. Redis saves its database periodicly into a file located at /var/lib/redis/6379/dump.rdb. It is safe to copy the file, e.g., for backups.

To install Redis, you may follow these quickstart instructions, but we'll show you how.

These steps are from the quickstart guide:

tar xvzf redis-stable.tar.gz
cd redis-stable

Now install the server:

sudo make install
cd utils
sudo ./

Note, by default, it runs as root. Let's stop it and make it run under www-data:

sudo invoke-rc.d redis_6379 stop
sudo adduser --system --group www-data
sudo chown -R www-data:www-data /var/lib/redis/6379/
sudo chown -R www-data:www-data /var/log/redis_6379.log

Edit the config file /etc/redis/6379.conf with the options like:

pidfile /var/run/shm/

Now tell the start up script to run it as www-data:

sudo nano /etc/init.d/redis_6379

Change the EXEC and CLIEXEC variables to use sudo -u www-data -g www-data:

EXEC="sudo -u www-data -g www-data /usr/local/bin/redis-server"
CLIEXEC="sudo -u www-data -g www-data /usr/local/bin/redis-cli"

To avoid catastrophe with background saves failing on fork() (Redis needs lots of memory), run:

sudo sysctl vm.overcommit_memory=1

The above setting will be lost after reboot. Add this line to /etc/sysctl.conf:


The log file will get big so we need a logrotate config. Create one at /etc/logrotate.d/redis with the config:

/var/log/redis_*.log {
      rotate 10
      size 10M

Start up Redis again using:

sudo invoke-rc.d redis_6379 start

Nginx with Passenger

Nginx is a web server. Phusion Passenger is a module within Nginx that runs Rails applications.

There is a guide on how to install Nginx with Passenger, the following instructions are similar.

Log in as tracker:

sudo -u tracker -i

We'll use RVM to install Ruby libraries:

curl -L | bash -s stable
source ~/.rvm/scripts/rvm
rvm requirements

A list of things needed to be installed will be shown. Log out of the tracker account, install them, and log back into the tracker account.

Install Ruby and Rails:

rvm install 2.0
rvm rubygems current
gem install rails
gem install bundle

Install Passenger:

gem install passenger

Install Nginx. This command will download, compile, and install a basic Nginx server.:


Use the following prefix for Nginx installation:


Change the location of the tracker software (to be installed later). Edit nginx/conf/nginx.conf. Use the lines under the "location /" option:

root /home/tracker/universal-tracker/public;
passenger_enabled on;
client_max_body_size 15M;

The logs will get big so we'll use logrotate. Save this into /home/tracker/logrotate.conf:

/home/tracker/nginx/logs/access.log {
     rotate 10
     size 10M

To call logrotate, we'll add an entry using crontab:

crontab -e

Now add the following line:

@daily /usr/sbin/logrotate --state /home/tracker/.logrotate.state /home/tracker/logrotate.conf

Log out of the tracker account at this point.

Let's create an Upstart configuration file to start up Nginx. Save this into /etc/init/ngixn-tracker.conf:

description "nginx http daemon"

start on runlevel [2]
stop on runlevel [016]

setuid tracker
setgid tracker

console output

exec /home/tracker/nginx/sbin/nginx -c /home/tracker/nginx/conf/nginx.conf -g "daemon off;"


Log in into the tracker account.

Download the Tracker software:

git clone

We'll need to configure the location of Redis. Copy the config file:

cp universal-tracker/config/redis.json.example universal-tracker/config/redis.json

Add a "production" object into the JSON file. Here is an example:

  "development": {
    "host": "",
    "port": 6379,
    "db":   13
  "test": {
    "host": "",
    "port": 6379,
    "db":   14
  "production": {
    "db": 1

Now we may need to fix an issue with Passenger forking after the Redis connection has been made. Please see for more information.

There is also an issue with non-ASCII names. See

Now install the necessary gems:

cd universal-tracker
bundle install

Log out of the tracker account at this point.


Node.js is required to run the fancy leaderboard using WebSockets. We'll use NPM to manage the Node.js libraries:

sudo apt-get install npm

Log into the tracker account.

Now, we manually edit the Node.js program because it has problems:

cp -R universal-tracker/broadcaster .
nano broadcaster/server.js

Modify env and trackerConfig variables to something like this:

var env = {
    tracker_config: {
        redis_pubsub_channel: "tracker-log"
    redis_db: 1
var trackerConfig = env['tracker_config'];

You also need to modify the "transports" configuration by adding websocket. The new line should look like this:

  io.set("transports", ["websocket", "xhr-polling"]);

Install the Node.js libraries needed:

npm install
npm install redis

Log out of the tracker account at this point.

Create an Upstart file at /etc/init/nodejs-tracker.conf:

description "tracker nodejs daemon"

start on runlevel [2]
stop on runlevel [016]

setuid tracker
setgid tracker

exec node /home/tracker/broadcaster/server.js

Tracker Setup

Start up the Tracker and Broadcaster:

sudo start nginx-tracker
sudo start nodejs-tracker

You now need to configure the tracker. Open up your web browser and visit http://localhost/.

  • In Global-Admin→Configuration→Live logging host, specify the public location of the Node.js app. By default, it uses port 8080.

You are now free to manage the tracker.


  • If you followed this guide, the rsync location is defined as rsync://HOSTNAME/PROJECT_NAME/:downloader/


You probably want to have Cron clearing out old claims. The Tracker includes a Ruby script that will do that for you. By default, it removes claims older than 6 hours. You may want to change that for big items by creating a copy of the script for each project.

To set up Cron, login as the tracker account, and run:

which ruby

Take note of which Ruby executable is used.

Now edit the Cron table:

crontab -e

Add the following line which runs release-stale.rb every 6 hours:

0 */6 * * * cd /home/tracker/universal-tracker && WHICH_RUBY scripts/release-stale.rb PROJECT_NAME


Since the Tracker stores logs into Redis, it will use up memory quickly. log-drainer.rb continuously writes the logs into a text file:

mkdir -p /home/tracker/universal-tracker/logs/
cd /home/tracker/universal-tracker && ruby scripts/log-drainer.rb

Pressing CTRL+C will stop it. Run this within a Screen session.

This crontab entry will compress the log files that haven't been modified in two days:

@daily find /home/tracker/universal-tracker/logs/ -iname "*.log" -mtime +2 -exec xz {} \;