Difference between revisions of "Dev/Tracker"
(→Tracker: add command bundle install) |
TheTechRobo (talk | contribs) (Improve the link to the Docker container) |
||
(31 intermediate revisions by 8 users not shown) | |||
Line 1: | Line 1: | ||
This article describes how to set up your own [[tracker]] | This article describes how to set up your own '''[[tracker]]''' <s>just like the official Archive Team tracker</s> just like the open-source core of the official Archive Team tracker. | ||
'''Note:''' A virtual machine appliance is available at [https://github.com/ArchiveTeam/archiveteam-dev-env ArchiveTeam/archiveteam-dev-env] which contains a ready-to-use tracker. A docker container is also at [https://github.com/marked/universal-tracker/tree/docker-redisgem2 this fork of the tracker (unofficial)]. | |||
* Environment: | Installation will cover: | ||
* Environment: Ubuntu/Debian | |||
* Languages: | * Languages: | ||
** Python | ** Python | ||
** Ruby | ** Ruby | ||
** JavaScript | ** JavaScript | ||
* Web: | * Web: | ||
** Nginx | ** Nginx | ||
** Phusion Passenger | ** Phusion Passenger | ||
** Redis | ** Redis | ||
** Node.js | ** Node.js | ||
Line 25: | Line 21: | ||
** Wget | ** Wget | ||
** regular expressions | ** regular expressions | ||
== The Tracker == | == The Tracker == | ||
Line 191: | Line 32: | ||
=== Redis === | === Redis === | ||
Redis is database stored in memory. So, | Redis is database stored in memory. So, item names should be engineered to be memory efficient. Redis saves its database periodically into a file located at /var/lib/redis/6379/dump.rdb. It is safe to copy the file, e.g., for backups. | ||
To install Redis, you may follow these [http://redis.io/topics/quickstart quickstart instructions], but we'll show you how. | To install Redis, you may follow these [http://redis.io/topics/quickstart quickstart instructions], but we'll show you how. | ||
Line 204: | Line 45: | ||
Now install the server: | Now install the server: | ||
sudo . | sudo make install | ||
cd utils | |||
sudo ./install_server.sh | |||
Note, by default, it runs as root. Let's stop it and make it run under www-data: | Note, by default, it runs as root. Let's stop it and make it run under www-data: | ||
Line 227: | Line 70: | ||
CLIEXEC="sudo -u www-data -g www-data /usr/local/bin/redis-cli" | CLIEXEC="sudo -u www-data -g www-data /usr/local/bin/redis-cli" | ||
PIDFILE=/var/run/shm/redis_6379.pid | PIDFILE=/var/run/shm/redis_6379.pid | ||
To avoid catastrophe with background saves failing on <code>fork()</code> (Redis needs lots of memory), run: | |||
sudo sysctl vm.overcommit_memory=1 | |||
The above setting will be lost after reboot. Add this line to <code>/etc/sysctl.conf</code>: | |||
vm.overcommit_memory=1 | |||
The log file will get big so we need a logrotate config. Create one at <code>/etc/logrotate.d/redis</code> with the config: | |||
/var/log/redis_*.log { | |||
daily | |||
rotate 10 | |||
copytruncate | |||
delaycompress | |||
compress | |||
notifempty | |||
missingok | |||
size 10M | |||
} | |||
Start up Redis again using: | Start up Redis again using: | ||
Line 250: | Line 114: | ||
A list of things needed to be installed will be shown. Log out of the tracker account, install them, and log back into the tracker account. | A list of things needed to be installed will be shown. Log out of the tracker account, install them, and log back into the tracker account. | ||
Install Ruby and | Install Ruby and Bundler: | ||
rvm install 2. | rvm install 2.2.2 | ||
rvm rubygems current | rvm rubygems current | ||
gem install | gem install bundler | ||
Install Passenger: | Install Passenger: | ||
Line 273: | Line 137: | ||
passenger_enabled on; | passenger_enabled on; | ||
client_max_body_size 15M; | client_max_body_size 15M; | ||
The logs will get big so we'll use logrotate. Save this into <code>/home/tracker/logrotate.conf</code>: | |||
/home/tracker/nginx/logs/error.log | |||
/home/tracker/nginx/logs/access.log { | |||
daily | |||
rotate 10 | |||
copytruncate | |||
delaycompress | |||
compress | |||
notifempty | |||
missingok | |||
size 10M | |||
} | |||
To call logrotate, we'll add an entry using crontab: | |||
crontab -e | |||
Now add the following line: | |||
@daily /usr/sbin/logrotate --state /home/tracker/.logrotate.state /home/tracker/logrotate.conf | |||
Log out of the tracker account at this point. | Log out of the tracker account at this point. | ||
Let's create an Upstart configuration file to start up Nginx. Save this into <code>/etc/init/ | Let's create an Upstart configuration file to start up Nginx. Save this into <code>/etc/init/nginx-tracker.conf</code>: | ||
description "nginx http daemon" | description "nginx http daemon" | ||
Line 289: | Line 175: | ||
exec /home/tracker/nginx/sbin/nginx -c /home/tracker/nginx/conf/nginx.conf -g "daemon off;" | exec /home/tracker/nginx/sbin/nginx -c /home/tracker/nginx/conf/nginx.conf -g "daemon off;" | ||
Or, if you use Systemd, put this into <code>/lib/systemd/system/nginx-tracker.service</code>: | |||
[Unit] | |||
Description="nginx http daemon" | |||
[Service] | |||
Type=simple | |||
ExecStart=/home/tracker/nginx/sbin/nginx -c /home/tracker/nginx/conf/nginx.conf -g "daemon off;" | |||
=== Tracker === | === Tracker === | ||
Line 322: | Line 217: | ||
} | } | ||
Now we may need to fix an issue with Passenger forking after the Redis connection has been made. Please see https://github.com/ArchiveTeam/universal-tracker/issues/5 for more information. | * Now we may need to fix an issue with Passenger forking after the Redis connection has been made. Please see https://github.com/ArchiveTeam/universal-tracker/issues/5 for more information. | ||
* There is also an issue with non-ASCII names. See https://github.com/ArchiveTeam/universal-tracker/issues/7. | |||
Now install the necessary gems: | Now install the necessary gems: | ||
Line 360: | Line 256: | ||
Install the Node.js libraries needed: | Install the Node.js libraries needed: | ||
npm install | npm install | ||
If you get an error while installing hiredis, you may need to provide Debian's "nodejs" as "node". Symlink "node" to the nodejs executable and try again. | |||
Log out of the tracker account at this point. | Log out of the tracker account at this point. | ||
Line 376: | Line 273: | ||
exec node /home/tracker/broadcaster/server.js | exec node /home/tracker/broadcaster/server.js | ||
Or, for Systemd, put this into <code>/lib/systemd/system/nodejs-tracker.service</code>: | |||
[Unit] | |||
Description="tracker nodejs daemon" | |||
[Service] | |||
Type=forking | |||
Group=tracker | |||
User=tracker | |||
ExecStart=/usr/bin/js /home/tracker/broadcaster/server.js | |||
=== Tracker Setup === | === Tracker Setup === | ||
Line 381: | Line 289: | ||
Start up the Tracker and Broadcaster: | Start up the Tracker and Broadcaster: | ||
Upstart: | |||
sudo start nginx-tracker | sudo start nginx-tracker | ||
sudo start nodejs-tracker | sudo start nodejs-tracker | ||
You now need to configure the tracker. Open up your web browser and visit http://localhost/. | Systemd: | ||
sudo systemctl start nginx-tracker | |||
sudo systemctl start nodejs-tracker | |||
You now need to configure the tracker. Open up your web browser and visit http://localhost/global-admin/. | |||
* In Global-Admin→Configuration→Live logging host, specify the public location of the Node.js app. By default, it uses port 8080. | * In Global-Admin→Configuration→Live logging host, specify the public location of the Node.js app. By default, it uses port 8080. | ||
Line 393: | Line 306: | ||
* If you followed this guide, the rsync location is defined as <code>rsync://HOSTNAME/PROJECT_NAME/:downloader/</code> | * If you followed this guide, the rsync location is defined as <code>rsync://HOSTNAME/PROJECT_NAME/:downloader/</code> | ||
* The '''''trailing slash''''' within the rsync URL is very important. Without it, files will not be uploaded within the directory. | |||
=== Claims === | ==== Claims ==== | ||
You probably want to have Cron clearing out old claims. The Tracker includes a Ruby script that will do that for you. By default, it removes claims older than 6 hours. You may want to change that for big items by creating a copy of the script for each project. | You probably want to have Cron clearing out old claims. The Tracker includes a Ruby script that will do that for you. By default, it removes claims older than 6 hours. You may want to change that for big items by creating a copy of the script for each project. | ||
Line 411: | Line 325: | ||
0 */6 * * * cd /home/tracker/universal-tracker && WHICH_RUBY scripts/release-stale.rb PROJECT_NAME | 0 */6 * * * cd /home/tracker/universal-tracker && WHICH_RUBY scripts/release-stale.rb PROJECT_NAME | ||
==== Logs ==== | |||
Since the Tracker stores logs into Redis, it will use up memory quickly. <code>log-drainer.rb</code> continuously writes the logs into a text file: | |||
mkdir -p /home/tracker/universal-tracker/logs/ | |||
cd /home/tracker/universal-tracker && ruby scripts/log-drainer.rb | |||
Pressing CTRL+C will stop it. Run this within a Screen session. | |||
This crontab entry will compress the log files that haven't been modified in two days: | |||
@daily find /home/tracker/universal-tracker/logs/ -iname "*.log" -mtime +2 -exec xz {} \; | |||
==== Reducing memory usage ==== | |||
The Passenger Ruby module may use up too much memory. You can add the following lines to your nginx config. Add these inside the <code>http</code> block: | |||
passenger_max_pool_size 2; | |||
passenger_max_requests 10000; | |||
The first line allows spawning maximum of 2 processes. The second line restarts Passenger after 10,000 requests to free memory caused by memory leaks. | |||
{{devnav}} | |||
{{Navigation box}} |
Latest revision as of 02:31, 14 August 2023
This article describes how to set up your own tracker just like the official Archive Team tracker just like the open-source core of the official Archive Team tracker.
Note: A virtual machine appliance is available at ArchiveTeam/archiveteam-dev-env which contains a ready-to-use tracker. A docker container is also at this fork of the tracker (unofficial).
Installation will cover:
- Environment: Ubuntu/Debian
- Languages:
- Python
- Ruby
- JavaScript
- Web:
- Nginx
- Phusion Passenger
- Redis
- Node.js
- Tools:
- Screen
- Rsync
- Git
- Wget
- regular expressions
The Tracker
The Tracker manages what items are claimed by users that run the Seesaw client. It also shows a pretty leaderboard.
Let's create a dedicated account to run the web server and tracker:
sudo adduser --system --group --shell /bin/bash tracker
Redis
Redis is database stored in memory. So, item names should be engineered to be memory efficient. Redis saves its database periodically into a file located at /var/lib/redis/6379/dump.rdb. It is safe to copy the file, e.g., for backups.
To install Redis, you may follow these quickstart instructions, but we'll show you how.
These steps are from the quickstart guide:
wget http://download.redis.io/redis-stable.tar.gz tar xvzf redis-stable.tar.gz cd redis-stable make
Now install the server:
sudo make install cd utils sudo ./install_server.sh
Note, by default, it runs as root. Let's stop it and make it run under www-data:
sudo invoke-rc.d redis_6379 stop sudo adduser --system --group www-data sudo chown -R www-data:www-data /var/lib/redis/6379/ sudo chown -R www-data:www-data /var/log/redis_6379.log
Edit the config file /etc/redis/6379.conf
with the options like:
bind 127.0.0.1 pidfile /var/run/shm/redis_6379.pid
Now tell the start up script to run it as www-data:
sudo nano /etc/init.d/redis_6379
Change the EXEC and CLIEXEC variables to use sudo -u www-data -g www-data
:
EXEC="sudo -u www-data -g www-data /usr/local/bin/redis-server" CLIEXEC="sudo -u www-data -g www-data /usr/local/bin/redis-cli" PIDFILE=/var/run/shm/redis_6379.pid
To avoid catastrophe with background saves failing on fork()
(Redis needs lots of memory), run:
sudo sysctl vm.overcommit_memory=1
The above setting will be lost after reboot. Add this line to /etc/sysctl.conf
:
vm.overcommit_memory=1
The log file will get big so we need a logrotate config. Create one at /etc/logrotate.d/redis
with the config:
/var/log/redis_*.log { daily rotate 10 copytruncate delaycompress compress notifempty missingok size 10M }
Start up Redis again using:
sudo invoke-rc.d redis_6379 start
Nginx with Passenger
Nginx is a web server. Phusion Passenger is a module within Nginx that runs Rails applications.
There is a guide on how to install Nginx with Passenger, the following instructions are similar.
Log in as tracker:
sudo -u tracker -i
We'll use RVM to install Ruby libraries:
curl -L get.rvm.io | bash -s stable source ~/.rvm/scripts/rvm rvm requirements
A list of things needed to be installed will be shown. Log out of the tracker account, install them, and log back into the tracker account.
Install Ruby and Bundler:
rvm install 2.2.2 rvm rubygems current gem install bundler
Install Passenger:
gem install passenger
Install Nginx. This command will download, compile, and install a basic Nginx server.:
passenger-install-nginx-module
Use the following prefix for Nginx installation:
/home/tracker/nginx/
Change the location of the tracker software (to be installed later). Edit nginx/conf/nginx.conf
. Use the lines under the "location /" option:
root /home/tracker/universal-tracker/public; passenger_enabled on; client_max_body_size 15M;
The logs will get big so we'll use logrotate. Save this into /home/tracker/logrotate.conf
:
/home/tracker/nginx/logs/error.log /home/tracker/nginx/logs/access.log { daily rotate 10 copytruncate delaycompress compress notifempty missingok size 10M }
To call logrotate, we'll add an entry using crontab:
crontab -e
Now add the following line:
@daily /usr/sbin/logrotate --state /home/tracker/.logrotate.state /home/tracker/logrotate.conf
Log out of the tracker account at this point.
Let's create an Upstart configuration file to start up Nginx. Save this into /etc/init/nginx-tracker.conf
:
description "nginx http daemon" start on runlevel [2] stop on runlevel [016] setuid tracker setgid tracker console output exec /home/tracker/nginx/sbin/nginx -c /home/tracker/nginx/conf/nginx.conf -g "daemon off;"
Or, if you use Systemd, put this into /lib/systemd/system/nginx-tracker.service
:
[Unit] Description="nginx http daemon" [Service] Type=simple ExecStart=/home/tracker/nginx/sbin/nginx -c /home/tracker/nginx/conf/nginx.conf -g "daemon off;"
Tracker
Log in into the tracker account.
Download the Tracker software:
git clone https://github.com/ArchiveTeam/universal-tracker.git
We'll need to configure the location of Redis. Copy the config file:
cp universal-tracker/config/redis.json.example universal-tracker/config/redis.json
Add a "production" object into the JSON file. Here is an example:
{ "development": { "host": "127.0.0.1", "port": 6379, "db": 13 }, "test": { "host": "127.0.0.1", "port": 6379, "db": 14 }, "production": { "host":"127.0.0.1", "port":6379, "db": 1 } }
- Now we may need to fix an issue with Passenger forking after the Redis connection has been made. Please see https://github.com/ArchiveTeam/universal-tracker/issues/5 for more information.
- There is also an issue with non-ASCII names. See https://github.com/ArchiveTeam/universal-tracker/issues/7.
Now install the necessary gems:
cd universal-tracker bundle install
Log out of the tracker account at this point.
Node.js
Node.js is required to run the fancy leaderboard using WebSockets. We'll use NPM to manage the Node.js libraries:
sudo apt-get install npm
Log into the tracker account.
Now, we manually edit the Node.js program because it has problems:
cp -R universal-tracker/broadcaster . nano broadcaster/server.js
Modify env
and trackerConfig
variables to something like this:
var env = { tracker_config: { redis_pubsub_channel: "tracker-log" }, redis_db: 1 }; var trackerConfig = env['tracker_config'];
You also need to modify the "transports" configuration by adding websocket
. The new line should look like this:
io.set("transports", ["websocket", "xhr-polling"]);
Install the Node.js libraries needed:
npm install
If you get an error while installing hiredis, you may need to provide Debian's "nodejs" as "node". Symlink "node" to the nodejs executable and try again.
Log out of the tracker account at this point.
Create an Upstart file at /etc/init/nodejs-tracker.conf
:
description "tracker nodejs daemon" start on runlevel [2] stop on runlevel [016] setuid tracker setgid tracker exec node /home/tracker/broadcaster/server.js
Or, for Systemd, put this into /lib/systemd/system/nodejs-tracker.service
:
[Unit] Description="tracker nodejs daemon" [Service] Type=forking Group=tracker User=tracker ExecStart=/usr/bin/js /home/tracker/broadcaster/server.js
Tracker Setup
Start up the Tracker and Broadcaster:
Upstart:
sudo start nginx-tracker sudo start nodejs-tracker
Systemd:
sudo systemctl start nginx-tracker sudo systemctl start nodejs-tracker
You now need to configure the tracker. Open up your web browser and visit http://localhost/global-admin/.
- In Global-Admin→Configuration→Live logging host, specify the public location of the Node.js app. By default, it uses port 8080.
You are now free to manage the tracker.
Notes:
- If you followed this guide, the rsync location is defined as
rsync://HOSTNAME/PROJECT_NAME/:downloader/
- The trailing slash within the rsync URL is very important. Without it, files will not be uploaded within the directory.
Claims
You probably want to have Cron clearing out old claims. The Tracker includes a Ruby script that will do that for you. By default, it removes claims older than 6 hours. You may want to change that for big items by creating a copy of the script for each project.
To set up Cron, login as the tracker account, and run:
which ruby
Take note of which Ruby executable is used.
Now edit the Cron table:
crontab -e
Add the following line which runs release-stale.rb
every 6 hours:
0 */6 * * * cd /home/tracker/universal-tracker && WHICH_RUBY scripts/release-stale.rb PROJECT_NAME
Logs
Since the Tracker stores logs into Redis, it will use up memory quickly. log-drainer.rb
continuously writes the logs into a text file:
mkdir -p /home/tracker/universal-tracker/logs/ cd /home/tracker/universal-tracker && ruby scripts/log-drainer.rb
Pressing CTRL+C will stop it. Run this within a Screen session.
This crontab entry will compress the log files that haven't been modified in two days:
@daily find /home/tracker/universal-tracker/logs/ -iname "*.log" -mtime +2 -exec xz {} \;
Reducing memory usage
The Passenger Ruby module may use up too much memory. You can add the following lines to your nginx config. Add these inside the http
block:
passenger_max_pool_size 2; passenger_max_requests 10000;
The first line allows spawning maximum of 2 processes. The second line restarts Passenger after 10,000 requests to free memory caused by memory leaks.