Difference between revisions of "Tracker"

From Archiveteam
Jump to navigation Jump to search
(Get rid of section header for the lede)
 
(26 intermediate revisions by 12 users not shown)
Line 1: Line 1:
== General Overview ==
[[File:Tracker_test_project_overview_screenshot.png|right|thumb|Project admin overview]]
[[File:Tracker_test_project_overview_screenshot.png|right|thumb|Project admin overview]]


The [https://github.com/ArchiveTeam/universal-tracker Tracker] software is the center-pivot of Archiveteam's distributed archiving efforts. It hands out items to be downloaded and keeps track of what is completed. Items can be usernames, subdomains, full urls, basically any unit we can use to break the site into manageable chunks. The progress of each project can be viewed via the leader board interface on http://tracker.archiveteam.org .
The tracker software is the center-pivot of Archiveteam's distributed archiving efforts. It hands out items to be downloaded and keeps track of what is completed. Items can be usernames, subdomains, full URLs, basically any unit we can use to break the site into manageable chunks. The progress of each project can be viewed via the leaderboard interface on https://tracker.archiveteam.org.


[[File:Xanga_leaderboard.png|right|thumb|A leaderboard]]
[[File:Xanga leaderboard.png|right|thumb|A leaderboard]]


The [[ArchiveTeam Warrior|Warrior]] is the yang to the Tracker's yin. The warriors get the list of current projects from the project file on http://warriorhq.archiveteam.org/ .
The [[ArchiveTeam Warrior|Warrior]] is the yang to the Tracker's yin. The warriors get the list of current projects from the project file on https://warriorhq.archiveteam.org/.


__TOC__
__TOC__
== Userscript ==
A custom (but unofficial) [https://gist.github.com/Nothing4You/b127b3cc4d4fabe372dcf078dbe86070/ userscript], which can be installed using Tampermonkey or other userscript managers, can be used to replace the tracker leaderboard JS to enable the display of additional information including b/s and i/s values as well as time estimates.


== API ==
== API ==
 
This is a sample project snippet from the projects.json file (line breaks included for readability):
This is a sample from the project file (line breaks included for readability):


<pre>
<pre>
Line 37: Line 37:


It shows where to get the grab code and other project information.
It shows where to get the grab code and other project information.
Here is an example root of the file:
<pre>
{
    "auto_project": "projectslug",
    "broadcast_message": "<p>This message is shown only
        in the warrior VM web UI at time of writing.</p>",
    "tracker_banner_html": "This is shown on the tracker
        front page. <em>Wow!</em>",
    "warrior": {"seesaw_version":"0.7.0"},
    "projects": [],
}
</pre>


== Hardware ==
== Hardware ==
The tracker runs on a [http://www.archiveteam.org/index.php?title=Clown_hosting#linode Linode 1 GB] instance operated by [[User:Chronomex|chronomex]].
In 20??-present day (2023), the tracker runs on "several beefy Hetzner servers".


== Monitoring ==
== Software ==
* A proprietary system consisting of a large wrapper around the original [https://github.com/ArchiveTeam/universal-tracker Universal Tracker], a Ruby HTTP application that sends and receives JSON payloads and uses Redis for the data store.
* Redis A memory-based key-value store
* [https://debian.org/ Debian] is the Linux distribution the stack is built upon.
* [https://github.com/ArchiveTeam/warrior-hq warrior-hq] a small Sinatra web app to manage the Warriors and display the geo-location world map.


http://tracker.archiveteam.org has a Munin instance located at http://zeppelin.xrtc.net/corp.xrtc.net/shilling.corp.xrtc.net/.
You can also [[Tracker Setup|set up your own tracker]].


== Software used: ==


* [https://github.com/ArchiveTeam/universal-tracker Universal Tracker] is a Ruby HTTP application that sends and receives JSON payloads and uses Redis for the data store.
== History ==
* Redis A memory based key value store
This history is both incomplete (it stops in early 2012) and probably wrong in areas.
* [http://debian.org/ Debian] is the Linux distribution the stack is built upon.
* [https://github.com/ArchiveTeam/warrior-hq warrior-hq] a small Sinatra web app to manage the Warriors and display the geo-location world map.


You can also [[Tracker_Setup|set up your own tracker]].
Originally, ArchiveTeam coordinated large projects through the wiki, keeping tables of (the analogues of what are called in tracker terminology) items and letting people claim them and update their progress by editing the wiki.<ref>[[Yahoo_Video_Warroom]]</ref>Midway through the [[Google Video]] project, in April 2011, [[User:Underscor|Underscor]] created a system called "Listerine" that did this automatically<ref>[[Google Video Warroom#Get_Involved_With_Listerine]]</ref><ref>[[Google Video#A_Brief_History]], take caution that public IRC logs are not available from this time</ref> (a client for the Listerine protocol can be found [https://github.com/ArchiveTeam/googlegargle/blob/master/listerine here].) This remained an attractive concept (and the name "tracker" seems to have originated during a wistful discussion about this<ref>https://archive.fart.website/bin/irclogger_log/archiveteam?date=2011-07-13,Wed&sel=94#l94</ref>), and later in the year, [[User:Alard|Alard]], the de facto project lead/resident enthusiast of [[MobileMe]], wanted Underscor to set up another Listerine instance for it<ref>https://archive.fart.website/bin/irclogger_log/archiveteam?date=2011-10-16,Sun&sel=265#l265</ref>. Apparently this never happened, since a few weeks later Alard wrote his own system, which had something resembling the current tracker protocol.<ref>https://github.com/ArchiveTeam/mobileme-grab/commit/006bc0388f3ce3867bb3fd654c72c936cec57c02</ref> A few months later, this was replaced by a shell script called "Seesaw", which added automatic uploading with Rsync<ref>https://github.com/ArchiveTeam/mobileme-grab/commit/94873e7d81ea68b3f99cfba56d64bd57fd68dcaa</ref>


== People ==
Sometime in the late 2010s the open-source tracker was gradually replaced with the proprietary one. Then or in the early 2020s backfeed and multi-item support was added. As of 2023 most of the admin functionality is broken, as far as [[User:OrIdow6|I]] know<!-- If you update this with more accurate info please remove the attribution to me -->; everything but setting the minimum version and the rate limit is done with non-public methods by the tracker admins.
These are the volunteers who take care of the different services that form Archive Team and URLTeam.


{| class="wikitable"
{{Navigation box}}
! Service:
! Admins:
|-
|Wiki Admins
|SketchCow, winr4r
|-
|warriorhq.archiveteam.org (projects.json)
|Smiley
|-
|Universal Tracker SSH
|alard, Smiley, underscor, yipdw, xmc
|-
|Universal Tracker web interface
|alard, omf_, Smiley, underscor
|-
|Anarchive server
|GLaDOS, omf_, Smiley
|-
|URLTeam Tracker software
|GLaDOS, omf_, Smiley
|-
|Github Organization Admins
|GLaDOS, ivan, omf_
|-
|#archiveteam-twitter twitter to IRC bot
|GLaDOS
|-
|pad.archivingyoursh.it
paste.archivingyoursh.it
|GLaDOS
|-
|Domain registration (archiveteam.org urlte.am)
|SketchCow
|}

Latest revision as of 05:37, 2 August 2023

Project admin overview

The tracker software is the center-pivot of Archiveteam's distributed archiving efforts. It hands out items to be downloaded and keeps track of what is completed. Items can be usernames, subdomains, full URLs, basically any unit we can use to break the site into manageable chunks. The progress of each project can be viewed via the leaderboard interface on https://tracker.archiveteam.org.

A leaderboard

The Warrior is the yang to the Tracker's yin. The warriors get the list of current projects from the project file on https://warriorhq.archiveteam.org/.

Userscript

A custom (but unofficial) userscript, which can be installed using Tampermonkey or other userscript managers, can be used to replace the tracker leaderboard JS to enable the display of additional information including b/s and i/s values as well as time estimates.

API

This is a sample project snippet from the projects.json file (line breaks included for readability):

{
    "name": "streetfiles",
    "title": "Streetfiles",
    "description": "Streetfiles is closing April, 30th, 2013.",
    "repository": "https://github.com/ArchiveTeam/streetfiles-grab.git",
    "logo": "http://archiveteam.org/images/7/7b/Streetfiles-logo.png",
    "marker_html": 
        "<a href='http://tracker.archiveteam.org/streetfiles/'>
        <img src='http://archiveteam.org/images/7/7b/Streetfiles-logo.png'
        alt='Streetfiles' width='235' height='50' /></a>",
    "deadline": "2013-04-30T23:59:59Z",
    "host": "streetfiles.org",
    "leaderboard": "http://tracker.archiveteam.org/streetfiles/",
    "lat_lng": [
        51,
        9
    ]
},

It shows where to get the grab code and other project information.

Here is an example root of the file:

{
    "auto_project": "projectslug",
    "broadcast_message": "<p>This message is shown only 
        in the warrior VM web UI at time of writing.</p>",
    "tracker_banner_html": "This is shown on the tracker 
        front page. <em>Wow!</em>",
    "warrior": {"seesaw_version":"0.7.0"},
    "projects": [],
}

Hardware

In 20??-present day (2023), the tracker runs on "several beefy Hetzner servers".

Software

  • A proprietary system consisting of a large wrapper around the original Universal Tracker, a Ruby HTTP application that sends and receives JSON payloads and uses Redis for the data store.
  • Redis A memory-based key-value store
  • Debian is the Linux distribution the stack is built upon.
  • warrior-hq a small Sinatra web app to manage the Warriors and display the geo-location world map.

You can also set up your own tracker.


History

This history is both incomplete (it stops in early 2012) and probably wrong in areas.

Originally, ArchiveTeam coordinated large projects through the wiki, keeping tables of (the analogues of what are called in tracker terminology) items and letting people claim them and update their progress by editing the wiki.[1]Midway through the Google Video project, in April 2011, Underscor created a system called "Listerine" that did this automatically[2][3] (a client for the Listerine protocol can be found here.) This remained an attractive concept (and the name "tracker" seems to have originated during a wistful discussion about this[4]), and later in the year, Alard, the de facto project lead/resident enthusiast of MobileMe, wanted Underscor to set up another Listerine instance for it[5]. Apparently this never happened, since a few weeks later Alard wrote his own system, which had something resembling the current tracker protocol.[6] A few months later, this was replaced by a shell script called "Seesaw", which added automatic uploading with Rsync[7]

Sometime in the late 2010s the open-source tracker was gradually replaced with the proprietary one. Then or in the early 2020s backfeed and multi-item support was added. As of 2023 most of the admin functionality is broken, as far as I know; everything but setting the minimum version and the rate limit is done with non-public methods by the tracker admins.