Chromebot
chromebot aka. crocoite was a IRC bot parallel to ArchiveBot that used Google Chrome and thus was able to archive JavaScript-heavy and bottomless websites. On 2021-04-21 the bot was shut down (see #Shutdown), and the captures it made are no longer in the Wayback Machine. WARCs were uploaded twice a day to the archiveteam_chromebot. For a given item in the collection, you can see what URLs were saved in the WARC by looking at the associated jobs.json.gz file.
By default the bot only grabbed a single URL. However it supported recursion, which was rather slow, since every single page needed to be loaded and rendered by a browser. A dashboard http://chromebot.6xq.net/ dashboard[dashboard IA•dashboard Wcite•dashboard .today] was available for watching the progress of such jobs.
Usage
| Command | Description |
|---|---|
|
|
Archive <url> with <concurrency> processes according to recursion <policy>. |
chromebot: s <uuid> |
Get job status for <uuid>. |
chromebot: r <uuid> |
Revoke or abort running job with <uuid>. |
Please note that the commands were case-sensitive.
URL lists could be archived using recursion, for example:
chromebot: a https://transfer.notkiska.pw/inline/UpfR/HollyConrad-tweets -r 1 -j 4
chromebot would assume all lines starting with http(s):// are valid links. Note that the list itself must be returned by the server as an *inline* document, not as a download (attachment).
Restrictions
chromebot was been blacklisted by Instagram. When trying to archive any Instagram.com website, chromebot responds with the following error:
<Instagram.com URL> cannot be queued: Banned by Instagram
Cloudflare DDoS protection
chromebot was able to circumvent Cloudflare's DDoS protection, but scrolling and other behaviour may be disabled after the reload (issue #13 on GitHub).
People
PurpleSym maintained software, scripts, paid the server bills and had administrative access. katocala is a server administrator.
Shutdown
In April 2021, it was discovered that the WARCs written by crocoite had incorrect dates. Namely, the revisit records received the date of the daily deduplication run rather than copying the date of retrieval from the replaced response record, leading to a misrepresentation of when the identical capture was found. Further, all records were presented as HTTP/1.1 with made-up headers, including ones using HTTP/2 or any other protocol supported by Chrome (e.g. WebSockets, HTTP/3). These major data integrity problems led to the bot's WARCs being removed from the Wayback Machine index and the bot being shut down indefinitely. The old revisit records' dates can likely not be fixed reliably because the log information is incomplete, hence a reversal of the WBM exclusion is unlikely.