Difference between revisions of "Chromebot"

From Archiveteam
Jump to navigation Jump to search
m (→‎People: add devops scripts)
(Added mention of WBM exclusion to lede)
 
(3 intermediate revisions by 3 users not shown)
Line 1: Line 1:
'''chromebot''' aka. '''crocoite''' is an [[IRC]] bot parallel to [[ArchiveBot]] that uses Google Chrome and thus is able to archive JavaScript-heavy and bottomless websites. [[WARC]]s are uploaded twice a day to the [https://archive.org/details/archiveteam_chromebot?sort=-publicdate chromebot collection] on archive.org.
'''chromebot''' aka. '''crocoite''' was an [[IRC]] bot parallel to [[ArchiveBot]] that uses Google Chrome and thus is able to archive JavaScript-heavy and bottomless websites. On 2021-04-21 the bot was shut down (see [[#Shutdown]]), and the captures it made are no longer in the Wayback Machine. [[WARC]]s were uploaded twice a day to the [https://archive.org/details/archiveteam_chromebot?sort=-publicdate chromebot collection] on archive.org. For a given item in the collection, you can see what URLs are saved in the warc by looking at the associated jobs.json.gz file.


By default the bot only grabs a single URL. However it supports recursion, which is rather slow, since every single page needs to be loaded and rendered by a browser. A [http://chromebot.6xq.net/ dashboard] is available for watching the progress of such jobs.
By default the bot only grabs a single URL. However it supports recursion, which is rather slow, since every single page needs to be loaded and rendered by a browser. A [http://chromebot.6xq.net/ dashboard] is available for watching the progress of such jobs.


== Usage ==
== Usage ==
[https://github.com/PromyLOPh/crocoite/blob/184189f0a535996edca01a68182ed07d32e26e9c/README.rst#IRC-bot crocoite usage documentation on GitHub]
[https://6xq.net/crocoite/usage/ crocoite usage documentation]
 
You can call ''chromebot'' on the {{IRC|archivebot}} IRC channel, which chromebot shares with [[ArchiveBot]]. Both “<code>chromebot</code>” and “<code>chromebot:</code>” work, with or without the colon.


{| class="wikitable"
{| class="wikitable"
Line 27: Line 25:
<code>chromebot: a https://transfer.notkiska.pw/inline/UpfR/HollyConrad-tweets -r 1 -j 4</code>
<code>chromebot: a https://transfer.notkiska.pw/inline/UpfR/HollyConrad-tweets -r 1 -j 4</code>


chromebot will assume all lines starting with http(s):// are valid links. Note that the list itself must be retured by the server as an *inline* document, not as a download (attachment).
chromebot will assume all lines starting with http(s):// are valid links. Note that the list itself must be returned by the server as an *inline* document, not as a download (attachment).


== Restrictions ==
== Restrictions ==
Line 40: Line 38:


[[User:PurpleSymphony|PurpleSym]] maintains [https://github.com/PromyLOPh/crocoite software], [https://github.com/PromyLOPh/chromebot scripts], pays the server bills and has administrative access. katocala is a server administrator.
[[User:PurpleSymphony|PurpleSym]] maintains [https://github.com/PromyLOPh/crocoite software], [https://github.com/PromyLOPh/chromebot scripts], pays the server bills and has administrative access. katocala is a server administrator.
== Shutdown ==
In April 2021, it was discovered that the WARCs written by crocoite had incorrect dates. Namely, the revisit records received the date of the daily deduplication run rather than copying the date of retrieval from the replaced response record, leading to a misrepresentation of when the identical capture was found. Further, all records were presented as HTTP/1.1 with made-up headers, including ones using HTTP/2 or any other protocol supported by Chrome (e.g. WebSockets, HTTP/3). These major data integrity problems led to the bot's WARCs being removed from the Wayback Machine index and the bot being shut down indefinitely. The old revisit records' dates can likely not be fixed reliably because the log information is incomplete, hence a reversal of the WBM exclusion is unlikely.


[[Category:Bots]]
[[Category:Bots]]

Latest revision as of 13:46, 17 October 2021

chromebot aka. crocoite was an IRC bot parallel to ArchiveBot that uses Google Chrome and thus is able to archive JavaScript-heavy and bottomless websites. On 2021-04-21 the bot was shut down (see #Shutdown), and the captures it made are no longer in the Wayback Machine. WARCs were uploaded twice a day to the chromebot collection on archive.org. For a given item in the collection, you can see what URLs are saved in the warc by looking at the associated jobs.json.gz file.

By default the bot only grabs a single URL. However it supports recursion, which is rather slow, since every single page needs to be loaded and rendered by a browser. A dashboard is available for watching the progress of such jobs.

Usage

crocoite usage documentation

Command Description

chromebot: a <url> -r <policy> -j <concurrency>

Archive <url> with <concurrency> processes according to recursion <policy>.
chromebot: s <uuid> Get job status for <uuid>.
chromebot: r <uuid> Revoke or abort running job with <uuid>.

Please note that the commands are case-sensitive.

URL lists can be archived using recursion, for example:

chromebot: a https://transfer.notkiska.pw/inline/UpfR/HollyConrad-tweets -r 1 -j 4

chromebot will assume all lines starting with http(s):// are valid links. Note that the list itself must be returned by the server as an *inline* document, not as a download (attachment).

Restrictions

Instagram

chromebot has been blacklisted by Instagram. When trying to archive any Instagram.com website, chromebot responds with the following error:

<Instagram.com URL> cannot be queued: Banned by Instagram

Cloudflare DDoS protection

chromebot should be able to circumvent Cloudflare's DDoS protection, but scrolling and other behaviour may be disabled after the reload (issue #13 on GitHub).

People

PurpleSym maintains software, scripts, pays the server bills and has administrative access. katocala is a server administrator.

Shutdown

In April 2021, it was discovered that the WARCs written by crocoite had incorrect dates. Namely, the revisit records received the date of the daily deduplication run rather than copying the date of retrieval from the replaced response record, leading to a misrepresentation of when the identical capture was found. Further, all records were presented as HTTP/1.1 with made-up headers, including ones using HTTP/2 or any other protocol supported by Chrome (e.g. WebSockets, HTTP/3). These major data integrity problems led to the bot's WARCs being removed from the Wayback Machine index and the bot being shut down indefinitely. The old revisit records' dates can likely not be fixed reliably because the log information is incomplete, hence a reversal of the WBM exclusion is unlikely.