User:TheTechRobo/Mnbot
mnbot is an experimental browser-based parallel to ArchiveBot, like Chromebot but without the data mangling. It currently hangs out in #jseater (on hackint).
Migration
mnbot has recently had a complete database rewrite. Unfortunately I ended up enjoying adding features way too much so it's pretty huge. Let me know if you find any bugs!
Most things should continue working. However, there are notably three major differences (off the top of my head):
- mnbot now has direct support for multiple pages in a job; use
!b <rather than!bulk. - The dashboard has been restructured (please let me know if you notice any friction here!)
The old dashboard is available at https://mnbot-old.very-good-quality-co.de[IA•Wcite•.today].
Background information
Pipelines
mnbot is designed to be scalable. More pipelines can easily be added, and while I haven't yet tested it, a single pipeline can be configured to process multiple pages in parallel.
Assuming it already has Docker installed, a pipeline requires roughly 2GB of space to build all the containers. As for temporary space, I don't yet have a good grasp of how much space it needs, but more is obviously better :-)
The Queue
Both jobs and pages are dequeued by niceness (lower nice values run first). Items with the same niceness use FIFO.
Effective niceness (what is actually used for computing order) is defined as nice + tries. Pages currently get tried up to 3 times before failing.
Deduplication
warcprox is configured to deduplicate by writing revisit records. The dedupe database is pipeline-specific. After any task completes, mnbot will remove all dedupe entries older than a week from the database.
Encountering HTTP Errors
Responses that result in HTTP status 401, 403, 404, 405, 410, and 451 are not retried, but all other errors are retried up to three times. A failure notification is not sent until all retries have been exhausted.
Screenshots are written to WARC unless the status code is ≥ 400.
Job Rules
The settings for how a particular page will be handled are decided using a configurable set of rules. Each option is given a default setting that applies to all pages, and then zero or more rules may overwrite that value. A rule consists of a scope (a regex pattern) and a value. Multiple values are not supported.
Currently, the following settings exist:
| Name (identifier) | Description |
|---|---|
| User agent (ua) | Sets user agent header behaviour. For a freeform input, prefix with $. Alternatively, takes one of the following values:
|
| Custom JavaScript (custom_js) | See #Custom JavaScript. Default is none. |
| Accept (accept) | Boolean value used only when discovering outlinks. Should this outlink be accepted into the queue? Default is False. (Note: this is recomputed for each discovered URL, it doesn't matter what page the URLs are discovered on) |
| Skip (skip) | Boolean value. If true, rather than dequeuing this page, the tracker will set its status to SKIPPED. Default is False. |
Using the bot
!brozzle, !b
Creates a job. To begin with one URL, use !b URL. To begin with multiple URLs, use !b < URL_LIST.
Requires one of +v or +o. +v is currently provided to all users registered with NickServ, although that will likely change soon.
Possible options:
--skip-url-validation: Operators (+o) only. Skips URL validation. Useful if the bot is rejecting a valid URL.--explanation/-e: Saves an explanation in the database for the crawl. This can be changed later with the!explaincommand.--nice/-n: Sets niceness (priority). Lower nice values are run first. Only operators can create jobs with a niceness lower than -10. Defaults to 0 for regular jobs, 10 for!b <jobs.--concurrency/-c: Sets the maximum concurrency for the job.--depth: Sets the maximum depth for the job to recurse (default 0, i.e. no recursion). mnbot defaults to not accepting URLs, so ensure you either use --accept or add a rule later on.--tag: Sets the tag of the job; only pipelines that match that tag will run the job.--accept: Supply a pattern with this option and mnbot will add an accept rule for it.--for/--cc: Appends another nick to completion messages. Can be specified multiple times.
For convenience, the following job rules can also be given defaults with this command:
--custom-js: Operators (+o) only. URL to a custom JavaScript file to run during the crawl. See the #Custom JavaScript section below.--user-agent/-u: Sets user agent behaviour; see the section on job rules for more information.
!status
Forms: !status, !status [IDENTS...]
If IDENTS is provided, returns the status of each ident. Otherwise, returns the number of active jobs and pipeline health.
!addrule, !ar
Creates a new job rule.
Usage: !addrule <JOB_ID> <SETTING> <PATTERN> <ARG>
SETTING can be any job rule type. For accept and skip rules, ARG should be either true or false.
The --add-before option can be used to insert into a particular position. It takes an index as argument.
The --default option can be used instead of <PATTERN> to set the default for a job. This cannot be used with --add-before.
!ignore, !ig
Usage: !ignore <JOB_ID> <PATTERN>
This command has the same effect as !addrule <JOB_ID> accept <PATTERN> false followed by !addrule <JOB_ID> skip <PATTERN> true. In other words, outlinks matching PATTERN will no longer be extracted, and they will be skipped when dequeuing. If you'd like to save the URLs for later, create a skip rule manually instead.
Note: there is no !unignore command. Instead, !dr <JOB_ID> <PATTERN> has a similar effect (but note that doing so will wipe all rules with that pattern, not just the reject/skip rules).
!delrule, !dr
Removes one or more job rules.
This command has two forms:
- Remove every rule with a certain scope from a particular setting:
!delrule <JOB_ID> [SETTING] <PATTERN>- In this form, 'all' is an accepted value for SETTING. (Alternatively, omitting SETTING has the same effect.)
- Remove a specific index from a particular setting:
!delrule <JOB_ID> <SETTING> --index <INDEX>
!depth
Usage: !depth <JOB_ID> <NEWVAL>
Sets the current maximum depth for a job. This can be 'inf' for infinite depth. Depth starts at 0 for the initial pages, 1 for their outlinks, etc. The smallest possible path to the page is always considered when determining whether a page is in scope.
Changing this value will recalculate the job status, thereby possibly stopping or resuming it, unless the job was aborted.
!nice
Usage: !nice <JOB_ID> <NEWVAL>
Sets the niceness for a job. Only ops can queue below -10.
!tag
Manipulate pipeline tags.
Available subcommands:
!tag list <PIPELINE>: Lists tags of a particular pipeline.!tag add <PIPELINE> <TAG>: Operators (+o) only. Applies a tag to a pipeline.!tag remove <PIPELINE> <TAG>: Operators (+o) only. Removes a tag from a pipeline.!tag <JOB_ID> <TAG>: Sets the tag of a job.
!page
Manipulate pages.
Available subcommands:
!page status [NEWVAL]: Gets or sets the status of a page. Only voiced users (+v) can set a new value.!page tries [NEWVAL]: Gets or sets the number of remaining tries for a page. Only voiced users (+v) can set a new value.
!concurrency, !con
Sets a job's concurrency. Setting concurrency to 0 effectively pauses the job.
Requires at least +v.
!explain, !e
Sets the explanation for a job.
Usage: !explain <IDENT> [EXPLANATION...]
!abort
Cancels a job. Any pages that are currently running are finished, and then the job is stopped. Jobs can be "unaborted" by operators, but there is currently no command to do so (to pause a job, consider setting concurrency to 0).
Usage: !abort <IDENT>
Custom JavaScript
Operators may add custom JavaScript behaviour to a crawl.
The script is run after the screenshot is taken, but before brozzler is instructed to search for outlinks or visit anchors. An additional screenshot (not written to WARC) will be visible on the dashboard. The return value (the last line) of the script is saved in the WARC and available on the job viewer webpage (see below) -- useful for generating a list of URLs for ArchiveBot, for example. If you want to use the return value, it must be JSON-encodable or you will run into issues. The file must begin with //! mnbot v1 followed by a newline.
The script is run in REPL mode, which means top-level await and let redeclaration are both allowed. Basically, it runs it as if you typed it into the browser console.
Result format
JSON object:
status: success if mnbot believes it to be successful,exceptionif an exception occurred,unknownif unknown- If the status is
unknown, the only other field will befullResultwhich contains the full result as sent from Chrome
- If the status is
exceptionDetails: if an exception occurred, a CDP ExceptionDetails object; otherwise, nullremoteObject: a CDP RemoteObject of the script's return value
API
JSON output can be achieved on some dashboard endpoints by appending .json to the URL. Better documentation coming soon™ :-)
Useful helpers
Append /outlinks or /requisites to the page endpoint to get outlinks and requisites respectively. They are retrieved from all attempts of that page, and aren't deduplicated, so the same URL may appear multiple times. (The exact order isn't defined.)
Example: https://mnbot.very-good-quality-co.de/page/01a07e8e-29c9-7913-b917-6da27d301e61/outlinks, https://mnbot.very-good-quality-co.de/page/01a07e8e-29c9-7913-b917-6da27d301e61/requisites
Current active pipelines
If a website employs IP reputation checks or geoblocking, try queuing to a specific pipeline. If no pipeline is specified, the tracker may hand the job out to any pipeline.
Be sure to follow any appropriate laws/regulations when using the bot. Content must be legal in Germany and whatever region the pipeline is in.
The tracker/IRC bot/dashboard is hosted on a Luna Node VPS in Toronto, Canada. See https://status.lunanode.com/ for status updates from the provider.
| Pipeline ID | Hosting provider | Region | Notes |
|---|---|---|---|
thetechrobo-netcup-1 |
Netcup | Nuremberg, Germany |
Data
There is currently no collection. Data is uploaded to @thetechrobo3641's archive.org account.