View source for Wikibot

{{Infobox project
| title = wikibot
| image = Dummy.png
| description = IRC bot to run MediaWiki, DokuWiki and PukiWiki dumps
| project_status = {{specialcase}}
| archiving_status = {{inprogress}} (manual)
| archiving_type = other
| source = [https://github.com/DigitalDwagon/WikiBot wikibot]
| irc = wikibot
| data = {{IA id|wikiteam}} (all wikiteam uploads) {{IA id|wikiteam_inbox_1}} (recent wikibot-only uploads)
| lead = [[User:DigitalDragon]]
}}

'''wikibot''' is an IRC bot that will dump the contents of MediaWiki, DokuWiki, and PukiWiki instances. These dumps are uploaded to the [[WikiTeam]] collection at the [[Internet Archive]] for preservation.

== Details ==
wikibot runs in the {{IRC|wikibot}} IRC channel. Various commands (explained below) are available to interact with the bot. In order to create and manage dumps, you'll need voice (<code>+</code>) or operator (<code>@</code>) permissions. If you don't have those, just ask in the channel and someone with permission will be able to help you. If your request gets missed, you may want to ask in the less noisy {{IRC|wikiteam}} or {{IRC|archiveteam-bs}} channels instead. A dashboard to view running jobs is available [https://wikibot.digitaldragon.dev here]. 

If you're writing automation around wikibot, you can get a list of queued and running jobs at <code><nowiki>https://wikibot.digitaldragon.dev/api/jobs</nowiki></code>, get information about a specific job at <code><nowiki>https://wikibot.digitaldragon.dev/api/jobs/JOB_ID</nowiki></code>, get a websocket of updates to jobs at <code>wss://wikibot.digitaldragon.dev/api/jobevents</code>, and get a firehose of job logs at <code>wss://wikibot.digitaldragon.dev/api/logfirehose</code>. Please note that these APIs are not stable and will likely change in the future.

== Commands ==
* <code>!help</code> shows a help message.
* <code>!status [job ID]</code> will show the status of a specific job, or a summary of all jobs currently running.
* <code>!abort <job ID></code> will stop a job that is currently running.
* <code>!reupload <job ID></code> can be used to retry a failed upload to the Internet Archive.
* <code>!check <search></code> will generate an Internet Archive search link for a provided domain name, to check if the wiki has already been downloaded. The bot will also run a search, and it requires a special parameter to download a wiki that has been dumped within the last year.
* <code>!bulk <url></code> will run all of the commands in the linked text file. Please note that !bulk only supports <code>!mw</code>, <code>!dw</code>, and <code>!pw</code> for now. Jobs will run with <code>--silent-mode fail</code> unless otherwise specified. Avoid running large lists of jobs with <code>--silent-mode all</code> as this will flood the channel with messages about each job starting. https://transfer.archivete.am is the preferred place to upload files.
* <code>!savedb</code> (ops only) saves job metadata to the database again, which is useful if the disk fills and some writes are missed

=== Queue Management ===
The bot has multiple different queues that jobs can go into. Queues each have a set concurrency (the maximum number of jobs in the queue that can run at once) and priority (a queue with a higher priority value will start jobs before a queue that has a lower priority value). If you're planning to run many wikis at once, especially if they're on the same wiki farm, please contact an operator to set up a special queue to avoid the bot getting banned or overwhelming the site. 
* <code>!getqueue <queue></code> checks the concurrency and priority level of a given queue.
* <code>!setqueue <queue> <concurrency> <priority></code> (ops only) sets the concurrency and priority of a queue
* <code>!movejob <job ID> <queue></code> moves a job into the specified queue
* <code>!pause</code> (ops only) stops all users from submitting new jobs

=== All job types ===
The following options can be used across all job types.
{| class="wikitable" 
|-
! Option
! Description
|-
| --url
| The URL of the wiki to archive<br />
|-
| --explain
| Adds an explanation (or note) to the job
|-
| --delay
| The delay between requests in '''seconds''' (NOT ms, unlike [[ArchiveBot]])
|-
| --insecure
| Ignore invalid HTTPS certificates and other SSL errors
|-
| --force
| Run the job even if an archive of the wiki from the last 365 days already exists<br />
|-
| --resume
| Job ID of a previous (failed) job to resume from. Useful if a job fails due to a temporary ban or other transient error.
|-
| --queue
| Put this job into a specific queue (if unspecified, jobs go into the <code>default</code> queue)
|-
| --silent-mode<br />
| What notifications to get about a job.<br/><code>all</code> will send all messages<br/><code>end</code> will skip the "Queued job!" message (useful for <code>!bulk</code>)<br/><code>fail</code> will only send a message if the job fails<br/><code>silent</code>won't send any messages about the job<br />
|}

=== MediaWiki ===
Uses [https://github.com/saveweb/wikiteam3 wikiteam3] to archive the contents of a MediaWiki wiki. 
* <code>!mw <options></code> will dump a MediaWiki wiki.
{| class="wikitable" 
|-
! Option
! Description
|-
| --api, -A<br />
| Direct link to the <code>api.php</code> of the wiki. The bot will try to automatically detect this unless specified. (e.g. <code><nowiki>https://wiki.example.com/w/api.php</nowiki></code>
|-
| --index, -N
| Direct link to the <code>index.php</code> of the wiki. The bot will try to automatically detect this unless specified. (e.g. <code><nowiki>https://wiki.example.com/w/index.php</nowiki></code>
|-
| --api_chunksize, -C<br />
| The number of pages, revisions, etc to ask for in each API request (default <code>50</code>, most wikis will ignore values above 50)
|-
| --index-check-threshold
| Skip index.php check if likeliness for index.php to exist is (>) this value (default: <code>0.80</code>)
|-
| --xml, -x
| Export an XML dump. If no other dump option is specified, it will use <code>Special:Export</code> to dump page content. It is highly recommended to use <code>--xmlapiexport</code> or <code>--xmlrevisions</code> if possible
|-
| --xmlapiexport, -a
| Use the revisions API to export page XML
|-
| --xmlrevisions, -r
| (Recommended) Use the allrevisions API to export page XML. This is the fastest and most efficient method, but is only supported on wikis using MediaWiki <code>1.27</code> or later
|-
| --images, -i
| Include images in the dump. Recommended unless the images are over 500GiB in size as per the wiki's <code>Special:MediaStatistics</code> page
|-
| --redirects
| Dumps the page redirect map using <code>API:Allredirects</code>. Redirects will normally be saved in XML, but this may be useful for wikis with strange redirect behavior.
|-
| --bypass-cdn-image-compression
| Bypass lossy CDN image compression used by some wikis (ex. Cloudflare Polish)
|-
| --disable-image-verify, -V
| Don't verify the image size and hash while downloading
|-
| --retries
| How many times to retry each request before the job fails
|-
| --hard-retries
| How many times to retry hard failures on requests (for example, interrupted connections)
|-
| --curonly, -n
| Only download the latest revision of each page. Not compatible with <code>--xmlrevisions</code>
|}

==== Usage Tips ====
* <code>KeyError: 'allrevisions'</code> means that the wiki is too old to support <code>--xmlrevisions</code>, try <code>--xmlapiexport</code> or just <code>--xml</code> on its own instead.
* If you get an error like <code>ERROR: Unsupported wiki. Wiki engines supported are: MediaWiki</code> but you're sure you tried to dump a MediaWiki:
** Try to find the <code>index.php</code> and <code>api.php</code> paths of the wiki and pass them directly to the bot. Clicking the login link will usually help find the <code>index.php</code> page. <code>index.php</code> and <code>api.php</code> are almost always interchangeable 
** Open inspect element and look for a comment at the top about <code>HTTrack</code>. If this appears, the wiki is a static conversion, and [[ArchiveBot]] will be needed instead
* A <code>ChunkedEncodingError</code> usually just means the job needs to be resumed. You may also want to consider a higher <code>--hard-retries</code> if the wiki seems to consistently have this problem ([[Fandom]] is an example that often has these issues)
* If the wiki is in another language, replacing the page name in the URL with <code>Special:Version</code> or the English name of another special page will usually redirect you to the right place.
* You can use the wiki's <code>Special:Version</code> page to check if it is new enough to use `--xmlrevisions`. 
* The <code>--delay</code> of a job can be changed while it is running by someone with access to the server wikibot runs on, so ask in the channel if you need to adjust the speed of a job.

=== DokuWiki ===
Uses [https://github.com/saveweb/dokuwiki-dumper DokuWiki Dumper] to archive the contents of a DokuWiki wiki. 
* <code>!dw <options></code> will dump a DokuWiki wiki.
{| class="wikitable" 
|-
! Parameter
! Description
|-
| --auto
| Dump: content+media+html, threads=5, ignore-action-disable-edit. (threads is overridable)
|-
| --ignore-disposition-header-missing
| Do not check Disposition header, useful for outdated (<2014) DokuWiki versions [default: False]
|-
| --threads
| Number of sub threads to use [default: 1], not recommended to set > 5
|-
| --ignore-action-disabled-edit
| Some sites disable edit action for anonymous users and some core pages. This option will ignore this error, and the textarea not found error. You may only get a partial dump. (only works with <code>--content</code>)
|-
| --current-only
| Download only the latest revision of each page
|-
| --retry
| How many times to retry each request before the job fails
|-
| --hard-retry
| How many times to retry hard failures on requests (for example, interrupted connections)
|-
| --content
| Dump content
|-
| --media
| Dump media
|-
| --html
| Dump HTML
|-
| --pdf
| Dump PDFs on wikis with the PDF export plugin
|}

==== Usage Tips ====
* A bot bug will cause jobs to become permanently stuck if <code>--auto</code> and <code>--delay X</code> are used without <code>--threads 1</code>

=== PukiWiki ===
Uses [https://github.com/saveweb/pukiwiki-dumper PukiWiki Dumper] to archive the contents of a PukiWiki wiki. 
* <code>!pw <options></code> will dump a PukiWiki wiki.
{| class="wikitable" 
|-
! Parameter
! Description
|-
| --auto
| Dump: content+media, threads=2, current-only. (threads is overridable)
|-
| --threads
| Number of sub threads to use [default: 1], not recommended to set > 5
|-
| --ignore-action-disabled-edit
| Some sites disable edit action for anonymous users and some core pages. This option will ignore this error, and the textarea not found error. You may only get a partial dump. (only works with <code>--content</code>)
|-
| --trim-php-warnings
| Trim PHP warnings from responses
|-
| --verbose
| verbose output
|-
| --current-only
| Download only the latest revision of each page
|-
| --retry
| How many times to retry each request before the job fails
|-
| --hard-retry
| How many times to retry hard failures on requests (for example, interrupted connections)
|-
| --content
| Dump content
|-
| --media
| Dump media
|}