https://wiki.archiveteam.org/api.php?action=feedcontributions&user=Larsenv&feedformat=atomArchiveteam - User contributions [en]2024-03-29T07:28:16ZUser contributionsMediaWiki 1.37.1https://wiki.archiveteam.org/index.php?title=Photobucket&diff=48537Photobucket2022-05-03T02:20:27Z<p>Larsenv: I wouldn't say photobucket is stable, I'd say it's dying</p>
<hr />
<div>{{Infobox project<br />
| title = Photobucket<br />
| image = Photobucket 1294868817444.png<br />
| description = The Photobucket home page as seen on 2011-01-12<br />
| URL = http://photobucket.com<br />
| project_status = {{online}}<br />
| archiving_status = {{notsavedyet}}<br />
}}<br />
<br />
'''Photobucket''' is an image hosting service. [http://www.washingtonpost.com/business/technology/twitter-partners-with-photobucket-on-photos-and-firefox-on-search/2011/06/01/AGVDSVGH_story.html They are also hosting Twitter images].<br />
<br />
== Vital signs ==<br />
<br />
Had a major outage though around year turnover from 2019 to 2020: https://thedeadpixelssociety.com/power-outage-impacts-photobucket-over-key-holiday-period/<br />
<br />
== Archiving Photobucket ==<br />
<br />
[https://github.com/bibanon/PB_Shovel/ PB_Shovel] can be used to download all images from an album and all subalbums (recursively).<br />
<br />
This version of PB_Shovel has also been modified with a `--links-only` option to crawl a list of every single URLs from the album and subalbums instead of downloading. This makes it possible to use [[wget]] to grab all the files yourself, or [https://github.com/ludios/grab-site grab-site] to archive with WARC.<br />
<br />
Usage: Obtain all the urls of a Photobucket Album, even subalbums (using the -r parameter), and put them in links-<datetime>.txt. This url file can be given to wget or grab-site to download.<br />
<br />
python pb_shovel.py 'http://s160.photobucket.com/user/Spinningfox/library/Internet Fads' -r --links-only<br />
wget -i links-2016-03-20_02-03-01.txt<br />
grab-site -i links-2016-03-20_02-03-01.txt<br />
<br />
<br />
== External links ==<br />
* http://photobucket.com<br />
* https://www.thephotoforum.com/threads/how-to-get-around-photobuckets-watermarks-on-your-images.438126/ - how to modify URLs in order to archive images without watermarks.<br />
<br />
{{Navigation box}}<br />
<br />
[[Category:Image hosting]]</div>Larsenvhttps://wiki.archiveteam.org/index.php?title=Nintendo_eShop&diff=48316Nintendo eShop2022-02-24T05:31:39Z<p>Larsenv: </p>
<hr />
<div>There have been efforts to archive Nintendo eShop contents with [[ArchiveBot]]. They include:<br />
<br />
{| class="wikitable"<br />
!colspan="1"|ArchiveBot Job ID<br />
!colspan="1"|Date<br />
!colspan="1"|Content Archived<br />
!colspan="1"|Domain<br />
|-<br />
|2nd2ue7ff5793uqbyoqxk02ho<br />
|2021-02-13<br />
|Wii Shop Channel CDN game contents<br />
|ccs.cdn.shop.wii.com<br />
|-<br />
|163w2i786uuk594lklvl6usur<br />
|2021-02-13<br />
|Nintendo DSi Shop CDN game contents<br />
|ccs.cdn.t.shop.nintendowifi.net<br />
|-<br />
|e9qq834ecthy77ffka7sggqgn<br />
|2021-02-13<br />
|Wii U Nintendo eShop CDN game contents<br />
|ccs.wup.shop.nintendo.net<br />
|-<br />
|76l6j49h6460r9j52v2g2mvrr<br />
|2022-02-18<br />
|Nintendo 3DS Nintendo eShop XML metadata<br />
|samurai.ctr.shop.nintendo.net<br />
|-<br />
|bopw5bvt8jl3abz6tgznct5yr<br />
|2022-02-18<br />
|Wii U Nintendo eShop videos and images<br />
|kanzashi-wup.cdn.nintendo.net<br>kanzashi-movie-wup.cdn.nintendo.net<br />
|-<br />
|4bfq34ven52ycye7d23f854g4<br />
|2022-02-19<br />
|Nintendo 3DS Nintendo eShop videos and images<br />
|kanzashi-ctr.cdn.nintendo.net<br>kanzashi-movie-ctr.cdn.nintendo.net<br />
|-<br />
|4jtfugxg9g8jwkeoq03zq4rjn<br />
|2022-02-21<br />
|Wii U Nintendo eShop delisted videos and images<br />
|kanzashi-wup.cdn.nintendo.net<br>kanzashi-movie-wup.cdn.nintendo.net<br />
|-<br />
|175jcxi8jvitz3ufycb0l8r0<br />
|2022-02-22<br />
|Wii U Nintendo eShop CDN game contents missing from Archive.org. Aborted to make room for more urgent jobs.<br />
|ccs.wup.shop.nintendo.net<br />
|}<br />
<br />
== References ==<br />
* [https://www.3dbrew.org/wiki/EShop Nintendo 3DS Nintendo eShop information]<br />
* [https://dsibrew.org/wiki/Nintendo_CDN_Files Nintendo CDN Files]<br />
* [[Wii Shop Channel]]</div>Larsenvhttps://wiki.archiveteam.org/index.php?title=Nintendo_eShop&diff=48309Nintendo eShop2022-02-22T18:54:06Z<p>Larsenv: </p>
<hr />
<div>There have been efforts to archive Nintendo eShop contents with [[ArchiveBot]]. They include:<br />
<br />
{| class="wikitable"<br />
!colspan="1"|ArchiveBot Job ID<br />
!colspan="1"|Date<br />
!colspan="1"|Content Archived<br />
!colspan="1"|Domain<br />
|-<br />
|2nd2ue7ff5793uqbyoqxk02ho<br />
|2021-02-13<br />
|Wii Shop Channel CDN game contents<br />
|ccs.cdn.shop.wii.com<br />
|-<br />
|163w2i786uuk594lklvl6usur<br />
|2021-02-13<br />
|Nintendo DSi Shop CDN game contents<br />
|ccs.cdn.t.shop.nintendowifi.net<br />
|-<br />
|e9qq834ecthy77ffka7sggqgn<br />
|2021-02-13<br />
|Wii U Nintendo eShop CDN game contents<br />
|ccs.wup.shop.nintendo.net<br />
|-<br />
|76l6j49h6460r9j52v2g2mvrr<br />
|2022-02-18<br />
|Nintendo 3DS Nintendo eShop XML metadata<br />
|samurai.ctr.shop.nintendo.net<br />
|-<br />
|bopw5bvt8jl3abz6tgznct5yr<br />
|2022-02-18<br />
|Wii U Nintendo eShop videos and images<br />
|kanzashi-wup.cdn.nintendo.net<br>kanzashi-movie-wup.cdn.nintendo.net<br />
|-<br />
|4bfq34ven52ycye7d23f854g4<br />
|2022-02-19<br />
|Nintendo 3DS Nintendo eShop videos and images<br />
|kanzashi-ctr.cdn.nintendo.net<br>kanzashi-movie-ctr.cdn.nintendo.net<br />
|-<br />
|4jtfugxg9g8jwkeoq03zq4rjn<br />
|2022-02-21<br />
|Wii U Nintendo eShop delisted videos and images<br />
|kanzashi-wup.cdn.nintendo.net<br>kanzashi-movie-wup.cdn.nintendo.net<br />
|-<br />
|175jcxi8jvitz3ufycb0l8r0<br />
|2022-02-22<br />
|Wii U Nintendo eShop CDN game contents missing from Archive.org<br />
|ccs.wup.shop.nintendo.net<br />
|}<br />
<br />
== References ==<br />
* [https://www.3dbrew.org/wiki/EShop Nintendo 3DS Nintendo eShop information]<br />
* [https://dsibrew.org/wiki/Nintendo_CDN_Files Nintendo CDN Files]<br />
* [[Wii Shop Channel]]</div>Larsenvhttps://wiki.archiveteam.org/index.php?title=Nintendo_eShop&diff=48305Nintendo eShop2022-02-21T18:59:25Z<p>Larsenv: </p>
<hr />
<div>There have been efforts to archive Nintendo eShop contents with [[ArchiveBot]]. They include:<br />
<br />
{| class="wikitable"<br />
!colspan="1"|ArchiveBot Job ID<br />
!colspan="1"|Date<br />
!colspan="1"|Content Archived<br />
!colspan="1"|Data Source<br />
!colspan="1"|Domain<br />
|-<br />
|2nd2ue7ff5793uqbyoqxk02ho<br />
|2021-02-13<br />
|Wii Shop Channel CDN game contents<br />
|Galaxy<br />
|ccs.cdn.shop.wii.com<br />
|-<br />
|163w2i786uuk594lklvl6usur<br />
|2021-02-13<br />
|Nintendo DSi Shop CDN game contents<br />
|Galaxy<br />
|ccs.cdn.t.shop.nintendowifi.net<br />
|-<br />
|e9qq834ecthy77ffka7sggqgn<br />
|2021-02-13<br />
|Wii U Nintendo eShop CDN game contents<br />
|Galaxy <br />
|ccs.wup.shop.nintendo.net<br />
|-<br />
|76l6j49h6460r9j52v2g2mvrr<br />
|2022-02-18<br />
|Nintendo 3DS Nintendo eShop XML metadata<br />
|Maschell<br />
|samurai.ctr.shop.nintendo.net<br />
|-<br />
|bopw5bvt8jl3abz6tgznct5yr<br />
|2022-02-18<br />
|Wii U Nintendo eShop videos and images<br />
|Maschell<br />
|kanzashi-wup.cdn.nintendo.net<br>kanzashi-movie-wup.cdn.nintendo.net<br />
|-<br />
|4bfq34ven52ycye7d23f854g4<br />
|2022-02-19<br />
|Nintendo 3DS Nintendo eShop videos and images<br />
|Maschell<br />
|kanzashi-ctr.cdn.nintendo.net<br>kanzashi-movie-ctr.cdn.nintendo.net<br />
|-<br />
|4jtfugxg9g8jwkeoq03zq4rjn<br />
|2022-02-21<br />
|Wii U Nintendo eShop delisted videos and images<br />
|Maschell<br />
|kanzashi-wup.cdn.nintendo.net<br>kanzashi-movie-wup.cdn.nintendo.net<br />
|}<br />
<br />
== References ==<br />
* [https://www.3dbrew.org/wiki/EShop Nintendo 3DS Nintendo eShop information]<br />
* [https://dsibrew.org/wiki/Nintendo_CDN_Files Nintendo CDN Files]<br />
* [[Wii Shop Channel]]</div>Larsenvhttps://wiki.archiveteam.org/index.php?title=Nintendo_eShop&diff=48304Nintendo eShop2022-02-21T18:55:08Z<p>Larsenv: </p>
<hr />
<div>There have been efforts to archive Nintendo eShop contents with [[ArchiveBot]]. They include:<br />
<br />
{| class="wikitable"<br />
!colspan="1"|ArchiveBot Job ID<br />
!colspan="1"|Date<br />
!colspan="1"|Content Archived<br />
!colspan="1"|Data Source<br />
!colspan="1"|Domain<br />
|-<br />
|2nd2ue7ff5793uqbyoqxk02ho<br />
|2021-02-13<br />
|Wii Shop Channel contents<br />
|Galaxy<br />
|ccs.cdn.shop.wii.com<br />
|-<br />
|163w2i786uuk594lklvl6usur<br />
|2021-02-13<br />
|Nintendo DSi Shop contents<br />
|Galaxy<br />
|ccs.cdn.t.shop.nintendowifi.net<br />
|-<br />
|e9qq834ecthy77ffka7sggqgn<br />
|2021-02-13<br />
|Wii U Nintendo eShop contents<br />
|Galaxy <br />
|ccs.wup.shop.nintendo.net<br />
|-<br />
|76l6j49h6460r9j52v2g2mvrr<br />
|2022-02-18<br />
|Nintendo 3DS Nintendo eShop XML metadata<br />
|Maschell<br />
|samurai.ctr.shop.nintendo.net<br />
|-<br />
|bopw5bvt8jl3abz6tgznct5yr<br />
|2022-02-18<br />
|Wii U Nintendo eShop videos and images<br />
|Maschell<br />
|kanzashi-wup.cdn.nintendo.net<br>kanzashi-movie-wup.cdn.nintendo.net<br />
|-<br />
|4bfq34ven52ycye7d23f854g4<br />
|2022-02-19<br />
|Nintendo 3DS Nintendo eShop videos and images<br />
|Maschell<br />
|kanzashi-ctr.cdnnintendo.net<br>kanzashi-movie-ctr.cdn.nintendo.net<br />
|-<br />
|4jtfugxg9g8jwkeoq03zq4rjn<br />
|2022-02-21<br />
|Wii U Nintendo eShop delisted videos and images<br />
|Maschell<br />
|kanzashi-wup.cdn.nintendo.net<br>kanzashi-movie-wup.cdn.nintendo.net<br />
|}</div>Larsenvhttps://wiki.archiveteam.org/index.php?title=Nintendo_eShop&diff=48303Nintendo eShop2022-02-21T18:50:36Z<p>Larsenv: Created page with "There have been efforts to archive Nintendo eShop contents with ArchiveBot. They include: * Job 2nd2ue7ff5793uqbyoqxk02ho - 2021-02-13 - Wii Shop Channel contents from data from Galaxy (ccs.cdn.shop.wii.com) * Job 163w2i786uuk594lklvl6usur - 2021-02-13 - Nintendo DSi Shop contents from data from Galaxy (ccs.cdn.t.shop.nintendowifi.net) * Job e9qq834ecthy77ffka7sggqgn - 2021-02-13 - Wii U Nintendo eShop contents from data from Galaxy (ccs.wup.shop.nintendo.net) * J..."</p>
<hr />
<div>There have been efforts to archive Nintendo eShop contents with [[ArchiveBot]]. They include:<br />
<br />
* Job 2nd2ue7ff5793uqbyoqxk02ho - 2021-02-13 - Wii Shop Channel contents from data from Galaxy (ccs.cdn.shop.wii.com)<br />
* Job 163w2i786uuk594lklvl6usur - 2021-02-13 - Nintendo DSi Shop contents from data from Galaxy (ccs.cdn.t.shop.nintendowifi.net)<br />
* Job e9qq834ecthy77ffka7sggqgn - 2021-02-13 - Wii U Nintendo eShop contents from data from Galaxy (ccs.wup.shop.nintendo.net)<br />
* Job 76l6j49h6460r9j52v2g2mvrr - 2022-02-18 - Nintendo 3DS Nintendo eShop XML metadata from data from Maschell (samurai.ctr.shop.nintendo.net)<br />
* Job bopw5bvt8jl3abz6tgznct5yr - 2022-02-18 - Wii U Nintendo eShop videos and images from data from Maschell (kanzashi-wup.cdn.nintendo.net and kanzashi-movie-wup.cdn.nintendo.net)<br />
* Job 4bfq34ven52ycye7d23f854g4 - 2022-02-19 - Nintendo 3DS Nintendo eShop videos and images from data from Maschell (kanzashi-ctr.cdnnintendo.net and kanzashi-movie-ctr.cdn.nintendo.net)<br />
* Job 4jtfugxg9g8jwkeoq03zq4rjn - 2022-02-21 - Wii U Nintendo eShop delisted videos and images from data from Maschell (kanzashi-wup.cdn.nintendo.net and kanzashi-movie-wup.cdn.nintendo.net)</div>Larsenvhttps://wiki.archiveteam.org/index.php?title=ArchiveBot&diff=45660ArchiveBot2020-10-13T15:28:33Z<p>Larsenv: </p>
<hr />
<div>[[File:Librarianmotoko.jpg|200px|right|thumb|Imagine Motoko Kusanagi as an archivist.]]<br />
<br />
'''ArchiveBot''' is an [[IRC]] bot designed to automate the archival of smaller websites (e.g. up to a few hundred thousand URLs). You give it a URL to start at, and it grabs all content under that URL, records it in a [[WARC]] file, and then uploads that WARC to ArchiveTeam servers for eventual injection into the [https://archive.org/details/archivebot Internet Archive]'s Wayback Machine (or other archive sites).<br />
<br />
== Details ==<br />
To use ArchiveBot, drop by the IRC channel '''#archivebot''' on hackint. To interact with ArchiveBot, you issue [https://archivebot.readthedocs.io/en/latest/commands.html '''commands'''] by typing them into the channel. Note that you will need channel operator (<code>@</code>) or voice (<code>+</code>) permissions in order to issue archiving jobs; please ask for assistance or leave a message describing the website you want to archive.<br />
<br />
The [http://dashboard.at.ninjawedding.org/3 '''ArchiveBot dashboard'''] publicly shows the sites being currently downloaded. The [http://archivebot.at.ninjawedding.org:4567/pipelines pipeline monitor station] shows the status of deployed instances of crawlers. The [http://archive.fart.website/archivebot/viewer/ viewer] assists in browsing and searching archives.<br />
<br />
== Components ==<br />
IRC interface<br />
:The bot listens for commands in the IRC channel and then reports back status on the IRC channel. You can ask it to archive a whole website or single webpage, check whether the URL has been saved, change the delay time between requests, or add some ignore rules to avoid crawling certain web cruft. This IRC interface is collaborative, meaning anyone with permission can adjust the parameter of jobs. Note that the bot isn't a chat bot so it will ignore you if it doesn't understand a command.<br />
<br />
Dashboard<br />
:The [http://dashboard.at.ninjawedding.org/3 '''ArchiveBot dashboard'''] is a web-based front-end displaying the URLs being downloaded by the various web crawls. Each URL line in the dashboard is categorized by its HTTP code into successes, warnings, and errors. It will be highlighted in yellow or red. The dashboard also provides RSS feeds and [http://dashboard.at.ninjawedding.org/pending a list of pending jobs].<br />
<br />
Backend<br />
:The backend contains the database of all jobs and several maintenance tasks such as trimming logs and posting Tweets on Twitter. The backend is the centralized portion of ArchiveBot.<br />
<br />
Crawler<br />
:The crawler will download and spider the website into WARC files. The crawler is the distributed portion of ArchiveBot. Volunteers run pipeline nodes connected to the backend. The backend will tell the nodes/pipelines what jobs to run. Once the crawl job has finished, the pipeline reports back to the backend and uploads the WARC files to the staging server. This process is handled by a supervisor script called a pipeline.<br />
<br />
Staging server<br />
:The staging server, known as [[Fortress of Solitude|Fortress of Solitude (FOS)]], is the place where all the WARC files are temporarily uploaded. Once the current batch has been approved, the files will be uploaded to the Internet Archive for consumption by the Wayback Machine.<br />
<br />
== Source Code ==<br />
<br />
ArchiveBot's source code can be found at https://github.com/ArchiveTeam/ArchiveBot. [[Dev|Contributions welcomed]]! Any issues or feature requests may be filed at [https://github.com/ArchiveTeam/ArchiveBot/issues the issue tracker].<br />
<br />
== People ==<br />
The main server that controls the IRC bot, pipeline manager backend, and web dashboard is operated by [[User:yipdw|yipdw]], although a few other ArchiveTeam members were given SSH access in late 2017. The staging server [[Fortress of Solitude|Fortress of Solitude (FOS)]], where the data sits for final checks before being moved over to the Internet Archive serves, is operated by [[User:jscott|SketchCow]]. The pipelines are operated by various volunteers around the world. Each pipeline typically runs two or three web crawl jobs at any given time.<br />
<br />
== Volunteer to run a Pipeline ==<br />
As of November 2017, ArchiveBot has again started accepting applications from volunteers who want to set up new pipelines. You'll need to have a machine with:<br />
<br />
* lots of disk space (40 GB minimum / 200 GB recommended / 500 GB atypical)<br />
* 512 MB RAM (2 GB recommended, 2 GB swap recommended)<br />
* 10 Mb/s upload/download speeds (100 Mb/s recommended)<br />
* long-term availability (2 months minimum)<br />
* always-on unrestricted internet access (absolutely no firewall/proxies/censorship/ISP-injected-ads/DNS-redirection/free-cafe-wifi)<br />
<br />
Suggestion: the $40/month Digital Ocean droplets (4 GB memory/2 CPU/60 GB hard drive) running Ubuntu work pretty well.<br />
<br />
Note that we currently only accept pipelines from people who have been active on ArchiveTeam for a while.<br />
<br />
If you have a suitable server available and would like to volunteer, please review the [https://github.com/ArchiveTeam/ArchiveBot/blob/master/INSTALL.pipeline Pipeline Install] instructions. Then contact ArchiveTeam members [[User:Asparagirl|Asparagirl]], [[User:astrid|astrid]], [[User:JustAnotherArchivist|JAA]], [[User:yipdw|yipdw]], or other ArchiveTeam members hanging out in #archivebot, and we can hook you up, adding your machine to the list of approved pipelines, so that it will start processing incoming ArchiveBot jobs.<br />
<br />
=== Caveats ===<br />
As of August 2018, there are a few things you need to be aware of when operating an ArchiveBot pipeline:<br />
<br />
* '''Never, ever press ^C on the pipeline.''' Use <code>touch STOP</code> in the <code>ArchiveBot/pipeline</code> directory instead to stop the pipeline.<br />
* Please give access to the pipeline for maintenance work when you're away (e.g. holidays, busy IRL) to someone who's around frequently. This is to avoid situations where jobs or pipelines are stuck for weeks or months without anyone being able to intervene.<br />
* Jobs that crash with an error need to be killed manually using <code>kill -9</code>.<br />
* The log files of jobs that are aborted or crash are not uploaded to the Internet Archive. Please keep the temporary <code>tmp-wpull-*.log.gz</code> files in the pipeline directory, rename them so the filename follows the same format as the JSON file (with extension <code>.log.gz</code> instead of <code>.json</code>), and upload them to FOS manually.<br />
** You can find the job ID for these files in the second line.<br />
** Finding the correct filename can be a bit tricky. You can use the viewer or the [https://github.com/JustAnotherArchivist/archivebot-archives archivebot-archives] repository. Keep in mind that the timestamp in the filename should approximately match the one at the beginning of the log file, though there is usually a difference between the two of at least a few seconds (the log file timestamps being later than the filename timestamp).<br />
** Be careful with the filename if there were multiple jobs for the same URL (i.e. the same job ID).<br />
** Here is a public gist on GitHub explaining step by step how to find the proper log file for your crashed or killed job, how to properly rename it, and how to rsync it up to FOS: [[https://gist.github.com/Asparagirl/155bd3c8ee4b8ad5ed737e45bcad1a5a]]<br />
** Contact [[User:JustAnotherArchivist]] if you need help with this.<br />
* Due to a bug somewhere deep in the network stack, connections get stuck from time to time. This causes jobs to slow down or halt entirely.<br />
** As a workaround, you can use the [https://github.com/JustAnotherArchivist/kill-wpull-connections kill-wpull-connections] script; it requires pgrep, lsof, and gdb. Depending on the machine configuration (specifically, the value of <code>kernel.yama.ptrace_scope</code> in <code>/proc/sys/kernel/yama/ptrace_scope</code>), it may also require root/sudo privileges.<br />
** In very rare cases, you may need to use [http://killcx.sourceforge.net/ killcx] to close the connections.<br />
** [https://github.com/kristrev/tcp_closer tcp_closer] works even when the two methods above fail. It uses the SOCK_DESTROY kernel operation provided by Linux >= 4.5.<br />
* Also due to a bug suspected to be in the network stack, wpull processes sometimes use a lot of RAM (and CPU). If a process uses more than 300 MB continuously, that's likely the case. kill-wpull-connections seems to "fix" this issue, though it takes a while (minutes, rarely even an hour or more) from running the script until the usage actually drops down.<br />
** If wpull paused due to high RAM usage try creating a swap file and forcing RAM pages to swap. wpull only checks RAM usage.<br />
<pre>dd if=/dev/zero of=swapfile bs=1024 count=1024000<br />
mkswap swapfile<br />
swapon swapfile<br />
perl -e '$tmp = "a" x 999999999'<br />
swapoff swapfile<br />
rm swapfile</pre><br />
* Make sure that you don't have any <code>search</code> or <code>domain</code> line in <code>/etc/resolv.conf</code>. We've grabbed a number of copies of the websites of OVH and Online.net as a result of such lines and broken <code>http://www/</code> links... (Cf [https://github.com/ArchiveTeam/ArchiveBot/issues/318 this issue on GitHub])<br />
<br />
== Installation ==<br />
Installing the ArchiveBot can be difficult. The [https://github.com/ArchiveTeam/ArchiveBot/blob/master/INSTALL.pipeline Pipeline Install] instructions are online, but are tricky.<br />
<br />
But there is a [https://github.com/ArchiveTeam/ArchiveBot/blob/master/.travis.yml Travis.yml automated install script] for [https://travis-ci.org/ArchiveTeam/ArchiveBot Travis-cl] that is designed to test the ArchiveBot. <br />
<br />
Since it's good enough for testing... it's good enough for installation, right? There must be a way to convert it into an installer script.<br />
<br />
== Disclaimers ==<br />
# Everything is provided on a best-effort basis; nothing is guaranteed to work. (We're volunteers, not a support team.)<br />
# We can decide to stop a job or ban a user if a job is deemed unnecessary. (We don't want to run up operator bandwidth bills and waste Internet Archive donations on costs.)<br />
# We're not Internet Archive. (We do what we want.)<br />
# We're not the Wayback Machine. Specifically, we are not <code>ia_archiver</code> or <code>archive.org_bot</code>. (We don't run crawlers on behalf of other crawlers.)<br />
<br />
Occasionally, we had to ban blocks of IP addresses from the channel. If you think a ban does not apply to you but cannot join the #archivebot channel, please join the main #archiveteam channel instead.<br />
<br />
== Bad behavior ==<br />
If you are a website operator and you notice ArchiveBot misbehaving, please contact us on #archivebot or #archiveteam on EFnet (see top of page for links).<br />
<br />
ArchiveBot understands [[robots.txt]] (please read the article) but does not match any directives. It uses it for discovering more links such as sitemaps however.<br />
<br />
Also, please remember that '''we are not the [[Internet Archive|Internet Archive]]'''.<br />
<br />
== Trivia ==<br />
* One of the ArchiveBot commands, <code>!yahoo</code>, has been named after [[Yahoo!]]. This command makes the bot archive the page in a more aggressive manner to speed up the archival process.<br />
<br />
== Usage, Dashboards, and Completed Job Viewer ==<br />
<br />
{| class="wikitable sortable"<br />
|-<br />
! Function !! URL<br />
|-<br />
| ArchiveBot documentation, usage guide, manual || http://archivebot.rtfd.io/<br />
|-<br />
| ArchiveBot traditional dashboard - shows currently active jobs || http://dashboard.at.ninjawedding.org<br />
|-<br />
| ArchiveBot newer-style dashboard - shows currently active jobs || http://dashboard.at.ninjawedding.org/3<br />
|-<br />
| ArchiveBot dashboard - shows pending jobs || http://dashboard.at.ninjawedding.org/pending<br />
|-<br />
| ArchiveBot dashboard - shows ignores for a specific job || http://dashboard.at.ninjawedding.org/ignores/<JOBID><br />
|-<br />
| ArchiveBot Pipeline Monitor Station - shows the status of the deployed crawler pipeline || http://dashboard.at.ninjawedding.org/pipelines<br />
|-<br />
| ArchiveBot Viewer - explore previously archived items || https://archive.fart.website/archivebot/viewer/<br />
|-<br />
| ChromeBot dashboard - shows pending, running, and finished jobs || http://chromebot.6xq.net/<br />
|}<br />
<br />
== Suggested things to focus on archiving with ArchiveBot ==<br />
* [[Company acquisitions and mergers]] - companies that are being, or have been, acquired or merged<br />
* [[ISP Hosting]] - user homepages on ISPs that have not previously been saved<br />
<br />
== Related Links ==<br />
* http://archivebot.com is a DNS alias for the ArchiveBot dashboard<br />
* [[Chromebot]] is an IRC bot parallel to ArchiveBot that uses Google Chrome and thus is able to archive JavaScript-heavy and pages with endless scrolling. It is available in the #archivebot channel.<br />
* https://twitter.com/ArchiveBot - dormant current Twitter feed of ArchiveBot activity. Tweets may lag live dashboard. Dormant since 2018-04-12.<br />
* https://twitter.com/ATArchiveBot - former Twitter feed of ArchiveBot activity. Last used 2014-07-28 as @ATArchiveBot was replaced by @ArchiveBot.<br />
* https://archive.org/details/archivebot - ArchiveTeam ArchiveBot collection at the Internet Archive<br />
<br />
== More ==<br />
Like ArchiveBot? Check out our [[Main_Page|homepage]] and other [[projects]]!<br />
<br />
== Notes ==<br />
<references/><br />
<br />
{{archivebot}}<br />
{{navigation_box}}<br />
<br />
[[Category:Bots]]</div>Larsenvhttps://wiki.archiveteam.org/index.php?title=Move_Archiveteam_to_Hackint&diff=45429Move Archiveteam to Hackint2020-08-26T22:22:13Z<p>Larsenv: </p>
<hr />
<div>__NOTOC__<br />
<big>PROPOSED: MOVE ARCHIVETEAM IRC COMMUNICATION PRIMARILY TO HACKINT FROM EFNET</big><br />
<br />
Proposed by JAA a good long while ago, the proposal is to move Archive Team's IRC channels (and many project sub-channels) from EFNet to HackINT.<br />
<br />
As is typical, we're currently split between the two networks, with many channels in HackInt and many others in EFNet, depending on the preferences and inclinations of various members. Honestly, this can't continue. As most activity is happening in Hackint anyway, and because we might as well use this for a quorum discussion, this page exists for discussion (along with the talk page) *NOW* (Mid-August) to September 30th, at which point it will (hopefully) be very clear which direction we should go. This page is likely to get increased changes and traffic as time goes, so check back often.<br />
<br />
* Information on hackint is here: https://www.hackint.org/<br />
* Information on EFNet is here: http://www.efnet.org/<br />
<br />
== Arguments for moving Archive Team to Hackint ==<br />
* IRC Services: No need to micromanage ops in each project channel, ease of administration<br />
* Stability: Handful of netsplits that lasted just a few minutes on Hackint compared to the ones on EFnet<br />
* Support: IRC staff has proven to be very helpful and useful and helped us get running with channel and user administration<br />
* Limits: Per-connection channel limits are much higher on Hackint and do not differ across servers (123 is the current joined-channels per connection limit and opers are willing to increase this network-wide limit in case we hit it)<br />
<br />
== Arguments for keeping Archive Team on EFnet ==<br />
* IRC Services: Inherently leads to a certralisation of permissions vs current system.<br />
* It has Always Been EFNet, we shouldn't uproot our long-standing relationship and work with that network.<br />
* EFNet is the longest-lived Network, showing it's here to stay.<br />
* We should just engage with EFNet to make them more hospitable for Archive Team needs.<br />
<br />
== How Moving to HackInt would Work ==<br />
<br />
Jason says "There would almost certainly be a #archiveteam channel on EFNet forever, with some people sitting in related long-time channels like #archiveteam-bs and #archiveteam-ot - but project channels would shut down and move to HackInt. So we'd still have a split, but the channels on EFNet would be more like either social hangouts or represent outreach to guide people to the other location."<br />
<br />
== Signatories ==<br />
<br />
* This is not a vote; this is a show of support in one direction.<br />
<br />
Edit and add your name to one of the lists below if you have a strong opinion one way or another. Describe your thinking, if you'd like. Do not add others if they are not on the Wiki.<br />
<br />
=== In Favor of Move to Hackint ===<br />
* [[User:Kiska]]<br />
* [[User:wessel1512]]<br />
* [[User:Aoede]]<br />
* [[User:Fusl]]<br />
* [[User:Katocala]]<br />
* [[User:ivan]]<br />
* [[User:JAA]] – My [[User:JustAnotherArchivist/hackint vs EFnet|comparison of the two networks]] (= why I think hackint is vastly superior to EFnet); my [[Talk:Move Archiveteam to Hackint#Counterpoints to the pro-EFnet arguments|counterpoints to the pro-EFnet arguments]]<br />
* [[User:Flashfire42]]<br />
* [[User:Jake]]<br />
* [[User:Kaz]]<br />
* [[User:Larsenv]]<br />
* [[User:Maxfan8]]<br />
* [[User:Jrwr]]<br />
* [[User:Craigle]]<br />
* [[User:systwi]]<br />
* [[User:PurpleSymphony]]<br />
* [[User:Sanqui]]<br />
<br />
=== In Favor of Staying at EFNet ===</div>Larsenvhttps://wiki.archiveteam.org/index.php?title=Wii_Shop_Channel&diff=34451Wii Shop Channel2019-02-04T22:20:10Z<p>Larsenv: </p>
<hr />
<div>{{Infobox project<br />
| title = Wii Shop Channel<br />
| logo = Wii Shop Channel Logo.png<br />
| image = Wii Shop Channel Screenshot.PNG<br />
| description = The main menu<br />
| URL = https://oss-auth.shop.wii.com<br />
| project_status = {{closing}}<br />
| archiving_status = {{saved}}<br />
}}<br />
<br />
{{url|https://www.nintendo.co.jp/support/information/2017/0929.html|Nintendo of Japan announced on September 29th, 2017}} that the Wii Shop Channel would be discontinued January 31st, 2019. However, users would no longer be able to add "Wii Points" to their account in order to purchase things on March 27th, 2018.<br />
<br />
Quite a bit of the software available has been added to No-Intro's "Nintendo - Nintendo Wii (DLC)" database, but it is incomplete. Work has been underway and completed to preserve all of the missing titles and they will be released soon (only the Korean version of Yoshi's Cookie is missing).<br />
<br />
It was recently rediscovered how to access the Wii Shop Channel on a PC, so creating a project to download the public-facing version of the site was done. {{url|http://wiibrew.org/wiki/Accessing_Wii_Shop_Channel_From_PC|Originally documented on WiiBrew.}}<br />
<br />
Larsenv has archived all of the HTML storefront he could find and it can be downloaded {{url|https://archive.org/details/WiiShopChannelBackup|on archive.org.}} He also backed up almost all of the CDN and has been datted on No-Intro's "Nintendo - Nintendo Wii (DLC) (CDN)" database.<br />
<br />
Nintendo also announced the shutdown on the Wii Shop Channel itself.<br />
[[File:Wii Shop Channel Shutdown Notice.PNG]]<br />
<br />
The English notice is reproduced in full below.<br />
<br />
Dear Nintendo fans,<br />
<br />
On January 30, 2019, we plan to close the Wii Shop Channel, which has been available on Wii systems since December 2006. We sincerely thank our loyal customers for their support.<br />
<br />
You can still add Wii Points until March 26, 2018, and purchase content on the Wii Shop Channel until January 30, 2019. In the future, we will be closing all services related to the Wii Shop Channel, including redownloading purchased WiiWare, Virtual Console titles, and Wii Channel, as well as Wii System Transfer Tool, which transfers data from Wii to the Wii U system.<br />
<br />
If you have Wii Points to spend, content you want to re-download, or content you’d like to transfer from a Wii system to a Wii U system, we recommend you do so while the services are still available.<br />
<br />
If you have any questions, please see our Q&A.<br />
<br />
Thank you for supporting the Wii Shop Channel and for being such great fans of Nintendo.<br />
<br />
Sincerely,<br />
Your Friends at Nintendo<br />
<br />
== Accessing the Wii Shop Channel on PC ==<br />
First, get "Wii NWC Prod 1" from here: {{url|https://larsenv.github.io/NintendoCerts/index.html}} and install it, certificate password is "alpine"<br />
<br />
Then, change your browser user agent to "Opera/9.30 (Nintendo Wii; U; ; 2077-4; Wii Shop Channel/21.0(A); en)"<br />
<br />
Next, go to these pages in order:<br />
<br />
1. https://oss-auth.shop.wii.com/oss/serv/index.jsp<br />
<br />
2. https://oss-auth.shop.wii.com/oss/serv/CheckRegistered.jsp<br />
<br />
3. https://oss-auth.shop.wii.com/oss/serv/Register.jsp?country=US&region=2&language=en&age=&serial=<br />
<br />
Then try to go to https://oss-auth.shop.wii.com/oss/serv/W_01.jsp?country=US&region=2&language=en&age=&serial= and it should work<br />
<br />
https://oss-auth.shop.wii.com/oss/serv/B_05.jsp?country=US&region=2&language=&age=&serial=&titleId=00050000101d6b00<br />
<br />
Any valid Nintendo title ID works, since it queries the entire Nintendo ecommerce database<br />
<br />
https://oss-auth.shop.wii.com/oss/serv/B_05.jsp?country=US&region=2&language=&age=&serial=&titleId=010000A00218E000<br />
<br />
This is a Switch title, and it doesn't show its name but it shows its title ID, it would show nothing if it didn't exist, so it exists<br />
<br />
==Commands==<br />
Here's a CURL command which will iterate through the 1,679,616 possible Wii Shop title IDs and save the Title Metadata (TMD) file for the latest version if that title ID is valid. This can be used to identify proper shop pages to save if we want to skip the junk results that pull data from the online stores of other Nintendo systems.<br />
<br />
<nowiki>curl -f http://ccs.cdn.shop.wii.com/ccs/download/00010001{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}/tmd --create-dirs --output .\TMDs\00010001#1#2#3#4.tmd</nowiki><br />
<br />
Here is a faster way to do so, using aria2 (multithreaded):<br />
<br />
<nowiki><br />
for i in {30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}<br />
do echo 'http://ccs.cdn.shop.wii.com/ccs/download/00010001'$i'/tmd <br />
out='$i'.tmd'; done | aria2c -i - -j16 -x16<br />
</nowiki><br />
<br />
This example uses 16 threads and no delay. Note that the spaces before out= have to be preserved.<br />
<br />
{{Navigation box}}</div>Larsenvhttps://wiki.archiveteam.org/index.php?title=Wii_Shop_Channel&diff=34438Wii Shop Channel2019-02-03T14:48:08Z<p>Larsenv: </p>
<hr />
<div>{{Infobox project<br />
| title = Wii Shop Channel<br />
| logo = Wii Shop Channel Logo.png<br />
| image = Wii Shop Channel Screenshot.PNG<br />
| description = The main menu<br />
| URL = https://oss-auth.shop.wii.com<br />
| project_status = {{closing}}<br />
| archiving_status = {{saved}}<br />
}}<br />
<br />
{{url|https://www.nintendo.co.jp/support/information/2017/0929.html|Nintendo of Japan announced on September 29th, 2017}} that the Wii Shop Channel would be discontinued January 31st, 2019. However, users would no longer be able to add "Wii Points" to their account in order to purchase things on March 27th, 2018.<br />
<br />
Quite a bit of the software available has been added to No-Intro's "Nintendo - Nintendo Wii (DLC)" database, but it is incomplete. Work has been underway and completed to preserve all of the missing titles and they will be released soon (only the Korean version of Yoshi's Cookie is missing).<br />
<br />
It was recently rediscovered how to access the Wii Shop Channel on a PC, so creating a project to download the public-facing version of the site was done. {{url|http://wiibrew.org/wiki/Accessing_Wii_Shop_Channel_From_PC|Originally documented on WiiBrew.}}<br />
<br />
Larsenv has archived all of the HTML storefront he could find and it can be downloaded {{url|https://archive.org/details/WiiShopChannelBackup|on archive.org.}} He also backed up almost all of the CDN and has been ratted on No-Intro's "Nintendo - Nintendo Wii (DLC) (CDN)" database.<br />
<br />
Nintendo also announced the shutdown on the Wii Shop Channel itself.<br />
[[File:Wii Shop Channel Shutdown Notice.PNG]]<br />
<br />
The English notice is reproduced in full below.<br />
<br />
Dear Nintendo fans,<br />
<br />
On January 30, 2019, we plan to close the Wii Shop Channel, which has been available on Wii systems since December 2006. We sincerely thank our loyal customers for their support.<br />
<br />
You can still add Wii Points until March 26, 2018, and purchase content on the Wii Shop Channel until January 30, 2019. In the future, we will be closing all services related to the Wii Shop Channel, including redownloading purchased WiiWare, Virtual Console titles, and Wii Channel, as well as Wii System Transfer Tool, which transfers data from Wii to the Wii U system.<br />
<br />
If you have Wii Points to spend, content you want to re-download, or content you’d like to transfer from a Wii system to a Wii U system, we recommend you do so while the services are still available.<br />
<br />
If you have any questions, please see our Q&A.<br />
<br />
Thank you for supporting the Wii Shop Channel and for being such great fans of Nintendo.<br />
<br />
Sincerely,<br />
Your Friends at Nintendo<br />
<br />
== Accessing the Wii Shop Channel on PC ==<br />
First, get "Wii NWC Prod 1" from here: {{url|https://larsenv.github.io/NintendoCerts/index.html}} and install it, certificate password is "alpine"<br />
<br />
Then, change your browser user agent to "Opera/9.30 (Nintendo Wii; U; ; 2077-4; Wii Shop Channel/21.0(A); en)"<br />
<br />
Next, go to these pages in order:<br />
<br />
1. https://oss-auth.shop.wii.com/oss/serv/index.jsp<br />
<br />
2. https://oss-auth.shop.wii.com/oss/serv/CheckRegistered.jsp<br />
<br />
3. https://oss-auth.shop.wii.com/oss/serv/Register.jsp?country=US&region=2&language=en&age=&serial=<br />
<br />
Then try to go to https://oss-auth.shop.wii.com/oss/serv/W_01.jsp?country=US&region=2&language=en&age=&serial= and it should work<br />
<br />
https://oss-auth.shop.wii.com/oss/serv/B_05.jsp?country=US&region=2&language=&age=&serial=&titleId=00050000101d6b00<br />
<br />
Any valid Nintendo title ID works, since it queries the entire Nintendo ecommerce database<br />
<br />
https://oss-auth.shop.wii.com/oss/serv/B_05.jsp?country=US&region=2&language=&age=&serial=&titleId=010000A00218E000<br />
<br />
This is a Switch title, and it doesn't show its name but it shows its title ID, it would show nothing if it didn't exist, so it exists<br />
<br />
==Commands==<br />
Here's a CURL command which will iterate through the 1,679,616 possible Wii Shop title IDs and save the Title Metadata (TMD) file for the latest version if that title ID is valid. This can be used to identify proper shop pages to save if we want to skip the junk results that pull data from the online stores of other Nintendo systems.<br />
<br />
<nowiki>curl -f http://ccs.cdn.shop.wii.com/ccs/download/00010001{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}/tmd --create-dirs --output .\TMDs\00010001#1#2#3#4.tmd</nowiki><br />
<br />
Here is a faster way to do so, using aria2 (multithreaded):<br />
<br />
<nowiki><br />
for i in {30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}<br />
do echo 'http://ccs.cdn.shop.wii.com/ccs/download/00010001'$i'/tmd <br />
out='$i'.tmd'; done | aria2c -i - -j16 -x16<br />
</nowiki><br />
<br />
This example uses 16 threads and no delay. Note that the spaces before out= have to be preserved.<br />
<br />
{{Navigation box}}</div>Larsenvhttps://wiki.archiveteam.org/index.php?title=Wii_Shop_Channel&diff=34437Wii Shop Channel2019-02-03T14:46:08Z<p>Larsenv: </p>
<hr />
<div>{{Infobox project<br />
| title = Wii Shop Channel<br />
| logo = Wii Shop Channel Logo.png<br />
| image = Wii Shop Channel Screenshot.PNG<br />
| description = The main menu<br />
| URL = https://oss-auth.shop.wii.com<br />
| project_status = {{closing}}<br />
| archiving_status = {{saved}}<br />
}}<br />
<br />
{{url|https://www.nintendo.co.jp/support/information/2017/0929.html|Nintendo of Japan announced on September 29th, 2017}} that the Wii Shop Channel would be discontinued January 31st, 2019. However, users would no longer be able to add "Wii Points" to their account in order to purchase things on March 27th, 2018.<br />
<br />
Quite a bit of the software available has been added to No-Intro's "Nintendo - Nintendo Wii (DLC)" database, but it is incomplete. Work has been underway and completed to preserve all of the missing titles and they will be released soon (only the Korean version of Yoshi's Cookie is missing).<br />
<br />
It was recently rediscovered how to access the Wii Shop Channel on a PC, so creating a project to download the public-facing version of the site was done. {{url|http://wiibrew.org/wiki/Accessing_Wii_Shop_Channel_From_PC|Originally documented on WiiBrew.}}<br />
<br />
Larsenv has archived all of the HTML storefront he could find and it can be downloaded {{url|https://archive.org/details/WiiShopChannelBackup|on archive.org}}.<br />
<br />
Nintendo also announced the shutdown on the Wii Shop Channel itself.<br />
[[File:Wii Shop Channel Shutdown Notice.PNG]]<br />
<br />
The English notice is reproduced in full below.<br />
<br />
Dear Nintendo fans,<br />
<br />
On January 30, 2019, we plan to close the Wii Shop Channel, which has been available on Wii systems since December 2006. We sincerely thank our loyal customers for their support.<br />
<br />
You can still add Wii Points until March 26, 2018, and purchase content on the Wii Shop Channel until January 30, 2019. In the future, we will be closing all services related to the Wii Shop Channel, including redownloading purchased WiiWare, Virtual Console titles, and Wii Channel, as well as Wii System Transfer Tool, which transfers data from Wii to the Wii U system.<br />
<br />
If you have Wii Points to spend, content you want to re-download, or content you’d like to transfer from a Wii system to a Wii U system, we recommend you do so while the services are still available.<br />
<br />
If you have any questions, please see our Q&A.<br />
<br />
Thank you for supporting the Wii Shop Channel and for being such great fans of Nintendo.<br />
<br />
Sincerely,<br />
Your Friends at Nintendo<br />
<br />
== Accessing the Wii Shop Channel on PC ==<br />
First, get "Wii NWC Prod 1" from here: {{url|https://larsenv.github.io/NintendoCerts/index.html}} and install it, certificate password is "alpine"<br />
<br />
Then, change your browser user agent to "Opera/9.30 (Nintendo Wii; U; ; 2077-4; Wii Shop Channel/21.0(A); en)"<br />
<br />
Next, go to these pages in order:<br />
<br />
1. https://oss-auth.shop.wii.com/oss/serv/index.jsp<br />
<br />
2. https://oss-auth.shop.wii.com/oss/serv/CheckRegistered.jsp<br />
<br />
3. https://oss-auth.shop.wii.com/oss/serv/Register.jsp?country=US&region=2&language=en&age=&serial=<br />
<br />
Then try to go to https://oss-auth.shop.wii.com/oss/serv/W_01.jsp?country=US&region=2&language=en&age=&serial= and it should work<br />
<br />
https://oss-auth.shop.wii.com/oss/serv/B_05.jsp?country=US&region=2&language=&age=&serial=&titleId=00050000101d6b00<br />
<br />
Any valid Nintendo title ID works, since it queries the entire Nintendo ecommerce database<br />
<br />
https://oss-auth.shop.wii.com/oss/serv/B_05.jsp?country=US&region=2&language=&age=&serial=&titleId=010000A00218E000<br />
<br />
This is a Switch title, and it doesn't show its name but it shows its title ID, it would show nothing if it didn't exist, so it exists<br />
<br />
==Commands==<br />
Here's a CURL command which will iterate through the 1,679,616 possible Wii Shop title IDs and save the Title Metadata (TMD) file for the latest version if that title ID is valid. This can be used to identify proper shop pages to save if we want to skip the junk results that pull data from the online stores of other Nintendo systems.<br />
<br />
<nowiki>curl -f http://ccs.cdn.shop.wii.com/ccs/download/00010001{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}/tmd --create-dirs --output .\TMDs\00010001#1#2#3#4.tmd</nowiki><br />
<br />
Here is a faster way to do so, using aria2 (multithreaded):<br />
<br />
<nowiki><br />
for i in {30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}<br />
do echo 'http://ccs.cdn.shop.wii.com/ccs/download/00010001'$i'/tmd <br />
out='$i'.tmd'; done | aria2c -i - -j16 -x16<br />
</nowiki><br />
<br />
This example uses 16 threads and no delay. Note that the spaces before out= have to be preserved.<br />
<br />
{{Navigation box}}</div>Larsenvhttps://wiki.archiveteam.org/index.php?title=Wii_Shop_Channel&diff=30870Wii Shop Channel2018-08-13T14:34:11Z<p>Larsenv: </p>
<hr />
<div>{{Infobox project<br />
| title = Wii Shop Channel<br />
| logo = Wii Shop Channel Logo.png<br />
| image = Wii Shop Channel Screenshot.PNG<br />
| description = The main menu<br />
| URL = https://oss-auth.shop.wii.com<br />
| project_status = {{closing}}<br />
| archiving_status = {{upcoming}}<br />
| source =<br />
| tracker = <br />
| irc = <br />
}}<br />
<br />
{{url|https://www.nintendo.co.jp/support/information/2017/0929.html|Nintendo of Japan announced on September 29th, 2017}} that the Wii Shop Channel would be discontinued January 31st, 2019. However, users would no longer be able to add "Wii Points" to their account in order to purchase things on March 27th, 2018.<br />
<br />
Quite a bit of the software available has been added to No-Intro's "Nintendo - Nintendo Wii (DLC)" database, but it is incomplete. Work has been underway and completed to preserve all of the missing titles and they will be released soon (only the Korean version of Yoshi's Cookie is missing).<br />
<br />
Work has also been underway to archive the whole storefront, however that's not yet released earlier. Larsenv has archived a chunk of the data himself.<br />
<br />
In addition, it was recently rediscovered how to access the Wii Shop Channel on a PC, so creating a project to download the public-facing version of the site is possible. {{url|http://wiibrew.org/wiki/Accessing_Wii_Shop_Channel_From_PC|Originally documented on WiiBrew.}}<br />
<br />
Nintendo also announced the shutdown on the Wii Shop Channel itself.<br />
[[File:Wii Shop Channel Shutdown Notice.PNG]]<br />
<br />
The English notice is reproduced in full below.<br />
<br />
Dear Nintendo fans,<br />
<br />
On January 30, 2019, we plan to close the Wii Shop Channel, which has been available on Wii systems since December 2006. We sincerely thank our loyal customers for their support.<br />
<br />
You can still add Wii Points until March 26, 2018, and purchase content on the Wii Shop Channel until January 30, 2019. In the future, we will be closing all services related to the Wii Shop Channel, including redownloading purchased WiiWare, Virtual Console titles, and Wii Channel, as well as Wii System Transfer Tool, which transfers data from Wii to the Wii U system.<br />
<br />
If you have Wii Points to spend, content you want to re-download, or content you’d like to transfer from a Wii system to a Wii U system, we recommend you do so while the services are still available.<br />
<br />
If you have any questions, please see our Q&A.<br />
<br />
Thank you for supporting the Wii Shop Channel and for being such great fans of Nintendo.<br />
<br />
Sincerely,<br />
Your Friends at Nintendo<br />
<br />
== Accessing the Wii Shop Channel on PC ==<br />
First, get "Wii NWC Prod 1" from here: {{url|https://larsenv.github.io/NintendoCerts/index.html}} and install it, certificate password is "alpine"<br />
<br />
Then, change your browser user agent to "Opera/9.30 (Nintendo Wii; U; ; 2077-4; Wii Shop Channel/21.0(A); en)"<br />
<br />
Next, go to these pages in order:<br />
<br />
1. https://oss-auth.shop.wii.com/oss/serv/index.jsp<br />
<br />
2. https://oss-auth.shop.wii.com/oss/serv/CheckRegistered.jsp<br />
<br />
3. https://oss-auth.shop.wii.com/oss/serv/Register.jsp?country=US&region=2&language=en&age=&serial=<br />
<br />
Then try to go to https://oss-auth.shop.wii.com/oss/serv/W_01.jsp?country=US&region=2&language=en&age=&serial= and it should work<br />
<br />
https://oss-auth.shop.wii.com/oss/serv/B_05.jsp?country=US&region=2&language=&age=&serial=&titleId=00050000101d6b00<br />
<br />
Any valid Nintendo title ID works, since it queries the entire Nintendo ecommerce database<br />
<br />
https://oss-auth.shop.wii.com/oss/serv/B_05.jsp?country=US&region=2&language=&age=&serial=&titleId=010000A00218E000<br />
<br />
This is a Switch title, and it doesn't show its name but it shows its title ID, it would show nothing if it didn't exist, so it exists<br />
<br />
==Commands==<br />
Here's a CURL command which will iterate through the 1,679,616 possible Wii Shop title IDs and save the Title Metadata (TMD) file for the latest version if that title ID is valid. This can be used to identify proper shop pages to save if we want to skip the junk results that pull data from the online stores of other Nintendo systems.<br />
<br />
<nowiki>curl -f http://ccs.cdn.shop.wii.com/ccs/download/00010001{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}/tmd --create-dirs --output .\TMDs\00010001#1#2#3#4.tmd</nowiki><br />
<br />
Here is a faster way to do so, using aria2 (multithreaded):<br />
<br />
<nowiki><br />
for i in {30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}<br />
do echo 'http://ccs.cdn.shop.wii.com/ccs/download/00010001'$i'/tmd <br />
out='$i'.tmd'; done | aria2c -i - -j16 -x16<br />
</nowiki><br />
<br />
This example uses 16 threads and no delay. Note that the spaces before out= have to be preserved.<br />
<br />
{{Navigation box}}</div>Larsenvhttps://wiki.archiveteam.org/index.php?title=Wii_Shop_Channel&diff=30869Wii Shop Channel2018-08-13T14:33:10Z<p>Larsenv: </p>
<hr />
<div>{{Infobox project<br />
| title = Wii Shop Channel<br />
| logo = Wii Shop Channel Logo.png<br />
| image = Wii Shop Channel Screenshot.PNG<br />
| description = The main menu<br />
| URL = https://oss-auth.shop.wii.com<br />
| project_status = {{closing}}<br />
| archiving_status = {{upcoming}}<br />
| source =<br />
| tracker = <br />
| irc = <br />
}}<br />
<br />
{{url|https://www.nintendo.co.jp/support/information/2017/0929.html|Nintendo of Japan announced on September 29th, 2017}} that the Wii Shop Channel would be discontinued January 31st, 2019. However, users would no longer be able to add "Wii Points" to their account in order to purchase things on March 27th, 2018.<br />
<br />
Quite a bit of the software available has been added to No-Intro's "Nintendo - Nintendo Wii (DLC)" database, but it is incomplete. Work has been underway and completed to preserve all of the missing titles and they will be released soon (only the Korean version of Yoshi's Cookie is missing).<br />
<br />
In addition, it was recently rediscovered how to access the Wii Shop Channel on a PC, so creating a project to download the public-facing version of the site is possible. {{url|http://wiibrew.org/wiki/Accessing_Wii_Shop_Channel_From_PC|Originally documented on WiiBrew.}}<br />
<br />
Nintendo also announced the shutdown on the Wii Shop Channel itself.<br />
[[File:Wii Shop Channel Shutdown Notice.PNG]]<br />
<br />
The English notice is reproduced in full below.<br />
<br />
Dear Nintendo fans,<br />
<br />
On January 30, 2019, we plan to close the Wii Shop Channel, which has been available on Wii systems since December 2006. We sincerely thank our loyal customers for their support.<br />
<br />
You can still add Wii Points until March 26, 2018, and purchase content on the Wii Shop Channel until January 30, 2019. In the future, we will be closing all services related to the Wii Shop Channel, including redownloading purchased WiiWare, Virtual Console titles, and Wii Channel, as well as Wii System Transfer Tool, which transfers data from Wii to the Wii U system.<br />
<br />
If you have Wii Points to spend, content you want to re-download, or content you’d like to transfer from a Wii system to a Wii U system, we recommend you do so while the services are still available.<br />
<br />
If you have any questions, please see our Q&A.<br />
<br />
Thank you for supporting the Wii Shop Channel and for being such great fans of Nintendo.<br />
<br />
Sincerely,<br />
Your Friends at Nintendo<br />
<br />
== Accessing the Wii Shop Channel on PC ==<br />
First, get "Wii NWC Prod 1" from here: {{url|https://larsenv.github.io/NintendoCerts/index.html}} and install it, certificate password is "alpine"<br />
<br />
Then, change your browser user agent to "Opera/9.30 (Nintendo Wii; U; ; 2077-4; Wii Shop Channel/21.0(A); en)"<br />
<br />
Next, go to these pages in order:<br />
<br />
1. https://oss-auth.shop.wii.com/oss/serv/index.jsp<br />
<br />
2. https://oss-auth.shop.wii.com/oss/serv/CheckRegistered.jsp<br />
<br />
3. https://oss-auth.shop.wii.com/oss/serv/Register.jsp?country=US&region=2&language=en&age=&serial=<br />
<br />
Then try to go to https://oss-auth.shop.wii.com/oss/serv/W_01.jsp?country=US&region=2&language=en&age=&serial= and it should work<br />
<br />
https://oss-auth.shop.wii.com/oss/serv/B_05.jsp?country=US&region=2&language=&age=&serial=&titleId=00050000101d6b00<br />
<br />
Any valid Nintendo title ID works, since it queries the entire Nintendo ecommerce database<br />
<br />
https://oss-auth.shop.wii.com/oss/serv/B_05.jsp?country=US&region=2&language=&age=&serial=&titleId=010000A00218E000<br />
<br />
This is a Switch title, and it doesn't show its name but it shows its title ID, it would show nothing if it didn't exist, so it exists<br />
<br />
==Commands==<br />
Here's a CURL command which will iterate through the 1,679,616 possible Wii Shop title IDs and save the Title Metadata (TMD) file for the latest version if that title ID is valid. This can be used to identify proper shop pages to save if we want to skip the junk results that pull data from the online stores of other Nintendo systems.<br />
<br />
<nowiki>curl -f http://ccs.cdn.shop.wii.com/ccs/download/00010001{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}/tmd --create-dirs --output .\TMDs\00010001#1#2#3#4.tmd</nowiki><br />
<br />
Here is a faster way to do so, using aria2 (multithreaded):<br />
<br />
<nowiki><br />
for i in {30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}{30,31,32,33,34,35,36,37,38,39,41,42,43,44,45,46,47,48,49,4A,4B,4C,4D,4E,4F,50,51,52,53,54,55,56,57,58,59,5A}<br />
do echo 'http://ccs.cdn.shop.wii.com/ccs/download/00010001'$i'/tmd <br />
out='$i'.tmd'; done | aria2c -i - -j16 -x16<br />
</nowiki><br />
<br />
This example uses 16 threads and no delay. Note that the spaces before out= have to be preserved.<br />
<br />
{{Navigation box}}</div>Larsenv