Niconico

From Archiveteam
Jump to navigation Jump to search

Niconico (http://www.nicovideo.jp/), formerly known as Nico Nico Douga is a Japanese collection of websites centered around a video sharing site similar to Youtube called Niconico Douga. It is managed by Niwango, a subsidiary of Dwango. As of December 2014, Niconico is the ninth most visited site in Japan. The site is known for its anime related content and memes.

Forced 2-factor login on every login began in February 2022 (see [1]).

In June 2024, Niconico was taken down by ransomware on the Kadokawa network (timeline). COO mentions that a 2023 server migration helped Niconico preserve data. ([2])

nndownload - general downloader script, works on Douga, Seiga, and Namahousou.

General network status

  • April 2020 - there is a scare when a clause relating to "service ending" was added to their Terms of Service, leading to rumours of the site potentially shutting down, but it turned out to be due to a change in legal requirements.
  • June 2024 - Niconico is taken down by ransomware on the Kadokawa network (timeline). The COO mentioned that a 2023 server migration helped Niconico preserve data. ([3])
  • October 30 2024 - to appease foreigners (viz., foreign governments and corporations that don't like porn), they delete some material and restrict large swarths of the rest from access from outside Japan.[1] Subsequent to this Nico Nico Shizuka [what does this refer to?] started blocking overseas IPs.

As of 2022 activity is declining, its place in Japanese society being overtaken by YouTube. Premium accounts are still declining in number.

Spinoff sites

Nico Nico Douga

Niconico videos metadata
Niconico logo
Wwwnicovideojp-full.png
URL https://www.nicovideo.jp[IAWcite.todayMemWeb]
Status Online!
Archiving status Partially saved
Archiving type Unknown
Project source niconico-grab
Project tracker niconico
IRC channel #niconino (on hackint)
Data[how to use] archiveteam_niconico

A video sharing website and the core of the Nico Nico family.

Comments are overlaid over the video flow from right-to-left, an iconic feature of the website and called danmaku (弾幕). Comments can use a scripting language (ニコスクリプト Nicoscript). Comment artisans (コメント職人) can produce particularly artful and extravagant ASCII art comments. Comment filters can replace keywords. Video authors can set their own video comments (投稿者コメント) that do not move into archives, unlike regular comments which disappear after being up for a few years(?).

Under each video is a "marketplace" (ニコニコ市場 Nico Nico Ichiba) where products from Amazon and the like can be attached by users, often facetiously.

History

  • October 17, 2012 - A separate English-language interface [also doubling as the Chinese-language interface, it sounds like?] is added, with a new player and tools allowing users to translate video descriptions into English or Chinese.
  • 2017 - An HTML5 player replaces the Flash player by default across the site, after being trialed starting 2016.
  • 2018 - All SWF videos are converted to MP4.[2] (Former) SWFs are the videos whose ID begins with "nm" instead of "sm". A number of videos exploited this to implement strange gimmicks ([4]). Whether the original SWF is still accessible or not after this change is unclear.
  • February 28, 2018 - an account is no longer required to watch videos.
  • 2020 - it is partially crippled [what does this mean specifically?] by the transition away from Flash.
  • Spring 2021 - Niconico will begin deleting metadata and comments from deleted videos (which were previously visible to logged-in users). Additionally, the "past log," allowing display of older comments that had been hidden in favor of newer ones, was be discontinued for all videos.[3] We ran a project for both these facets - Most comments, through their API pages, were archived (over 95%) and all metadata HTML pages were saved.
  • 2023 - The English and Chinese interface localizations are abolished[4]; the language switch remains, preserving video metadata translations and language-segregated video tags and comments.
  • 2024 - "Economy mode", saving on server resources by serving lower-quality videos, or even outright blocking new users, is abolished.[5]
  • October 30, 2024 - the restrictions on foreign viewing (see #General network status) take effect. Unavailable genres and videos are redirected to https://www.nicovideo.jp/region_restriction. However, RSS continues to work[6], in the form of https://www.nicovideo.jp/user/{id}/video/?rss=2.0.
  • January , 2025- Nico Nico Douga has begun enforcing stricter content regulations, resulting in the removal of certain MMD and R-18 videos.[7][8][9][10]
  • January 19, 2025,According to Nicochart, 53,998 videos and 900 million comments have already been deleted.

Detailed site info

Extraction tools

3rd-party metadata archival websites:

Audio extraction websites (all defunct):


There are several generic scripts and browser extension that can download videos from video sharing sites that will work with Niconico, but they can't download videos with the comments overlaid over the video. Listed here are ones that can:

Using nicovideo-dl

This method will require you to have an account registered on niconico. To get the highest quality video possible, you will need either need to buy a premium subscription (540 yen a month) or only download videos outside of the peak times (7pm - 2am JST). One way to tell if you're getting the higher quality download is that they usually have a .mp4 extension, whereas lower-quality downloads result in a .flv file.

nicovideo-dl --username "YOUR-USERNAME" --password "YOUR-PASSWORD" --get-comment --raw-comment --title http://www.nicovideo.jp/watch/sm20338084

This will grab the video and the comment XML file. The XML file can be converted into an ASS subtitle file compatible with most video players using the danmaku2ass script by m13253.

Using yt-dlp
yt-dlp --write-info-json --get-comments --write-sub --all-subs --write-thumbnail --username <USERNAME> --password <PASSWORD> https://www.nicovideo.jp/watch/sm21347640

Explanation for the arguments:

  • "--write-info-json" creates a "filename.info.json" file that contains all metadata for the video.
  • "--get-comments" informs yt-dlp that it should download comment data, and stores it in the info.json file
  • "--write-sub --all-subs" converts the above comment/danmaku data into .ASS subtitle files, for all 3 comment regions (English, Japanese and Chinese)
  • "--write-thumbnail" writes the thumbnail file to disk

If you are not opposed to editing the file container itself, then there are additional useful arguments:

  • "--add-metadata" embeds some metadata into the video file such that they show up in video players
  • "--embed-thumbnail --embed-subs" embed thumbnails and subtitle files into the video file respectively
  • "--remux mkv" remuxes the video into an MKV file, which is required for subtitles to be embedded and can handle more exotic Niconico stream formats than MP4. Does not work with some .swf file sources.

Similar to the above method, with a free account or with no account at all, there is a time range every day where downloading high-quality versions of videos is not available. However there are some nuances when downloading the highest quality videos.

If the video is sufficiently old enough (always prior to December 2016 and sometimes before June 2018), then the "smile_high" format is the highest quality as it is the source upload video. This format typically contains .swf, .flv or .mp4 files. Otherwise, the DMC formats are the highest quality. (smile_low is always the worst quality)

An indepth discussion of Niconico format qualities can be found here.

yt-dlp tries to automatically determine the best quality via these rules and other quality factors such as bitrate, however if you wish to examine yourself you can pass "-F" as an option and see the format selection yourself. Here is the format selection for the example video:

format code                extension  resolution note
h264_360p_low-aac_64kbps   mp4        640x360    DMC Low-Quality  364k , h264@ 300k, aac  @ 64k
h264_360p_low-aac_192kbps  mp4        640x360    DMC Low-Quality  492k , h264@ 300k, aac  @192k
h264_360p-aac_64kbps       mp4        640x360    DMC 360p  664k , h264@ 600k, aac  @ 64k
h264_360p-aac_192kbps      mp4        640x360    DMC 360p  792k , h264@ 600k, aac  @192k
h264_480p-aac_64kbps       mp4        854x480    DMC 480p 1664k , h264@1600k, aac  @ 64k
h264_480p-aac_192kbps      mp4        854x480    DMC 480p 1792k , h264@1600k, aac  @192k
smile_high                 mp4        864x486    High quality smile video 3419k , mp4 container, h264@3226k, aac  @189k, 94.59MiB (best)

Overview of Video IDs

A video ID is a number given to each video, consisting of a two-letter alphabetic string representing the video site where the video was posted or the content holder providing the video, and a unique sequential number unique to the video itself. In short, it is the http://www.nicovideo.jp/watch/aa9999 part of the video URL. The following is a list of the types of videos.

IDs of videos that can be posted by users
  • sm - SMILEVIDEO videos. Can be attached to almost any video.
  • nm - Videos posted with official Nico Nico Video editing software (such as Nico Nico Movie Maker).
IDs of videos that could be posted in the past

These things are not linked automatically. Containing the metadata of the video.

  • am - Videos uploaded to "AmebaVision", which was discontinued in October 2007, making it impossible to watch videos.
  • fz - Videos uploaded to "Photozo", discontinued in May 2008, but still available for viewing.
  • ut - A video uploaded to "Youtube". Cannot be viewed now.
  • dm - A video uploaded to "DailyMotion", which was used by niconico.com.
Official Videos
  • ax - "avex" official
  • ca - "Cho! Animelo", "Nyoko Nyoko Video"
  • cd - (unknown)
  • cw - "Callawood Video"
  • fx - "MTV official", not supposed to be posted to SMILEVIDEO
  • ig - "i*gura video exit_nicochannel"
  • na - "Livedoor net animation"
  • om - "Otome"
  • sd - Basically, it's still used for development purposes by the management.
  • sk - "spike NicoNico Channel exit_nicochannel"
  • yk - "YuriChannel video"
  • yo - "Yoshi Yoshi video"
  • za - "NicoNicoNico Anime Channel"
  • zb - "NicoRadio&TV"
  • zc - "NicoAni News"
  • zd - "ai sp@ce"
  • ze - "Niji Audio Visual Room"
  • nl - NicoNico Live Broadcast
  • so - SmileOfficial. the distribution of dedicated identifiers for official videos such as the above has been discontinued, and all official video identifiers have been unified to this.

If you watch official videos via NicoNico Channel, you will be assigned a different ID for the channel video, so you may not see these videos as rarely.


Deleted video metadata

Source: NicoChart Update 2024-11-30

Deletion Reason Count
Confirmed Deleted Videos 4,170,698 / 19,877,174 [20.982%]
User Deletion 147,035
Administrator Deletion 31,610
Account Deletion 19,866
Deleted by Uploader or Admin 457
Deletion with Unknown Reason 50,233
Copyright Holder Deletion 18,732
Rights Holder Deletion 406,406
Rights Holder Deletion (Video & Audio) 622
Rights Holder Deletion (Video) 1
Rights Holder Deletion (Audio) 43
Violation Deletion 4
Distribution Stopped 4,889

Blomaga

NicoNico Blomaga[IAWcite.todayMemWeb] is NicoNico's blog platform. It was scheduled to shut down October 7, 2021[11] (and did go down then, but oddly enough seemingly at midnight Pacific time, instead of Japan time). A grab script was mostly written but never run; instead, a series of nico.ms links (see below) were generated and run through ArchiveBot. From IDs going to about 2 million, about 1.2 million were saved before the site shut down. The rest were run anyhow, in order to try to capture official (business account) blogs not included in the shutdown, on the grounds that these are probably going down soon anyhow.

As of the time of writing, a search of the ArchiveBot viewer for "niconico" results in an accurate listing of jobs. As almost always, the easy way to access the data is through the Wayback Machine.

Site Structure

Blomaga is divided into blogs (i.e. users), which contain posts. Blogs are identified by an alphanumeric ID (distinct from a more customizable blog title), whereas posts are globally serially numbered, with the number generally prefixed by the string "ar". https://ch.nicovideo.jp/article/ar[number] and https://nico.ms/ar[number] seem to redirect to posts with that ID. More work is needed to determine correspondence between blogs and video channels, and whether this may serve as another means of discovery.

Nico Nico Seiga

For hosting static artistic works, similar to Pixiv or Deviantart. The entire site is blocked out of Japan, as far as I can tell. Besides this it seems pretty simple; a lot of faceted search but none of that is too Javascripty.

They get aggressive with trying to get you to sign up for an account, and in particular

  • Login is required to view any work in full quality
  • Login is required to view the thumbnails for Shunga works on their own individual pages at all; though a smaller thumbnail is visible on the search page anyway

The script here suggests that full images are (or the path to them includes) site:http://seiga.nicovideo.jp/image/source/[id number] but these require auth when they're visited too.

Comments do not flow across the image.

Nico Nico Shunga

The R-15 corner of Seiga. Removed January 29, 2025.[12]; we tried to run a project for this, setting up a one-time use ArchiveBot node to do a crawl of all of Seiga, but this happened too slowly, especially because after the first was all set up it turned out that the VPS geolocated to Singapore instead of Japan.

STWP downloaded all the image thumbnails, image metadata pages, and full images; [13] these are respectively in the IA CHUNGA-THUMB-20250121180740887-crawling, SHUNGA-WORK-DETAILS-crawling, and listed here, but will not go in the WBM; and they do not contain tag/user list pages.

けしからん性力図 is a user-created heatmap of where the most sexual part of the image is.[14]

Example search page: https://seiga.nicovideo.jp/tag/%E3%83%A1%E3%82%B9%E3%82%B1%E3%83%A2?page=8&target=shunga

Thumbnail shown there: https://lohas.nicoseiga.jp//thumb/10830631u?1634270638

Thumbnail modified to be larger: https://lohas.nicoseiga.jp//thumb/10830631l?1634270638

Image page with "inappropriate expressions" warning: https://seiga.nicovideo.jp/seiga/im10830631

Nico Nico Manga

A corner of Seiga hosting both user-created (ユーザー作品) and commercial (公式作品) comics. Each comic can hold a maximum of 100 chapters. Login is required to view past the first page of a chapter.

Chapters of commercial comics are usually available for free for one month, without geographic restriction, after which it is removed from the website.

Danmaku comments flow from right-to-left over the image.

Nico Nico jikkyo

Niconico Jikkyou is a service that lets users share real-time comments while watching TV or listening to the radio on the Niconico platform. Originally launched on November 28, 2009, after two trial runs, it was significantly revamped and integrated into Niconico Live Broadcasting on December 16, 2020.

Niconico Channel IDs for Jikkyou use either the "jk" or "ch" prefix.

This 2020 integration led to the discontinuation of the original Jikkyou system, ending support for devices like torne and BRAVIA, and nearly causing the loss of over a decade's worth of comment logs. To prevent this, users of the 5ch DTV board initiated a community archiving project. This effort successfully preserved all Jikkyou comments up to December 2020.

This archive, along with ongoing Jikkyou comment data up to the present, is available on API: jikkyo.tsukumijima.net Hugging Face: KakologArchives. This dataset incorporates all past logs from the legacy Niconico Jikkyou up to December 15, 2020 (originally distributed by Nekopanda), as well as logs from the new Niconico Jikkyou, including community-based programs. Additionally, starting from June 10, 2024, it includes current-day logs from NX-Jikkyo (an alternate comment server for Jikkyou), collected every 5 minutes and continuously integrated into the dataset.

While the new Niconico Jikkyou within Niconico Live Broadcasting offers a similar service, timeshift viewing (equivalent to the archived logs) is limited to three weeks. Furthermore, free accounts require advance timeshift reservations, reducing the spontaneity and accessibility of the original service.



Nico Nico Pedia

An encyclopedia website. It is a valuable source for Niconico culture, and along with Pixiv Encyclopedia, a valuable source for various trends.

As explained on the front page, articles are generally only editable by premium users. Articles for specific videos can also be edited by the video uploader, users can edit their own user pages, and community articles can be edited by community members.

Users can attach oekaki drawings (お絵カキコ oekakiko) or chiptune MML music (ピコカキコ pikokakiko) to article comments. The latter is at risk of JASRAC copyright takedowns.

Until December 2020, Nico Nico Pedia used a Flash-based Oekaki tool ([12]). The drawing process of posted drawings (お絵カキコ Oekakiko) could be replayed. The chiptune player was also Flash-based, but this writer believes nothing was lost.

Lists of latest posts of various types are accessible from the front page.

NicoNi-Commons

A commons website for content like free background music and 3D models. User ID numbers are different on Commons but still linked to a user's primary ID.

NicoNi-Community

NicoNi-Community (ニコニコミュニティ)
Status Offline
Archiving status Lost
Archiving type Unknown
IRC channel #archiveteam-bs (on hackint)

Videos, seiga illustrations, and livestreams can be registered with a community. Communities have textboards (掲示板) that are only accessible to members.

Videos can be only accessible to members (https://vocadb.net/T/7446), example https://www.nicovideo.jp/watch/sm13929471.

It was not restored after the 2024 cyberattack.

Game Atsumaru

Game Atsumaru (ゲームアツマール)
URL https://game.nicovideo.jp/atsumaru/
Status Offline
Archiving status Partially saved
Archiving type ArchiveBot, other
IRC channel #archiveteam-bs (on hackint)
Data[how to use] job:3tt38940863qg137qhybq2xbe
job:bsx1a496u5w4cblcwmq5lyjq4
game-atsumaru-partial

Game Atsumaru (ゲームアツマール) is Niconico's web game hosting website. Originally launched as RPG Atsumaru, it was the officially supported game upload platform for RPG Maker MV, and later added support for RPG Maker MZ and Akashic Engine. Since non-RPG games can be made using RPG Maker, the site was renamed Game Atsumaru in 2021.

In 2021 support was added for RPG Maker 2000/2003 using EasyRPG as an interpreter.

Danmaku comments are supported; they flow from right-to-left during the game.

On 20 December 2022, it was announced that the Game Atsumaru platform would be shutting down on 28 June 2023.[15] Uploaders have been given the ability to download their games. A new game hosting platform Nico Nama Game (ニコ生ゲーム) is scheduled to be be launched on 19 April 2023, at which point uploads to Game Atsumaru will be stopped.

After about 12pm JST, June 28, 2023, the entire https://game.nicovideo.jp/ will be redirected to https://blog.nicovideo.jp/niconews/194994.html .

Archiving status

Game homepages are archived by ArchiveBot.

There's a repo at https://github.com/yts98/game-atsumaru-discovery to discover as many static resources as possible. Volunteers from Archive Team and Save The Web Project joined this manual project.

  • Most of game frameworks lazily load resources, so staying on the page for a long time does not guarantee that all resources will be found. A difficult approach would be to categorize games by framework and then group them for static analysis.
    • Without archiving all the dynamically loaded data/[A-Za-z]+\.json and data/Map[0-9]{3}\.json, over 5989 games made with RPG Maker MV and over 1332 games made with RPG Maker MZ would be completely unplayable.
  • Danmaku comments and scoreboards are event-triggered, so it's better to check for:
    • possible sceneName (strings customized by the game author, such as __title, __gameover, map1) to archive danmaku comments.
    • boardId (1~10 or 1~30) to archive scoreboards.

Site structure

URLs are of the format https://game.nicovideo.jp/atsumaru/games/gm[number], where gm[number] is the gameId (up to approximately 28500 as of 2022-12-20, up to 29754 as of 2023-04-19). nico.ms short links work, as in https://nico.ms/gm[number]

As of June 24, 2023, 9490 games are publicly listed, 21 of which must be paid channel members to play; others were unlisted, private, or removed.

Unlisted games are of the format https://game.nicovideo.jp/atsumaru/games/gm29721?key=f45a1f70c172. They can be discovered by googling site:https://game.nicovideo.jp/atsumaru/games/ inurl:key or searching Wayback CDX. As of June 24, 2023, a total of 307 unlisted games have been found.[16]

Game loading mechanism

A game is actually an HTML with some static resources (JavaScript, CSS, WASM, images, audio), but the actual URL of the game is not embedded in the aforementioned page.

The ReactJS app will call the API method createPlayTicketV2() to get a ticket with a gameUrl in the format https://resource.game.nicovideo.jp/games/:gameId/:version/index.html and open a session that lasts 1800 seconds. If a client request game resources without a valid ticket, the server would simply respond with an HTTP 404.

curl commands 
curl -X GET -A "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0" \
     -H "X-Frontend-Id: 39" -H "X-Frontend-Version: 3.464.0" \
     -H "Origin: https://game.nicovideo.jp" -H "Referer: https://game.nicovideo.jp/" \
     "https://api.game.nicovideo.jp/v1/rpgtkool/games/gm9482/permission-info.json?sandbox=0"

curl -X POST -A "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0" \
     -H "X-Frontend-Id: 39" -H "X-Frontend-Version: 3.464.0" -H "X-Request-With: https://game.nicovideo.jp" \
     -H "Origin: https://game.nicovideo.jp" -H "Referer: https://game.nicovideo.jp/" -F "=" \
     -c "cookie.txt" "https://api.game.nicovideo.jp/v1/rpgtkool/games/gm9482/play-tickets.json?sandbox=0&wipAccessKey"

curl -X GET -A "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0" \
     -H "Referer: https://resource.game.nicovideo.jp/core/player/index.html?3f452cce70" \
     -b "cookie.txt" "https://resource.game.nicovideo.jp/games/gm9482/1147/index.html"

Unlisted games can only be accessed by appending a valid key to the URL:

curl commands 
curl -X GET -A "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0" \
     -H "X-Frontend-Id: 39" -H "X-Frontend-Version: 3.464.0" \
     -H "Origin: https://game.nicovideo.jp" -H "Referer: https://game.nicovideo.jp/" \
     "https://api.game.nicovideo.jp/v1/rpgtkool/games/gm289/permission-info.json?sandbox=0&wipAccessKey=b0495694f39e"

curl -X POST -A "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0" \
     -H "X-Frontend-Id: 39" -H "X-Frontend-Version: 3.464.0" -H "X-Request-With: https://game.nicovideo.jp" \
     -H "Origin: https://game.nicovideo.jp" -H "Referer: https://game.nicovideo.jp/" -F "=" \
     -c "cookie.txt" "https://api.game.nicovideo.jp/v1/rpgtkool/games/gm289/play-tickets.json?sandbox=0&wipAccessKey=b0495694f39e"

curl -X GET -A "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0" \
     -H "Referer: https://resource.game.nicovideo.jp/core/player/index.html?3f452cce70" \
     -b "cookie.txt" "https://resource.game.nicovideo.jp/games/gm289/12/index.html"

When window.origin is in the constant list validOrigins, the app loads the game player in the iframe, and further loads the gameUrl in the game player's iframe, providing an interactive API[17] object window.RPGAtsumaru for the game.

Metadata provided by niconico

Datasets from Dwango Co., Ltd. are available to researchers through the National Institute of Informatics (NII). https://www.nii.ac.jp/dsc/idr/nico/nico.html

Niconico Video Metadata and Comment Data

This dataset contains metadata for approximately 20 million videos posted on Niconico Douga from its launch until September 30, 2021, along with around 4.1 billion comments. The actual video files are not included, and user IDs have been removed.

Video Metadata (Updated:2021/12/22)

  • Data such as video title, description, tags, upload date and time, view count, and comment count.
  • Provided in JSON format, with 1,000 files (each file containing data for about 20,000 videos).
  • Total file size is approximately 15GB.

Comment Data (Updated:2021/12/22)

  • Data such as comment text, posting date and time, and playback position at the time of posting.
  • Provided in JSON format, with one file per video.
  • Compressed file size is about 86GB, expanding to approximately 573GB.

Niconico Encyclopedia Data (Updated: November 25, 2024)

  • This dataset contains article headers and body text for all articles posted on the Niconico Encyclopedia from May 2008 to January 2024. User pages and user IDs have been removed.
  • This dataset excludes both the bulletin board data related to the articles and the OekakiPictures within those bulletin boards.
  • Article Header Data
  1. Data such as article ID, article title, article reading (yomi), article type (word/video/product/live broadcast/official), and article creation date and time.
  • Article Body Data
  1. Data such as article ID, article text, and article update date and time.
  2. Data is provided in CSV format, organized by year or month.
  3. Compressed file size is approximately 0.6GB, expanding to about 2.3GB.

NicoNico API Reference



External links

References