Difference between revisions of "Niconico"

From Archiveteam
Jump to navigation Jump to search
(Blogmaga - site structure)
(What happened to Blomaga)
Line 124: Line 124:


== Blomaga ==
== Blomaga ==
NicoNico Blomage is NicoNico's blog platform. It is scheduled to shut down October 7, 2021<ref>https://site.nicovideo.jp/ch/userblomaga_thanks/</ref>.
NicoNico Blomage is NicoNico's blog platform. It was scheduled to shut down October 7, 2021<ref>https://site.nicovideo.jp/ch/userblomaga_thanks/</ref> (and did go down then, but oddly enough seemingly at midnight Pacific time, instead of Japan time). A grab script was mostly written but never run; instead, a series of nico.ms links (see below) were generated and run through [[ArchiveBot]]. From IDs going to about 2 million, about 1.2 million were saved before the site shut down. The rest were run anyhow, in order to try to capture official (business account) blogs not included in the shutdown, on the grounds that these are probably going down soon anyhow.
 
As of the time of writing, [https://archive.fart.website/archivebot/viewer/?q=niconico a search of the ArchiveBot viewer for "niconico"] results in an accurate listing of jobs. As almost always, the easy way to access the data is through the Wayback Machine.


=== Site Structure ===
=== Site Structure ===

Revision as of 20:28, 12 October 2021

Niconico
Niconico logo
Wwwnicovideojp-full.png
URL http://www.nicovideo.jp[IAWcite.todayMemWeb]
Status Online!
Archiving status Partially saved
Archiving type Unknown
Project source niconico-grab
Project tracker niconico
IRC channel #niconino (on hackint)

Niconico (http://www.nicovideo.jp/), formerly known as Nico Nico Douga is a Japanese video sharing site similar to Youtube managed by Niwango, a subsidiary of Dwango. As of December 2014, Niconico is the ninth most visited site in Japan. It has a unique comment system where the comments are overlaid directly over the video, synced to a specific video playback time. The site is known for its anime related content and memes. Due to the limited server capacity, Niconico restricts access to the website to free users at peak times (7 p.m. to 2 a.m.) by forcing them to stream lower quality versions of videos or blocking access entirely to newly registered users. An English-language version was added on October 17, 2012, replacing the Niconico.com website, featuring a new player and translation tools allowing users to translate video descriptions into English or Chinese. An HTML5 player was trialled beginning in 2016 and made default across the site in 2017, replacing the old Flash player. From February 28, 2018 an account is no longer required to watch videos.

Until December 2020, Nico Nico Pedia used a Flash-based Oekaki tool ([1]). The drawing process of posted drawings (お絵カキコ Oekakiko) could be replayed.

As of spring 2021, Niconico will begin deleting metadata and comments from deleted videos (which were previously visible to logged-in users).At the same time, the "past log," a feature that displays past comments that had been hidden due to restrictions on the number of comments that can be displayed, will also be discontinued with the renewal of the comment server, regardless of the availability of the video.[1] Most comments were archived (over 95%) and all metadata saved

Spinoff sites

Scripts

  • nicovideo-dl is a small command-line program (similar to youtube-dl) to download videos from www.nicovideo.jp. It is implemented in Python. It is platform-independent and should work on Unix/Windows/Mac.It seems to be unavailable now.
  • Nicofox NicoFox is a Mozilla Firefox extension for video website Nico Nico Douga. It contains a Download Manager, which can download videos from the website with their comments. (Nicofox will burn in comments into the video file with no way to watch the saved video with comments disabled, if that feature is enabled. Not recommended for serious archival.)

There are several other generic scripts and browser extension that can download videos from video sharing sites that will work with Niconico, but they can't download videos with the comments overlaid over the video. youtube-dl supports Niconico but will cut out during the video due to not properly supporting the required heartbeat requests to maintain the connection (introduced in 2018); pull requests have been made to fix this but as of April 2020 are still pending.

  • youtube-dlIt is possible to download videos through youtube-dl.
  • animelover1984/youtube-dl is a fork of youtube-dl that fixes the Niconico implementation and directly integrates other features (such as comment and livestream downloading).

Recommended way to archive niconico videos using nicovideo-dl

This method will require you to have an account registered on niconico. To get the highest quality video possible, you will need either need to buy a premium subscription (540 yen a month) or only download videos outside of the peak times (7pm - 2am JST). One way to tell if you're getting the higher quality download is that they usually have a .mp4 extension, whereas lower-quality downloads result in a .flv file.

nicovideo-dl --username "YOUR-USERNAME" --password "YOUR-PASSWORD" --get-comment --raw-comment --title http://www.nicovideo.jp/watch/sm20338084

This will grab the video and the comment XML file. The XML file can be converted into an ASS subtitle file compatible with most video players using the danmaku2ass script by m13253.

Recommended way to archive niconico videos using animelover1984/youtube-dl

python3 youtube_dl\__main__.py --write-info-json --get-comments --write-sub --all-subs --write-thumbnail --username <USERNAME> --password <PASSWORD> https://www.nicovideo.jp/watch/sm21347640

Explanation for the arguments:

  • "--write-info-json" creates a "filename.info.json" file that contains all metadata for the video.
  • "--get-comments" informs youtube-dl that it should download comment data, and stores it in the info.json file
  • "--write-sub --all-subs" converts the above comment/danmaku data into .ASS subtitle files, for all 3 comment regions (English, Japanese and Chinese)
  • "--write-thumbnail" writes the thumbnail file to disk

If you are not opposed to editing the file container itself, then there are additional useful arguments:

  • "--add-metadata" embeds some metadata into the video file such that they show up in video players
  • "--embed-thumbnail --embed-subs" embed thumbnails and subtitle files into the video file respectively
  • "--remux mkv" remuxes the video into an MKV file, which is required for subtitles to be embedded and can handle more exotic Niconico stream formats than MP4. Does not work with some .swf file sources.


Similar to the above method, with a free account or with no account at all, there is a time range every day where downloading high-quality versions of videos is not available. However there are some nuances when downloading the highest quality videos.

If the video is sufficiently old enough (always prior to December 2016 and sometimes before June 2018), then the "smile_high" format is the highest quality as it is the source upload video. This format typically contains .swf, .flv or .mp4 files. Otherwise, the DMC formats are the highest quality. (smile_low is always the worst quality)

An indepth discussion of Niconico format qualities can be found here.


The youtube-dl fork tries to automatically determine the best quality via these rules and other quality factors such as bitrate, however if you wish to examine yourself you can pass "-F" as an option and see the format selection yourself. Here is the format selection for the example video:

format code                extension  resolution note
h264_360p_low-aac_64kbps   mp4        640x360    DMC Low-Quality  364k , h264@ 300k, aac  @ 64k
h264_360p_low-aac_192kbps  mp4        640x360    DMC Low-Quality  492k , h264@ 300k, aac  @192k
h264_360p-aac_64kbps       mp4        640x360    DMC 360p  664k , h264@ 600k, aac  @ 64k
h264_360p-aac_192kbps      mp4        640x360    DMC 360p  792k , h264@ 600k, aac  @192k
h264_480p-aac_64kbps       mp4        854x480    DMC 480p 1664k , h264@1600k, aac  @ 64k
h264_480p-aac_192kbps      mp4        854x480    DMC 480p 1792k , h264@1600k, aac  @192k
smile_high                 mp4        864x486    High quality smile video 3419k , mp4 container, h264@3226k, aac  @189k, 94.59MiB (best)

Site reconnaissance

"As of October 31, 2011, Niconico has over 23,690,000 registered users, 6,870,000 mobile users and 1,390,000 premium users." Since then the number of premium users peaked at around 2,560,000 in mid-2016, before declining. As of December 2019 the number of active premium users is about 1,660,000.

Video IDs appear to be sequential, although very low video IDs may be handled specially.

Overview of Video IDs

A video ID is a number given to each video, consisting of a two-letter alphabetic string representing the video site where the video was posted or the content holder providing the video, and a unique sequential number unique to the video itself. In short, it is the http://www.nicovideo.jp/watch/aa9999 part of the video URL. The following is a list of the types of videos.

IDs of videos that can be posted by users

  • sm - SMILEVIDEO videos. Can be attached to almost any video.
  • nm - Videos posted with official Nico Nico Video editing software (such as Nico Nico Movie Maker).

IDs of videos that could be posted in the past

These things are not linked automatically. Containing the metadata of the video.

  • am - Videos uploaded to "AmebaVision", which was discontinued in October 2007, making it impossible to watch videos.
  • fz - Videos uploaded to "Photozo", discontinued in May 2008, but still available for viewing.
  • ut - A video uploaded to "Youtube". Cannot be viewed now.
  • dm - A video uploaded to "DailyMotion", which was used by niconico.com.

Official Videos

  • ax - "avex" official
  • ca - "Cho! Animelo", "Nyoko Nyoko Video"
  • cd - (unknown)
  • cw - "Callawood Video"
  • fx - "MTV official", not supposed to be posted to SMILEVIDEO
  • ig - "i*gura video exit_nicochannel"
  • na - "Livedoor net animation"
  • om - "Otome"
  • sd - Basically, it's still used for development purposes by the management.
  • sk - "spike NicoNico Channel exit_nicochannel"
  • yk - "YuriChannel video"
  • yo - "Yoshi Yoshi video"
  • za - "NicoNicoNico Anime Channel"
  • zb - "NicoRadio&TV"
  • zc - "NicoAni News"
  • zd - "ai sp@ce"
  • ze - "Niji Audio Visual Room"
  • nl - NicoNico Live Broadcast
  • so - SmileOfficial. the distribution of dedicated identifiers for official videos such as the above has been discontinued, and all official video identifiers have been unified to this.

If you watch official videos via NicoNico Channel, you will be assigned a different ID for the channel video, so you may not see these videos as rarely.

Blomaga

NicoNico Blomage is NicoNico's blog platform. It was scheduled to shut down October 7, 2021[2] (and did go down then, but oddly enough seemingly at midnight Pacific time, instead of Japan time). A grab script was mostly written but never run; instead, a series of nico.ms links (see below) were generated and run through ArchiveBot. From IDs going to about 2 million, about 1.2 million were saved before the site shut down. The rest were run anyhow, in order to try to capture official (business account) blogs not included in the shutdown, on the grounds that these are probably going down soon anyhow.

As of the time of writing, a search of the ArchiveBot viewer for "niconico" results in an accurate listing of jobs. As almost always, the easy way to access the data is through the Wayback Machine.

Site Structure

Blomaga is divided into blogs (i.e. users), which contain posts. Blogs are identified by an alphanumeric ID (distinct from a more customizable blog title), whereas posts are globally serially numbered, with the number generally prefixed by the string "ar". https://ch.nicovideo.jp/article/ar[number] and https://nico.ms/ar[number] seem to redirect to posts with that ID. More work is needed to determine correspondence between blogs and video channels, and whether this may serve as another means of discovery.

Vital signs

Seems stable, though premium user counts are still declining. There was a scare in April 2020 when a clause relating to "service ending" was added to their Terms of Service, leading to rumours of the site potentially shutting down, but it turned out to be due to a change in legal requirements.

External links