Difference between revisions of "YouTube"

From Archiveteam
Jump to navigation Jump to search
(→‎Inactive accounts policy: Significant revision to reflect accurate facts)
(link to YouTube Video Finder by TheTechRobo)
 
(74 intermediate revisions by 23 users not shown)
Line 1: Line 1:
{{DISPLAYTITLE:<span style=font-family:impact>You<span style=";color:white;background: linear-gradient(210deg, rgba(255,255,255,1) 9%, rgba(255,0,0,1) 34%, rgba(100,0,0,1) 100%);border-radius:5px">Tube</span>}}
{{DISPLAYTITLE:<span style=font-family:impact>You<span style=";color:white;background: linear-gradient(210deg, rgba(255,255,255,1) 9%, rgba(255,0,0,1) 34%, rgba(100,0,0,1) 100%);border-radius:5px">Tube</span>}}
{{Infobox project
{{Infobox project
| title = YouTube Metadata
| title = YouTube Metadata & Selected Videos
| logo = YouTube_logo_2017.png
| logo = YouTube_logo_2017.png
| image = YouTube2018.png
| image = YouTube2018.png
| description = Saving metadata for as many videos as possible including watch pages, captions, comments, thumbnails and saving selected YouTube videos
| description = Saving metadata for as many videos as possible including watch pages, captions, comments, thumbnails as well as saving [[YouTube#Scope|selected YouTube videos]]
| URL = {{url|1=https://www.youtube.com/}}
| URL = {{url|1=https://www.youtube.com/|2=YouTube}}
| project_status = {{online}} but possibly {{endangered}}, see [[YouTube#Vital signs|Vital signs]]
| project_status = {{online}} but possibly {{endangered}}, see [[YouTube#Vital signs|Vital signs]]
| archiving_status = {{inprogress}}
| archiving_status = {{inprogress}} (videos manually submitted)
| archiving_type = DPoS
| source = [https://github.com/ArchiveTeam/youtube-grab youtube-grab]
| source = [https://github.com/ArchiveTeam/youtube-grab youtube-grab]
| tracker = [https://tracker.archiveteam.org/youtube/ youtube]
| tracker = [https://tracker.archiveteam.org/youtube/ youtube]
| irc = down-the-tube
| irc = down-the-tube
| irc_network = hackint
| lead = [[User:Arkiver|Arkiver]]
| data = {{IA collection|archiveteam_youtube}}
}}
}}
{{Infobox project
{{Infobox project
Line 17: Line 19:
| logo = YouTube_logo_2017.png
| logo = YouTube_logo_2017.png
| image = YouTube2018.png
| image = YouTube2018.png
| description = Saving selected YouTube videos
| description = Saving selected YouTube videos. See [https://borg.xyz/ya.htm #youtubearchive's website] for scope and details.
| URL = {{url|1=https://www.youtube.com/}}
| URL = {{url|1=https://www.youtube.com/|2=YouTube}}
| project_status = {{online}} but possibly {{endangered}}, see [[YouTube#Vital signs|Vital signs]]
| project_status = {{online}} but possibly {{endangered}}, see [[YouTube#Vital signs|Vital signs]]
| archiving_status = {{inprogress}}
| archiving_status = {{onhiatus}} (videos manually submitted)
| irc = youtubearchive
| irc = youtubearchive
| irc_network = hackint
| lead = Ivan
| lead = Ivan
}}
}}


[[File:YouTube2011.png|thumb|YouTube homepage screenshot as of 2011-04-22.]]
[[File:YouTube2011.png|thumb|YouTube homepage screenshot as of {{datetime|2011-04-22}}.]]


'''YouTube''' is a [[Video hostings|video sharing]] website currently owned by [[Google]]. YouTube is currently the most popular video hosting website on Earth.
'''YouTube''' is a [[Video hostings|video sharing]] website currently owned by [[Google]]. YouTube is currently the most popular video hosting website on Earth.
== Archive Team project ==
Archive Team launched a project to archive certain videos and channels from YouTube. Videos archived in this project are ingested into the Wayback Machine, and will be playable in the Wayback Machine (WBM) after being added to a separate index (may take a few days). Note: YouTube includes your worker's IP address in the <code>videoplayback</code> urls. As such, '''your worker's IP will be recorded in the project's data.'''
=== Scope ===
Anyone is able to queue channels and videos to be archived in channel {{IRC|down-the-tube}}. However, the videos and channels to be archived should be limited to:
* Channels/videos almost sure to be deleted, like:
** Company gone bankrupt
** Person died
** YouTube banning certain channels/content
* The following channels/videos if they are connected to a major event that could cause them to be changed heavily or deleted:
** Government channels
** News channels
** Event channels
** Political party channels
** Politician accounts
* Unlisted videos (hidden from YouTube search, only accessible by URL)
* Advertisement video that YouTube actually treats as advertisement
* 'Newsworthy' content, for example:
** Channels/videos from opinion makers in the Russia-Ukraine war
** Channels/videos containing content of major events (terror attacks, wars, etc.)
*** Only queue an entire channel if the entire channel is related to the major event.
* Others are possible, check in with [[User:Arkiver|Arkiver]] on IRC.
When a channel of video is queued for archival, a reason should be given with <tt>-e</tt>/<tt>--explain</tt>.
== Why download? ==
Just like [[Why_Back_Up?|local backups]], it is advisable to download videos regularly, as any which is not backed up is hanging by a thread above the memory hole. Even downloading at a low resolution such as 240p or 360p still is infinitely better than nothing (0p).
While established platforms typically have a solid and failure-resistant technical infrastructure, any content is still endangered from '''human factors'''. Not only could a creator run afoul of YouTube's site staff (or any other platform for that matter), or the platform change their policies in a way that affects existing content retroactively, but a creator themselves might run into public controversity and according scrutiny, or emotionally decide that they no longer like some of their early videos, and as eternal as they once appeared, they are gone in a flash, or locked away in the vault of private videos.
Video creators are recommended to upload their work to no less than two platforms. If platform staff decides take down videos or even nuke an entire channel, which is done with the press of a button and takes an entire stash of videos with it, viewers will know where to still find those videos.
:"Google [parent company of YouTube] is an archive in the same way a supermarket is a food museum."  –– Jason Scott
:"For many years now I “favourite” videos on YT by downloading them. You can't expect any video on YT to still be there tomorrow/later."<ref>[https://news.ycombinator.com/item?id=27612198  YCombinator user ''sen'', June 24, 2021, on "Older unlisted videos will be made private unless you opt out"]</ref>


== Archiving tools ==
== Archiving tools ==


Several free FLV downloaders and video-to-URL converters exist on the web.
Several free FLV downloaders and video-to-URL converters exist on the web.
AT rescue projects usually use [https://github.com/rg3/youtube-dl/ youtube-dl].<br>
AT rescue projects usually use [https://github.com/yt-dlp/yt-dlp yt-dlp].<br>
YouTube annotations (speech bubbles and notes) are available as XML
YouTube annotations (speech bubbles and notes) are available as XML
<pre>
<pre>
Line 46: Line 83:
: '''Note:''' Be very careful dumping channels over 100 videos with this script. Let an admin know what you're doing, dump 50 videos, and have a collection created. Work is being started on adding a flag to specify a collection name instead of "Community Video" which is what it defaults to. Always try to create an item. For the time being the script will have to be hand edited to specify a different collection.
: '''Note:''' Be very careful dumping channels over 100 videos with this script. Let an admin know what you're doing, dump 50 videos, and have a collection created. Work is being started on adding a flag to specify a collection name instead of "Community Video" which is what it defaults to. Always try to create an item. For the time being the script will have to be hand edited to specify a different collection.


[https://github.com/bibanon/tubeup tubeup.py] is an automated archival script that uses [https://github.com/rg3/youtube-dl/ youtube-dl] to download a Youtube video (or any other provider supported by youtube-dl), and then uploads it with all metadata to the Internet Archive.  
[https://github.com/bibanon/tubeup tubeup.py] is an automated archival script that uses [https://github.com/yt-dlp/yt-dlp yt-dlp] to download a Youtube video (or any other provider supported by yt-dlp), and then uploads it with all metadata to the Internet Archive.  


This way, all metadata from the video, such as title, tags, categories, and description, are preserved in the corresponding Internet Archive item, without having to manually enter it.  
This way, all metadata from the video, such as title, tags, categories, and description, are preserved in the corresponding Internet Archive item, without having to manually enter it.  
Line 52: Line 89:
It also creates a standardized Internet Archive item name format that makes it easy to find the video using the Youtube ID, and reduces duplication: https://archive.org/details/youtube-v9sGhNoSG3o
It also creates a standardized Internet Archive item name format that makes it easy to find the video using the Youtube ID, and reduces duplication: https://archive.org/details/youtube-v9sGhNoSG3o


Youtube-dl [https://github.com/rg3/youtube-dl/blob/master/docs/supportedsites.md also works with many other video sites.]
yt-dlp [https://github.com/yt-dlp/yt-dlp/blob/master/supportedsites.md also works with many other video sites.]


* [https://github.com/bibanon/tubeup Github - tubeup.py]
* [https://github.com/bibanon/tubeup Github - tubeup.py]
Line 58: Line 95:
=== (Manual) Recommended way to archive YouTube videos ===
=== (Manual) Recommended way to archive YouTube videos ===


First, download the video/playlist/channel/user using youtube-dl:
First, download the video/playlist/channel/user using yt-dlp:


<tt>youtube-dl --continue --retries 4 --write-info-json --write-description --write-thumbnail --write-annotations --all-subs --ignore-errors -f bestvideo+bestaudio URL</tt>
<tt>yt-dlp --continue --retries 4 --write-info-json --write-description --write-thumbnail --write-annotations --all-subs --ignore-errors -f bestvideo+bestaudio URL</tt>


This can be simplified by running the [https://github.com/matthazinski/youtube2internetarchive script by emijrp and others], which also handles upload.
This can be simplified by running the [https://github.com/matthazinski/youtube2internetarchive script by emijrp and others], which also handles upload.


You need a recent (2014) ffmpeg or avconv for the <tt>bestvideo+bestaudio</tt> muxing to work.  On Windows, you also need to run youtube-dl with Python 3.3/3.4 instead of Python 2.7, otherwise non-ASCII filenames will fail to mux.  
You need a recent (2014) ffmpeg or avconv for the <tt>bestvideo+bestaudio</tt> muxing to work.


Also, make sure you're using the most recent version of youtube-dl. Previous versions didn't work if the highest quality video+audio was webm+m4a. New versions should automagically merge incompatible formats into a .mkv file.<ref>https://github.com/rg3/youtube-dl/pull/5456</ref>
Also, make sure you're using the most recent version of yt-dlp.


Then, upload it to https://archive.org/upload/ Make sure to upload not only the video itself (.mp4 and/or .mkv files), but also the metadata files created along with it (.info.json, .jpg, .annotations.xml and .description).
Then, upload it to https://archive.org/upload/ Make sure to upload not only the video itself (.mp4 and/or .mkv files), but also the metadata files created along with it (.info.json, .jpg, .annotations.xml and .description).


To capture a [[WARC]] using [https://github.com/internetarchive/warcprox Warcprox], once Warcprox is running, the URL and port of the proxy can be passed to youtube-dl with the proxy parameter, for example: <tt>--proxy="localhost:8000"</tt>
To capture a [[WARC]] using [https://github.com/internetarchive/warcprox Warcprox], once Warcprox is running, the URL and port of the proxy can be passed to yt-dlp with the proxy parameter, for example: <tt>--proxy="localhost:8000"</tt>
 
Two other yt-dlp parameters for getting some more data are <tt>--write-all-thumbnails</tt> and <tt>--write-auto-sub</tt>.


Two other youtube-dl parameters for getting some more data are <tt>--write-all-thumbnails</tt> and <tt>--write-auto-sub</tt>.
=== Wayback Machine ===
{{Notice|This section is a draft.}}
You can submit YouTube URLs to the [https://web.archive.org/save/ Wayback Machine], and the video will be archived on the Wayback Machine. I haven't checked if this still works but it used to possible. Some YouTube videos are on the Wayback Machine.


=== YouTube Archiving For Dummies ===
However, many of those videos that were saved stopped being accessible at some point in time due to an internal mistake at the Internet Archive. This is reported to be fixed, but recently, for a video that did not load, it wasn't lost to the sands of time, I managed to access the complete video by using the right URL: <code><nowiki>https://web.archive.org/web/2oe_/http://wayback-fakeurl.archive.org/yt/</nowiki>'''VideoID'''</code>


The ''[http://archiveteam.hu/lecsu Lecsű]'' service, run by [http://archiveteam.hu ArchiveTeam Hungary], is a convenient way of requesting YouTube videos for archival. It's basically an online wrapper for the above detailed ''tubeup'' service. The requests are reviewed manually, so it may take a few days until a video gets archived, but at least it's super easy to do.
(For more detail, see [https://www.reddit.com/r/DataHoarder/comments/odmiuc/youtube_videos_that_were_once_accessible_on_the/ YouTube videos that were once accessible on the Wayback Machine have since disappeared.])


=== Tools to check for regional restrictions ===
=== Tools to check for regional restrictions ===
Line 85: Line 126:
{{Infobox project
{{Infobox project
| title = YouTube Annotations
| title = YouTube Annotations
| logo = YouTube_logo_2017.png
| image = Youtube-annotations-example-fukkireta.jpg
| image = Youtube-annotations-example-fukkireta.jpg
| description =
| project_status = {{offline}} as of {{datetime|2019-01-15|~15:00}}
| URL = {{url|1=https://www.youtube.com/}}
| project_status = {{offline}}
| archiving_status = {{partiallysaved}}
| archiving_status = {{partiallysaved}}
| source = [https://github.com/omarroth/archive omarroth/archive]
| lead = [https://old.reddit.com/user/omarroth /u/omarroth]
| lead = [https://old.reddit.com/user/omarroth /u/omarroth]
| data = {{IA collection|youtubeannotations}} and {{IA item|youtubeannotations.tar.zstd}}
}}
}}


Annotations<ref>https://youtube.googleblog.com/2008/06/new-beta-feature-video-annotations.html</ref> were ''notes'' that could be added to videos after the upload, they provided plenty customization options such as color, size and internal & external links. People used them to correct mistakes in videos, created mini-games within YouTube and abused them too: "Please like and subscribe", "Watch in HD!!!"
Annotations<ref>https://youtube.googleblog.com/2008/06/new-beta-feature-video-annotations.html</ref> were ''notes'' that could be added to videos after the upload, they provided plenty customization options such as color, size and internal & external links. People used them to correct mistakes in videos, created mini-games within YouTube and abused them too: "Please like and subscribe", "Watch in HD!!!"


On May 2nd, 2017 YouTube disabled editing and creating annotations for videos. On January 16th, 2019 they were removed completely from the site and API responses (~15:00 UTC, same time when they disabled editing).
On {{datetime|2017-05-02}} YouTube disabled editing and creating annotations for videos. On {{datetime|2019-01-16}} they were removed completely from the site and API responses (~15:00 UTC, same time when they disabled editing).
 
An archiving project [https://discord.gg/dP4Pu6d "YouTube Annotation Archive"] organized by [https://www.reddit.com/user/omarroth u/omarroth] that achieved to discover ~1.4 billion videos and download the annotations, if they had any. The project did not completely finish due to the deadline. The annotation archive will be released at [https://www.reddit.com/r/DataHoarder/comments/al7exa/youtube_annotation_archive_update_and_preview/ a later point (Update)].


16GB of just video IDs that were encompassed by the project can be downloaded [https://archive.org/details/archived_annotations_video_ids.csv here].
An archiving project [https://discord.gg/dP4Pu6d "YouTube Annotation Archive"]<ref>{{URL|https://old.reddit.com/r/DataHoarder/comments/aa6czg/youtube_annotation_archive/}}</ref><ref>{{URL|https://old.reddit.com/r/DataHoarder/comments/al7exa/youtube_annotation_archive_update_and_preview/}}</ref><ref>{{URL|https://old.reddit.com/r/DataHoarder/comments/b7imx9/youtube_annotation_archive_annotation_data_from/}}</ref> organized by [https://www.reddit.com/user/omarroth u/omarroth] that achieved to discover ~1.4 billion videos and download the annotations, if they had any. The project did not completely finish due to the deadline. The annotation archive is available at {{IA collection|youtubeannotations}}, and a compressed copy can be found at {{IA item|youtubeannotations.tar.zstd}}. 16GB of just video IDs that were encompassed by the project can be downloaded [https://archive.org/details/archived_annotations_video_ids.csv here].


== Liked lists (December 2019) ==
== Liked lists (December 2019) ==
{{Infobox project
{{Infobox project
| title = YouTube Liked Lists
| title = YouTube Liked Lists
| logo = YouTube_logo_2017.png
| URL = {{url|1=https://www.youtube.com/}}
| project_status = {{offline}}
| project_status = {{offline}}
| archiving_status = {{partiallysaved}}
| archiving_status = {{partiallysaved}}
| tracker = [https://tracker.archiveteam.org/youtube-likedlists/ youtube-likedlists]
| tracker = [https://tracker.archiveteam.org/youtube-likedlists/ youtube-likedlists]
| source = [https://github.com/ArchiveTeam/youtube-likedlists-grab]
| source = [https://github.com/ArchiveTeam/youtube-likedlists-grab youtube-likedlists-grab]
| irc = down-the-tube
| irc = down-the-tube
| irc_network = hackint
| lead = marked
| lead = marked
| data = {{IA collection|archiveteam_youtubelikes}}
}}
}}


On 2019-12-05, the playlists containing users' liked videos were made private.<ref>{{URL|https://support.google.com/youtube/answer/6083270}}</ref>
On {{datetime|2019-12-05}}, the playlists containing users' liked videos were made private.<ref>{{URL|https://support.google.com/youtube/answer/6083270}}</ref>


== Playlist notes (October 2020) ==
== Playlist notes (October 2020) ==
{{Infobox project
{{Infobox project
| title = YouTube playlist notes
| title = YouTube playlist notes
| logo = YouTube_logo_2017.png
| URL = {{url|1=https://www.youtube.com/}}
| project_status = {{offline}}
| project_status = {{offline}}
| archiving_status = {{partiallysaved}}
| archiving_status = {{partiallysaved}}
Line 128: Line 162:
| source = [https://github.com/ArchiveTeam/youtube-playlistnotes-grab youtube-playlistnotes-grab]
| source = [https://github.com/ArchiveTeam/youtube-playlistnotes-grab youtube-playlistnotes-grab]
| irc = down-the-tube
| irc = down-the-tube
| irc_network = hackint
| data = [https://archive.org/details/archiveteam_youtube?query=playlist+notes archiveteam_youtube?query=playlist+notes]
}}
}}


On 2020-10-08, playlist notes were removed because they were 'rarely used'.<ref>https://web.archive.org/web/20200926110942/https://support.google.com/youtube/answer/6083634</ref> While the official documentation only talks of 'private notes', these could be public as well. Although the notes are no longer displayed, they are still included in the Polymer page source code for playlists as of 2021-07-11.
On {{datetime|2020-10-08}}, playlist notes were removed because they were 'rarely used'.<ref>https://web.archive.org/web/20200926110942/https://support.google.com/youtube/answer/6083634</ref> While the official documentation only talks of 'private notes', these could be public as well. Although the notes are no longer displayed, they are still included in the Polymer page source code for playlists as of {{datetime|2021-07-11}} and {{datetime|2023-11-11}}.


== Older unlisted videos (July 2021) ==
== Older unlisted videos (July 2021) ==
{{Infobox project
{{Infobox project
| title = YouTube Older Unlisted Videos
| title = YouTube Older Unlisted Videos
| logo = YouTube_logo_2017.png
| image = YouTube2018.png
| description = Saving 360p videos and metadata for as many unlisted videos from before 2017 as possible including watch pages, captions, comments, and thumbnails.
| URL = {{url|1=https://www.youtube.com/}}
| project_status = {{offline}} except for users who opted their channels out or have made/will make individual videos public or unlisted again
| project_status = {{offline}} except for users who opted their channels out or have made/will make individual videos public or unlisted again
| archiving_status = {{partiallysaved}} - saved 5,739,754 videos, 201.98 TiB total
| archiving_status = {{partiallysaved}} - saved 5,739,754 videos in 360p (including watch pages, captions, comments, thumbnails), 201.98 TiB total
| source = [https://github.com/ArchiveTeam/youtube-grab/tree/aaf61d346dbecf916906efeb4e4e0cd365e98efe youtube-grab]
| source = [https://github.com/ArchiveTeam/youtube-grab/tree/aaf61d346dbecf916906efeb4e4e0cd365e98efe youtube-grab]
| tracker = [https://web.archive.org/web/20210802070832/https://tracker.archiveteam.org/youtube/ youtube]
| tracker = [https://web.archive.org/web/20210802070832/https://tracker.archiveteam.org/youtube/ youtube]
| irc = down-the-tube
| irc = down-the-tube
| irc_network = hackint
| data = {{IA collection|archiveteam_youtube}}
}}
On {{datetime|2021-06-23}}, YouTube announced that they would make all unlisted videos uploaded before {{datetime|2017-01-01}} private in one month, on {{datetime|2021-07-23}}, unless the uploader opted-out of the change<ref>https://blog.youtube/news-and-events/update-youtube-unlisted-links/</ref><ref>https://support.google.com/youtube/answer/9230970</ref>. Uploaders could also opt all of the videos on their channel out of this change by completing a form<ref>https://support.google.com/youtube/contact/older_unlisted_update</ref>. This change is likely because YouTube video IDs are believed to have previously been generated from an encrypted sequential integer instead of randomly, meaning that according to YouTube, the possibility of an unlisted URL being guessed was too significant. What makes the argument dubious is that the possibility is marginal to begin with, and even in the unlikely possibility of it happening, that would be one view of a random unlisted video from somewhere in the world, meaning the possibility of harm from guessed URLs did practically not exist to begin with. An HTTP 429 rate limit would have presumably been hit anyway well before any URL could be guessed through brute-forcing. Additionally, uploaders of unlisted videos consented from the beginning to the ever-so-slight possibility of their video's URL being guessed. And if this alleged security risk was known since 2017, it would mean that Google has withheld this information from users for four years.
 
It is expected that unlisted content from older, inactive channels will be lost because of this change.
 
Although the support article was updated to use past-tense language and the opt-out form was made unavailable approximately as scheduled (around the beginning of {{datetime|2021-07-23}}), the videos were not made private until around the beginning of {{datetime|2021-07-28}} UTC. The change occurred gradually over a number of hours in alphabetical order based on the video ID, starting around {{datetime|2021-07-27|22:00}} and finishing around {{datetime|2021-07-29|03:00}} (give or take an hour or two at both ends).
 
== Discussions/Channel Comments (October 2021) ==
{{Infobox project
| title = YouTube Discussions
| project_status = {{closed}} on {{datetime|2021-10-12}} (UI), {{datetime|2021-10-19}} (InnerTube API), before {{datetime|2021-11-09}} (Public API)
| archiving_status = {{saved}} 257.96 million channels, 2.13TiB
| source = [https://github.com/ArchiveTeam/youtube-discussions-grab youtube-discussions-grab]
| tracker = [https://tracker.archiveteam.org/youtube-discussions/ youtube-discussions]
| irc = down-the-tube
| data = [https://archive.org/details/archiveteam_youtube?query=discussions archiveteam_youtube?query=discussions]
}}
}}
On 2021-06-23 YouTube announced that they would make all unlisted videos uploaded before 2017-01-01 private in one month, on 2021-07-23, unless the uploader opted-out of the change<ref>https://blog.youtube/news-and-events/update-youtube-unlisted-links/</ref><ref>https://support.google.com/youtube/answer/9230970</ref>. Uploaders could also opt all of the videos on their channel out of this change by completing a form<ref>https://support.google.com/youtube/contact/older_unlisted_update</ref>. This change is likely because YouTube video IDs are believe to have previously been generated from an encrypted sequential integer instead of randomly. It is expected that unlisted content from older, inactive channels will be lost because of this change.
In addition, YouTube decided that their new “Community Post” feature can not co-exist with the “Discussion” feature<ref name=discussion>{{URL|https://support.google.com/youtube/answer/7124175|''YouTube Help'' article: “Engage with creators on Community posts” mentions <small>''“'''Note:''' As creators get the Community tab, it will replace the Discussion tab. You can access or delete any comments you left on the Discussion tab for 30 days after creators receive the Community tab. Follow the instructions below.”''</small>}}</ref>, earlier known as “Channel Comments”. Therefore, all channels that reach a certain subscriber threshold (formerly 10000, 1500, 1000, now 500 as of {{datetime|2021-10-12}}<ref>{{URL|1=https://support.google.com/youtube/answer/9409631|2=Learn about Community posts - YouTube Help}}</ref>), all channel comments will be permanently erased instead of being merged or co-existing.


Although the support article was updated to use past-tense language and the opt-out form was made unavailable approximately as scheduled (around the beginning of 2021-07-23), the videos were not made private until around the beginning of 2021-07-28 UTC. The change occurred gradually over a number of hours in alphabetical order based on the video ID, starting around 2021-07-27 22:00 UTC and finishing around 2021-07-29 03:00 UTC (give or take an hour or two at both ends).
The owner from the YouTube channel is still able to access comments from the discussion tab for 30 days after their channel has received the community tab<ref name=discussion /> feature. For channel visitors, the “Discussion” tab immediately becomes inaccessible after it becomes replaced with the “Community” tab. Attempting to access the discussion tab via direct URL gets redirected to the main page of the channel.
 
As of 2021, the discussion tab on the mobile website now displays a message stating "The Discussion tab is no longer supported on Mobile Web. You can still access the Discussion tab on a computer or by using Desktop mode on a phone. Learn more". This change likely occurred at some point between {{datetime|2020-07-15}}<ref>{{URL|1=https://web.archive.org/web/20200715170644/https://m.youtube.com/user/1946Alberich/discussion?disable_polymer=1|2=Sample archive of mobile web discussion tab on {{datetime|2020-07-15}}}}</ref> and {{datetime|2021-06-02}}. It appears that the mobile app never supported the discussion tab.
 
On {{datetime|2021-09-09}}, YouTube announced that it would be removing all remaining Discussion tabs on {{datetime|2021-10-12}}. This will affect channels with 1,000 or fewer subscribers as well as channels which have reached more than 1,000 subscribers within the past week. On that date YouTube will also decrease the minimum number of subscribers required for a Community tab from over 1,000 to over 500<ref>{{URL|https://support.google.com/youtube/thread/125117248/expanding-community-posts-to-channels-with-500-subscribers-and-removing-the-discussion-tab}}</ref>.
 
[[User:Tech234a]] wrote a simple Python script to download discussions without an API key: https://github.com/tech234a/youtube-discussions-archive
 
Discussion tabs were removed from the YouTube web interface as scheduled on {{datetime|2021-10-12}}, with the earliest known report of removal occurring at 21:01 UTC. At some point afterwards (by {{datetime|2021-10-15|06:49}}) the drop down menu for enabling/disabling Discussion tabs or setting a moderation level was removed from YouTube Studio settings. The InnerTube API endpoint used by the Archive Team project scripts continued to return Discussion tab data for approximately another week, with data becoming inaccessible using this method by around {{datetime|2021-10-19|19:48}}. As of {{datetime|2021-10-20|05:56}}, Discussion tab data is still retrievable using the public YouTube Data API. However, by {{datetime|2021-11-09|05:54}}, the public YouTube Data API endpoint started returning empty responses (indicating 0 comments) for all channels. This change has not been documented in the official API changelog.
 
=== Archival Project ===
Archive Team ran a project from {{datetime|2021-10-10}} through {{datetime|2021-10-19}} to archive the InnerTube API data responses for Discussion tab data. In addition to retrieving data from channels with visible Discussion tabs, using the InnerTube API endpoint to retrieve data also allowed retrieval of Discussion tab comments from channels that had hidden their discussion tabs or had them replaced by community tabs. Note that this project did not archive the discussion pages themselves since on the Polymer YouTube layout those pages were just wrappers that contained no comments without first making a request to the InnerTube API used by this project.
 
Channels were discovered from various sources, including within the project by saving channels that posted comments on other channels.
 
The version of the script used during the first few hours of the project had a small bug that prevented archiving more than 10 replies for a given comment. This would have only affected a small number of comments on a small number of channels, and the first 10 replies were retrieved in all cases. The affected version of the script was 20211010.02. Unfortunately, these items were not retried.
 
Overall, the project successfully retrieved Discussion tabs for 257.96 million YouTube channels, with data totaling 2.13TiB, which will be made available in WARC format on the Internet Archive. Because all of the requests have the same two URLs differentiated by POST data, accessing this data using the Wayback Machine may be difficult.
 
<span id="dislikes"></span>
 
== Removal of public video dislikes (December 2021) ==
{{Infobox project
| title = YouTube Dislikes
| project_status = {{offline}} except for the person that uploaded the video and whitelisted API clients
| archiving_status = {{partiallysaved}}
| source = [https://github.com/ArchiveTeam/youtube-dislikes-grab youtube-dislikes-grab]
| tracker = [https://web.archive.org/web/20220309163428/https://tracker.archiveteam.org/youtube-dislikes/ youtube-dislikes]
| irc = down-the-tube
| data = {{IA collection|archiveteam_youtubedislikes}}
}}
 
YouTube announced on {{datetime|2021-11-10}} that they would remove public display of the display count to 'help better protect creators from harassment, and reduce dislike attacks'. This change was rolled out gradually, and API access will be disabled on December 13th. Dislike counts continue to be visible to the person that uploaded the video as well as whitelisted API clients.<ref>{{URL|1=https://support.google.com/youtube/thread/134791097/update-to-youtube-dislike-counts?hl=en}}</ref>


== Site reconnaissance  ==
== Site reconnaissance  ==


Little is known about its database, but according to [http://www.web-strategist.com/blog/2006/09/01/youtube-data-storage-stats-revealeddata from 2006], it was 45TB and doubling every 4 months. At this rate it would be '''660 Petabytes (Oct 2014)''' by now.
Little is known about its database, but according to [https://web.archive.org/web/20190909194120/http://www.web-strategist.com/blog/2006/09/01/youtube-data-storage-stats-revealed data from 2006], it was 45TB and doubling every 4 months. At this rate it would be '''660 Petabytes''' as of October 2014.


According to Leo Leung's calculations based on available information, [https://docs.google.com/spreadsheets/d/1w4TU9HZsajZtuvVlLZZgnNZsLVpYdnul47QE9iQq7Po/pub an often updated Google spreadsheet] estimates that in early 2015 YouTube's content reached 500 petabytes in size.
According to Leo Leung's calculations based on available information, [https://docs.google.com/spreadsheets/d/1w4TU9HZsajZtuvVlLZZgnNZsLVpYdnul47QE9iQq7Po/pub an often updated Google spreadsheet] estimates that in early 2015 YouTube's content reached 500 petabytes in size.


FYI, all of [[Google Video]] was about 45TB, and the Archive Team's previously biggest project, [[MobileMe]] was 200TB. The [[Internet Archive]]'s total capacity is 50PB as of August 2014. So let's hope YouTube stays healthy, because the Archive Team may have finally met its match.
Ethan Zuckerman's [https://ethanzuckerman.com/2023/12/22/how-big-is-youtube/ blog post] estimates that there are 13.3 billion videos on YouTube based on random sampling of search results, as of December 2023. The data is available on [https://tubestats.org/ TubeStats].
 
FYI, all of [[Google Video]] was about 45TB, and the Archive Team's current biggest project, [[URLs]] is 5.5PB. The [[Internet Archive]]'s total capacity is 150PB as of December 2023<ref>Used and Free Paired Space on https://archive.org/~tracey/stats</ref>. So let's hope YouTube stays healthy, because the Archive Team may have finally met its match.


== Vital signs ==
== Vital signs ==
Line 168: Line 248:


=== [[#Annotations_removal|Annotations]] ===
=== [[#Annotations_removal|Annotations]] ===
On November 27th, 2018, YouTube updated its <span title="Contents:&#x0A;“Update: We will stop showing existing annotations to viewers starting January 15, 2019. All existing annotations will be removed.”">{{url|https://support.google.com/youtube/answer/7342737|help page}}</span> to include that all annotations (which had been disabled for new videos and replaced with "cards" early May 2017, but old annotations remained visible) will be removed from videos hosted on the platform on 15 January 2019.
On {{datetime|2018-11-27}}, YouTube updated its <span title="Contents:&#x0A;“Update: We will stop showing existing annotations to viewers starting {{datetime|2019-01-15}}. All existing annotations will be removed.”">{{url|https://support.google.com/youtube/answer/7342737|help page}}</span> to include that all annotations (which had been disabled for new videos and replaced with "cards" early May 2017, but old annotations remained visible) will be removed from videos hosted on the platform on {{datetime|2019-01-15}}.


=== Discussions (Channel Comments) ===
=== Discussions (Channel Comments) ===
In addition, YouTube decided that their new “Community Post” feature can not co-exist with the “Discussion” feature<ref name=discussion>{{URL|https://support.google.com/youtube/answer/7124175|''YouTube Help'' article: “Engage with creators on Community posts” mentions <small>''“'''Note:''' As creators get the Community tab, it will replace the Discussion tab. You can access or delete any comments you left on the Discussion tab for 30 days after creators receive the Community tab. Follow the instructions below.”''</small>}}</ref>, earlier known as “Channel Comments”. Therefore, all channels that reach a certain subscriber threshold (formerly 10000, 1500, currently 1000<ref>{{URL|1=https://support.google.com/youtube/answer/9409631|2=Learn about Community posts - YouTube Help}}</ref>), all channel comments will be permanently erased instead of being merged or co-existing.
See [[#Discussions.2FChannel_Comments_.28October_2021.29|Discussions/Channel Comments (October 2021)]].
 
The owner from the YouTube channel is still able to access comments from the discussion tab for 30 days after their channel has received the community tab<ref name=discussion /> feature. For channel visitors, the “Discussion” tab immediately becomes inaccessible after it becomes replaced with the “Community” tab. Attempting to access the discussion tab via URL get redirected to the main page of the channel.
 
As of 2021, the discussion tab on the mobile website now displays a message stating "The Discussion tab is no longer supported on Mobile Web. You can still access the Discussion tab on a computer or by using Desktop mode on a phone. Learn more". This change likely occurred at some point between 2020-07-15<ref>{{URL|1=https://web.archive.org/web/20200715170644/https://m.youtube.com/user/1946Alberich/discussion?disable_polymer=1|2=Sample archive of mobile web discussion tab on 2020-07-15}}</ref> and 2021-06-02. It appears that the mobile app never supported the discussion tab.


=== Removal of AutoShare ===
=== Removal of AutoShare ===
Line 199: Line 275:




On 20190110, YouTube announced<ref>{{URL|1=https://www.influencerupdate.biz/news/67810/youtube-is-apparently-removing-the-ability-to-auto-share-videos-to-twitter/|2=''“YouTube is apparently removing the ability to auto-share videos to Twitter”'' – influencerupdate.biz}} </ref><ref>{{URL|1=https://techwiser.com/youtube-to-remove-automatic-sharing-to-twitter/|2=“''YouTube to Remove Automatic Sharing to Twitter – Here is How to Fix It''” article by TechWiser.com}}</ref><ref>{{URL|1=https://twitter.com/TeamYouTube/status/1083450619552194560|2=Tweet by @TeamYouTube: “After Jan 31st, we're saying goodbye 👋 to automated tweets like the one below.  You can still share your YouTube activity with your followers in more customized posts via the Share button. Full update →}} https://goo.gl/ef8Vc3”</ref> that they would get rid of that feature on 20190131.
On {{datetime|2019-01-10}}, YouTube announced<ref>{{URL|1=https://www.influencerupdate.biz/news/67810/youtube-is-apparently-removing-the-ability-to-auto-share-videos-to-twitter/|2=''“YouTube is apparently removing the ability to auto-share videos to Twitter”'' – influencerupdate.biz}} </ref><ref>{{URL|1=https://techwiser.com/youtube-to-remove-automatic-sharing-to-twitter/|2=“''YouTube to Remove Automatic Sharing to Twitter – Here is How to Fix It''” article by TechWiser.com}}</ref><ref>{{URL|1=https://twitter.com/TeamYouTube/status/1083450619552194560|2=Tweet by @TeamYouTube: “After Jan 31st, we're saying goodbye 👋 to automated tweets like the one below.  You can still share your YouTube activity with your followers in more customized posts via the Share button. Full update →}} https://goo.gl/ef8Vc3”</ref> that they would get rid of that feature on {{datetime|2019-01-31}}.


After the removal of ''AutoShare'', people still tweeted “I liked a @YouTube video […]” manually.
After the removal of ''AutoShare'', people still tweeted “I liked a @YouTube video […]” manually.
Line 216: Line 292:
* Total rating counts (no graph)
* Total rating counts (no graph)
* Like and dislike count (no graph)
* Like and dislike count (no graph)
* Referral sources with date of initial referral (exact dates available as of 20071130).
* Referral sources with date of initial referral (exact dates available as of {{datetime|2007-11-30}}).
** View counts from each referral source.
** View counts from each referral source.
*** First featured video view
*** First featured video view
Line 242: Line 318:


=== “One” channel layout ===
=== “One” channel layout ===
In March 2013, YouTube enforced all channels to change to the previously optional new channel layout called “One” channel layout. <ref name=1propaganda>{{URL|1=https://www.youtube.com/watch?v=bRJEmA6P1SQ|2=YouTube “One Channel” layout propaganda. 📅 Uploaded on 2013-03-08 01:06:18 UTC, 👁 309346 views, 👍 1125 likes, 👎 8648 dislikes, 💬4139 comments, mostly negative comments.}}</ref>
In March 2013, YouTube enforced all channels to change to the previously optional new channel layout called “One” channel layout. <ref name=1propaganda>{{URL|1=https://www.youtube.com/watch?v=bRJEmA6P1SQ|2=YouTube “One Channel” layout propaganda. 📅 Uploaded on {{datetime|2013-03-08|01:06:18}}, 👁 309346 views, 👍 1125 likes, 👎 8648 dislikes, 💬4139 comments, mostly negative comments.}}</ref>


==== Good bye, unified page. ====
==== Good bye, unified page. ====
Line 287: Line 363:
From March 2013 to August 2014<ref name=layout2014>{{URL|2=About YouTube's August 2014 layout that is still optionally accessible as of May 2019|1=https://thenextweb.com/google/2014/08/29/youtube-given-site-minor-facelift/}}</ref>, the video counter was still visible next the channel name,<ref name=belowvideo>{{URL|2=PSY's “Gangnam Style” hits 2 Billion views. YouTube UI below video as of 2013-2014.|1=https://socialblade.com/blog/wp-content/uploads/2014/06/Screen-Shot-2014-06-01-at-1.09.09-PM.png}}</ref> although clicking on it just refers to the ''/videos'' page of the uploaders channel instead of directly listing the recent uploads.<ref name=quickvideos>(Couldn't find a screenshot of this former feature in action (quickly see recent videos of uploader without needing to leave watch page, by clicking on video counter). Whoever finds a screenshot of the feature in action should please add it to this reference.)</ref>
From March 2013 to August 2014<ref name=layout2014>{{URL|2=About YouTube's August 2014 layout that is still optionally accessible as of May 2019|1=https://thenextweb.com/google/2014/08/29/youtube-given-site-minor-facelift/}}</ref>, the video counter was still visible next the channel name,<ref name=belowvideo>{{URL|2=PSY's “Gangnam Style” hits 2 Billion views. YouTube UI below video as of 2013-2014.|1=https://socialblade.com/blog/wp-content/uploads/2014/06/Screen-Shot-2014-06-01-at-1.09.09-PM.png}}</ref> although clicking on it just refers to the ''/videos'' page of the uploaders channel instead of directly listing the recent uploads.<ref name=quickvideos>(Couldn't find a screenshot of this former feature in action (quickly see recent videos of uploader without needing to leave watch page, by clicking on video counter). Whoever finds a screenshot of the feature in action should please add it to this reference.)</ref>


YouTube's August 2014 redesign<ref name=layout2014 />, which has slightly changed throughout the years, co-existed with the polymer design and was fully accessible until 2020-06-16 by appending the <code>&disable_polymer=1</code> or <code>&disable_polymer=true</code> parameter to the URL. This parameter later worked only on selected pages. In late 2020, this design required special headers to access, but as of 2020-12-14 the design became inaccessible entirely. As of 2021-06-02, only two YouTube pages are known to still use this design: <code>/attribution?v=[video ID]</code> and <code>/advanced_settings?ar=[current/recent Unix timestamp in milliseconds]</code>. It is expected that these pages will be updated or removed soon. As of approximately 2021-05-27 or 2021-05-28, the one known remaining feature of <code>/advanced_settings</code> was made obsolete and the link to that page was removed from the new YouTube Studio. The information on <code>/attribution</code> has been broken/missing for several years for many videos so it may also go away soon, though no timeline can be predicted for this.
YouTube's August 2014 redesign<ref name=layout2014 />, which has slightly changed throughout the years, co-existed with the polymer design and was fully accessible until {{datetime|2020-06-16}} by appending the <code>&disable_polymer=1</code> or <code>&disable_polymer=true</code> parameter to the URL. This parameter later worked only on selected pages. In late 2020, this design required special headers to access, but as of {{datetime|2020-12-14}} the design became inaccessible entirely. As of {{datetime|2021-06-02}}, only two YouTube pages are known to still use this design: <code>/attribution?v=[video ID]</code> and <code>/advanced_settings?ar=[current/recent Unix timestamp in milliseconds]</code>. It is expected that these pages will be updated or removed soon. As of approximately {{datetime|2021-05-27}} or {{datetime|2021-05-28}}, the one known remaining feature of <code>/advanced_settings</code> was made obsolete and the link to that page was removed from the new YouTube Studio. On June 10 2021, the <code>/advanced_settings</code> page was removed and then started returning 404 not found. As of 2022, it now redirets to a channel page. The information on <code>/attribution</code> has been broken/missing for several years for many videos so it may also go away soon, though no timeline can be predicted for this.
<br />Since that August 2014 redesign, the number of publicly available videos is also no longer visible directly from the watch page<ref name=layout2014 />. And also not from the channel's “about” page, but only from the search results.
<br />Since that August 2014 redesign, the number of publicly available videos is also no longer visible directly from the watch page<ref name=layout2014 />. And also not from the channel's “about” page, but only from the search results.


Line 293: Line 369:


=== Comment loading ===
=== Comment loading ===
Since YouTube's “One Channel Layout” redesign, comments “extraload” using [[Wikipedia:AJAX|AJAX]], which means that they do not get loaded within the page itself, but only start loading after scrolling down towards the comments. This made the comment section inaccessible to the Wayback Machine. However, there was a page called ''youtube.com/all_comments?v=<code><video ID></code>'', which actually loaded the first few comments without AJAX (included in HTML source code), but loading more comments required AJAX<!-- Did the comments load automatically on scrolldown or by pressing “load more comments”? Latter one likely true. all_comments originally worked with page numbers -->. But that one was discontinued and started redirecting to the main ''<code>/watch?v=</code>'' page since January of 2016. However, [[Archive.is#YouTube_comment_archival|archive.today]] still was able to archive YouTube comments until late 2017. As of April 2019, archiving YouTube comments using http://archive.today/ is still possible by linking directly to a comment using YouTube's <code>lc</code> URL parameter.
Since approximately July 9th, 2013,<ref>[http://web.archive.org/web/20130708084413/http://www.youtube.com/watch?v=tR1rvqW9fN0&gl=US&hl=en Watch page on July 8, 2013]; [http://web.archive.org/web/20130710000714/http://www.youtube.com/watch?v=9bZkp7q19f0 July 10, 2013]</ref> soon after YouTube's “One Channel Layout” redesign, comments “extraload” using [[Wikipedia:AJAX|AJAX]], which means that they do not get loaded within the page itself, but only start loading after scrolling down towards the comments. This made the comment section inaccessible to the Wayback Machine. However, there was a page called ''youtube.com/all_comments?v=<code><video ID></code>'', which actually loaded the first few comments without AJAX (included in HTML source code), but loading more comments required AJAX<!-- Did the comments load automatically on scrolldown or by pressing “load more comments”? Latter one likely true. all_comments originally worked with page numbers -->. But that one was discontinued and started redirecting to the main ''<code>/watch?v=</code>'' page since January of 2016. However, [[Archive.is#YouTube_comment_archival|archive.today]] still was able to archive YouTube comments until late 2017. As of April 2019, archiving YouTube comments using http://archive.today/ is still possible by linking directly to a comment using YouTube's <code>lc</code> URL parameter.


* Example comment URL: https://www.youtube.com/watch?v=W3GrSMYbkBE&lc=UgxC238Gea0KGOditl54AaABAg
* Example comment URL: https://www.youtube.com/watch?v=W3GrSMYbkBE&lc=UgxC238Gea0KGOditl54AaABAg
Line 302: Line 378:
* Example: https://www.youtube.com/watch?v=EaPfzgUFJyI&google_comment_id=UgyJ0eDN227JG2mDFg54AaABAg
* Example: https://www.youtube.com/watch?v=EaPfzgUFJyI&google_comment_id=UgyJ0eDN227JG2mDFg54AaABAg
* Archived: http://archive.vn/MRmNW
* Archived: http://archive.vn/MRmNW
[[Chromebot]], operated via {{IRC|ArchiveBot|network=hackint}} IRC, can still be used to archive YouTube comments thanks to it's bottomless page scrolling capabilities.


Comments on YouTube can be sorted by ''Top Comments'' or by ''Newest Comments''. The uploader of a video can specify which way of sorting is used for the video by default, but it can be adjusted manually by the user. The default preset is ''Top Comments''. However, <b>no known URL parameter is able to select the sorting methods for the comments so far.</b> Therefore, crawlers that can access the comments can only crawl them in the preselected way of sorting.
Comments on YouTube can be sorted by ''Top Comments'' or by ''Newest Comments''. The uploader of a video can specify which way of sorting is used for the video by default, but it can be adjusted manually by the user. The default preset is ''Top Comments''. However, <b>no known URL parameter is able to select the sorting methods for the comments so far.</b> Therefore, crawlers that can access the comments can only crawl them in the preselected way of sorting.
Line 318: Line 391:


=== Video reactions ===
=== Video reactions ===
YouTube's “Video Reactions” feature (2011) is similar to Facebook's {{URL|1=https://techcrunch.com/2016/02/24/facebook-reactions/|2=extended liking feature}}<ref>{{URL|1=https://techcrunch.com/2016/02/24/facebook-reactions/|2=TechCrunch article from 20160224: “Facebook Enhances Everyone’s Like With Love, Haha, Wow, Sad, Angry Buttons”.}}</ref> since 2016.
YouTube's “Video Reactions” feature (2011) is similar to Facebook's {{URL|1=https://techcrunch.com/2016/02/24/facebook-reactions/|2=extended liking feature}}<ref>{{URL|1=https://techcrunch.com/2016/02/24/facebook-reactions/|2=TechCrunch article from {{datetime|2016-02-24}}: “Facebook Enhances Everyone’s Like With Love, Haha, Wow, Sad, Angry Buttons”.}}</ref> since 2016.


The possible reactions were: “LOL, OMG, EPIC, CUTE, WTF or FAIL”.<ref>{{URL|1=https://www.adweek.com/digital/youtube-reactions/|2=''AdWeek'' article from 20110805:  “YouTube Trades In LOL, OMG, WTF Buttons For Drop Down Reactions Menu”.}}</ref><ref>{{URL|1=https://marketinghits.com/blog/youtube-testing-new-reaction-buttons-omg-epic-lol-fail-wtf-cute/ |2=''MarketingHits.com'' article “YouTube Testing New “Reaction” Buttons: OMG, Epic, LOL, Fail, WTF, & Cute”,}} originally from {{URL|1=http://www.reelseo.com/youtube-testing-reaction-buttons-omg-epic-lol-fail-wtf-cute/|2=ReelSEO}}.</ref><ref>{{URL|1=http://googlesystem.blogspot.com/2011/06/youtube-reactions.html|2=20110602 article by ''“Google Operating System”'' (unofficial blog): “YouTube Reactions”.}}</ref>
The possible reactions were: “LOL, OMG, EPIC, CUTE, WTF or FAIL”.<ref>{{URL|1=https://www.adweek.com/digital/youtube-reactions/|2=''AdWeek'' article from {{datetime|2011-08-05}}:  “YouTube Trades In LOL, OMG, WTF Buttons For Drop Down Reactions Menu”.}}</ref><ref>{{URL|1=https://marketinghits.com/blog/youtube-testing-new-reaction-buttons-omg-epic-lol-fail-wtf-cute/ |2=''MarketingHits.com'' article “YouTube Testing New “Reaction” Buttons: OMG, Epic, LOL, Fail, WTF, & Cute”,}} originally from {{URL|1=http://www.reelseo.com/youtube-testing-reaction-buttons-omg-epic-lol-fail-wtf-cute/|2=ReelSEO}}.</ref><ref>{{URL|1=http://googlesystem.blogspot.com/2011/06/youtube-reactions.html|2=20110602 article by ''“Google Operating System”'' (unofficial blog): “YouTube Reactions”.}}</ref>


To view reaction counts, one needed to be logged in.
To view reaction counts, one needed to be logged in.
Line 365: Line 438:
YouTube has an {{URL|https://support.google.com/youtube/answer/2801981|2=Inactive Accounts Policy}} which has text dating back to at least 2009 (it was originally included as part of the Username Squatting Policy, though it was moved and renamed to its current name around 2013 or 2014). This policy states that accounts that have not logged in for 6 months, or have not uploaded any videos, or are not actively watching/commenting on videos or channels may be "reclaimed". As of 2021, {{URL|https://datahorde.org/why-we-shouldnt-worry-about-youtubes-inactive-accounts-policy/|2=this policy is not in active use}}. YouTube would likely notify users via email if they decided to start enforcing this policy again in the future. This policy was noticed by archivists in December 2020.
YouTube has an {{URL|https://support.google.com/youtube/answer/2801981|2=Inactive Accounts Policy}} which has text dating back to at least 2009 (it was originally included as part of the Username Squatting Policy, though it was moved and renamed to its current name around 2013 or 2014). This policy states that accounts that have not logged in for 6 months, or have not uploaded any videos, or are not actively watching/commenting on videos or channels may be "reclaimed". As of 2021, {{URL|https://datahorde.org/why-we-shouldnt-worry-about-youtubes-inactive-accounts-policy/|2=this policy is not in active use}}. YouTube would likely notify users via email if they decided to start enforcing this policy again in the future. This policy was noticed by archivists in December 2020.


=== Attributions ===
On {{datetime|2021-08-18}}, YouTube announced video attribution pages would removed "after September"<ref>{{URL|https://support.google.com/youtube/answer/2797468}}</ref>. Video attribution pages were believed to be the final remaining page on YouTube to use the old, pre-polymer layout. As of some time between {{datetime|2021-09-20|05:19}}<ref>{{URL|https://web.archive.org/web/20210920210730/https://webcache.googleusercontent.com/search?q=cache%3ATKqcRDlg7U8J%3Ahttps%3A%2F%2Fwww.youtube.com%2Fattribution%3Fv%3DsYv6XRqRwDg+&cd=15&hl=en&ct=clnk&gl=us</ref> and {{datetime|2021-09-20|20:25}}, the video attributions page was removed and now returns a blank page with HTTP error code 410. As of this point, links to attribution pages still appear below video descriptions.
<!-- Section not ready yet:
<!-- Section not ready yet:
=== Volatile information ===
=== Volatile information ===
Line 372: Line 447:
When a comment has been edited once, no matter whether seconds or years after posting it, a comment will be marked as ''“(edited)”'' next to the relative time (absolute time can be requested via YouTube API but not included in website information, thus lost from web crawls), while keeping the original comment posting time.
When a comment has been edited once, no matter whether seconds or years after posting it, a comment will be marked as ''“(edited)”'' next to the relative time (absolute time can be requested via YouTube API but not included in website information, thus lost from web crawls), while keeping the original comment posting time.
-->
-->
=== Comment Removals ===
==== COPPA comment removal ====
* In early 2020, all comments for kids videos were removed to keep in touch with USA's COPPA privacy act and European Union's Rule.
==== Pedophilia comment removal ====
* YouTube removed comments of videos that contain kids in it to prevent pedophilia.
==== Music Comment Removal ====
* In late 2020, YouTube deactivated comments under auto-generated art tracks, though they later reverted that decision in 2023.<ref>[https://www.youtube.com/watch?v=bPKYcrLII9I Auto-generated Art Tracks and YouTube Comments] - {{datetime|2020-12-15}} (also see its description)</ref><ref>[https://support.google.com/youtube/answer/9706180 Learn why comments are disabled - YouTube help]</ref><ref>[https://reclaimthenet.org/youtube-mass-disables-comments-art-track-music-videos/ YouTube mass disables all comments on “Art Track” music videos, angering communities] - {{datetime|2020-12-18}}</ref>
=== Video Dislikes ===
See [[#Removal_of_public_video_dislikes_.28December_2021.29|Removal of public video dislikes (December 2021)]].


== Stories ==
== Stories ==
''“Stories”'' is a feature of YouTube<ref>{{URL|https://www.theverge.com/2018/11/29/18117670/youtube-stories-creators-subscribers-instagram-philip-defranco|201811291200 TheVerge article “YouTube is rolling out its Instagram-like Stories feature to more creators”}}</ref><ref>{{URL|1=https://support.google.com/youtube/answer/7573166?hl=en|2=YouTube Help article “Watch YouTube Stories”}}</ref><ref>{{URL|1=https://support.google.com/youtube/answer/7568166?hl=en|2=YouTube Help article: “YouTube Stories for creators”}}</ref><ref>{{URL|https://creatoracademy.youtube.com/page/lesson/stories|YouTube Creator Academy: Lessons: Stories.}}</ref> which works in a similar way to {{W2|Instagram Stories}}, {{W2|Facebook Stories}} and {{W2|Snapchat|Snapchat#Stories_and_Discover}}. It can exclusively be accessed from the YouTube application for mobile devices (not YouTube's mobile website or desktop website). <!--Restricted access for desktop users? We had that already. [[Instagram]]. --> It allows users, momentarily exclusively with channels that have over 10000 subscribers, to publish vertical videos, usually filmed from their mobile devices, and optionally edited with e.g. text, using their basic editing features.
''“Stories”'' was a feature of YouTube<ref>{{URL|https://www.theverge.com/2018/11/29/18117670/youtube-stories-creators-subscribers-instagram-philip-defranco|201811291200 TheVerge article “YouTube is rolling out its Instagram-like Stories feature to more creators”}}</ref><ref>{{URL|1=https://support.google.com/youtube/answer/7573166?hl=en|2=YouTube Help article “Watch YouTube Stories”}}</ref><ref>{{URL|1=https://support.google.com/youtube/answer/7568166?hl=en|2=YouTube Help article: “YouTube Stories for creators”}}</ref><ref>{{URL|https://creatoracademy.youtube.com/page/lesson/stories|YouTube Creator Academy: Lessons: Stories.}}</ref> which works in a similar way to {{W2|Instagram Stories}}, {{W2|Facebook Stories}} and {{W2|Snapchat|Snapchat#Stories_and_Discover}}. It can exclusively be accessed from the YouTube application for mobile devices (not YouTube's mobile website or desktop website). <!--Restricted access for desktop users? We had that already. [[Instagram]]. --> It allows users, momentarily exclusively with channels that have over 10000 subscribers, to publish vertical videos, usually filmed from their mobile devices, and optionally edited with e.g. text, using their basic editing features.
 
Creation of new Stories was discontinued in favor of Community Posts and Shorts on {{datetime|2023-06-26}}. All remaining Stories expired within 7 days<ref>{{URL|https://support.google.com/youtube/thread/217640760/youtube-stories-are-going-away-on-6-26-2023|YouTube Stories are Going Away on 6/26/2023}}</ref>.


=== Volatility ===
=== Volatility ===


“Story” videos automatically get deleted in 7 days (= 1 week) after being posted.
“Story” videos automatically got deleted in 7 days (= 1 week) after being posted.


=== Accessibility ===
=== Accessibility ===


“Stories” can be accessed from the “Stories” tab on YouTube channels when viewed in the mobile application.
“Stories” could be accessed from the “Stories” tab on YouTube channels when viewed in the mobile application.


== Removed or blocked channels ==
== Removed or blocked channels ==
* [https://sputniknews.com/world/201809121067978562-youtube-explanation-for-shutting-syrian-govt-channels/ Syrian government channels] got removed (2018)
* [https://sputniknews.com/world/201809121067978562-youtube-explanation-for-shutting-syrian-govt-channels/ Syrian government channels] got removed (2018)
* [https://www.youtube.com/user/hispantv/featured Iranian HispanTV channel] got blocked (2018) (Available from Australia FWIW)
* [https://www.youtube.com/channel/UCHb3cj2VXjAXC-mKZsRRspg Iranian HispanTV channel] got blocked (2018) (Available from Australia FWIW)
* [https://www.youtube.com/user/MaduroMoros Nicolas Maduro channel] (blocked?) (Available from Australia FWIW)
* [https://www.youtube.com/channel/UC4zVefzAohLcdju3uJpe-uQ Nicolas Maduro channel] (blocked?) (Available from Australia FWIW)
* See also: [[List of lost online videos]]
* See also: [[List of lost online videos]]


== Trivia ==
== Trivia ==
* All ytimg servers (where YouTube saved images, stylesheets and the .swf file of the flash-based YouTube video player) used to be intoxicated by [[robots.txt]] (“<code>user-agent:* disallow:/ noindex:/</code>”). When browsing YouTube layouts starting circa 2008 using the Wayback Machine, the website could only be viewed as black text on white background, just with different sizes (e.g. <nowiki><h1> video title</nowiki>), due to the missing images and stylesheet information. Only information visible in the HTML source code of the page itself could be rendered. '''On February 29th of 2012, the robots.txt file vanished off the ytimg servers,''' lifting the restrictions and making YouTube more properly browse-able through the Wayback Machine.
* All ytimg servers (where YouTube saved images, stylesheets and the .swf file of the flash-based YouTube video player) used to be intoxicated by [[robots.txt]] (“<code>user-agent:* disallow:/ noindex:/</code>”). When browsing YouTube layouts starting circa 2008 using the Wayback Machine, the website could only be viewed as black text on white background, just with different sizes (e.g. <nowiki><h1> video title</nowiki>), due to the missing images and stylesheet information. Only information visible in the HTML source code of the page itself could be rendered. '''On {{datetime|2012-02-29}}, the robots.txt file vanished off the ytimg servers,''' lifting the restrictions and making YouTube more properly browse-able through the Wayback Machine.
* Yahoo and Bing Video results might contain titles, upload dates and thumbnails of unavailable YouTube videos (i.e. after privated, deleted or channel terminated). Please help saving them to [[Archive.Today]] and [[ArchiveBot]]/[[ChromeBot]] if found, before they become purged from the results.
* Yahoo and Bing Video results might contain titles, upload dates and thumbnails of unavailable YouTube videos (i.e. after privated, deleted or channel terminated). Please help saving them to [[Archive.Today]] and [[ArchiveBot]] if found, before they become purged from the results.
** Yahoo URL format: https://video.search.yahoo.com/yhs/search?hspart=ddc&p=<code>search query</code>
** Yahoo URL format: https://video.search.yahoo.com/yhs/search?hspart=ddc&p=<code>search query</code>
** Bing URL format: http://www.bing.com/videos/search?FORM=HDRSC3&q=<code>search query</code>
** Bing URL format: http://www.bing.com/videos/search?FORM=HDRSC3&q=<code>search query</code>
== Finding Archived Videos ==
The [https://findyoutubevideo.thetechrobo.ca/ YouTube Video Finder] can be used to search several different YouTube archives for individual YouTube video IDs.


== References ==
== References ==
Line 403: Line 495:


== External links ==
== External links ==
* {{url|1=http://youtube.com}}
* [https://web.archive.org/web/20170102020347/https://www.youtube.com/yt/press/statistics.html Official YouTube statistics (until 2017)]
* {{url|1=https://www.youtube.com/yt/press/statistics.html|2=Statistics}}
* [https://tubestats.org/ TubeStats] (via [https://doi.org/10.51685/jqd.2023.022  Dialing for Videos: A Random Sample of YouTube] / [https://ethanzuckerman.com/2023/12/22/how-big-is-youtube/ How big is YouTube])
* [https://en.wikipedia.org/wiki/List_of_the_most_subscribed_users_on_YouTube List of the most subscribed users on YouTube]
* [https://en.wikipedia.org/wiki/List_of_the_most_subscribed_users_on_YouTube List of the most subscribed users on YouTube]
* [https://en.wikipedia.org/wiki/List_of_most_viewed_YouTube_videos List of most viewed YouTube videos]
* [https://en.wikipedia.org/wiki/List_of_most_viewed_YouTube_videos List of most viewed YouTube videos]
* [https://www.youtube.com/watch?v=kfECXqZtlR8 Video by “TheWheatIsHeat96”: “I hate when good videos on YouTube get deleted, private, blocked in our country.”, published on 2014-05-11 21:14:29 UTC.]


{{youtube}}
{{youtube}}

Latest revision as of 03:24, 22 February 2024

YouTube Metadata & Selected Videos
YouTube logo
Saving metadata for as many videos as possible including watch pages, captions, comments, thumbnails as well as saving selected YouTube videos
Saving metadata for as many videos as possible including watch pages, captions, comments, thumbnails as well as saving selected YouTube videos
URL YouTube[IAWcite.todayMemWeb]
Status Online! but possibly Endangered, see Vital signs
Archiving status In progress... (videos manually submitted)
Archiving type DPoS
Project source youtube-grab
Project tracker youtube
IRC channel #down-the-tube (on hackint)
Project lead Arkiver
Data[how to use] archiveteam_youtube
YouTube Videos
YouTube logo
Saving selected YouTube videos. See #youtubearchive's website for scope and details.
Saving selected YouTube videos. See #youtubearchive's website for scope and details.
URL YouTube[IAWcite.todayMemWeb]
Status Online! but possibly Endangered, see Vital signs
Archiving status On hiatus (videos manually submitted)
Archiving type Unknown
IRC channel #youtubearchive (on hackint)
Project lead Ivan
YouTube homepage screenshot as of 2011-04-22.

YouTube is a video sharing website currently owned by Google. YouTube is currently the most popular video hosting website on Earth.

Archive Team project

Archive Team launched a project to archive certain videos and channels from YouTube. Videos archived in this project are ingested into the Wayback Machine, and will be playable in the Wayback Machine (WBM) after being added to a separate index (may take a few days). Note: YouTube includes your worker's IP address in the videoplayback urls. As such, your worker's IP will be recorded in the project's data.

Scope

Anyone is able to queue channels and videos to be archived in channel #down-the-tube (on hackint). However, the videos and channels to be archived should be limited to:

  • Channels/videos almost sure to be deleted, like:
    • Company gone bankrupt
    • Person died
    • YouTube banning certain channels/content
  • The following channels/videos if they are connected to a major event that could cause them to be changed heavily or deleted:
    • Government channels
    • News channels
    • Event channels
    • Political party channels
    • Politician accounts
  • Unlisted videos (hidden from YouTube search, only accessible by URL)
  • Advertisement video that YouTube actually treats as advertisement
  • 'Newsworthy' content, for example:
    • Channels/videos from opinion makers in the Russia-Ukraine war
    • Channels/videos containing content of major events (terror attacks, wars, etc.)
      • Only queue an entire channel if the entire channel is related to the major event.
  • Others are possible, check in with Arkiver on IRC.

When a channel of video is queued for archival, a reason should be given with -e/--explain.

Why download?

Just like local backups, it is advisable to download videos regularly, as any which is not backed up is hanging by a thread above the memory hole. Even downloading at a low resolution such as 240p or 360p still is infinitely better than nothing (0p).

While established platforms typically have a solid and failure-resistant technical infrastructure, any content is still endangered from human factors. Not only could a creator run afoul of YouTube's site staff (or any other platform for that matter), or the platform change their policies in a way that affects existing content retroactively, but a creator themselves might run into public controversity and according scrutiny, or emotionally decide that they no longer like some of their early videos, and as eternal as they once appeared, they are gone in a flash, or locked away in the vault of private videos.

Video creators are recommended to upload their work to no less than two platforms. If platform staff decides take down videos or even nuke an entire channel, which is done with the press of a button and takes an entire stash of videos with it, viewers will know where to still find those videos.

"Google [parent company of YouTube] is an archive in the same way a supermarket is a food museum." –– Jason Scott
"For many years now I “favourite” videos on YT by downloading them. You can't expect any video on YT to still be there tomorrow/later."[1]

Archiving tools

Several free FLV downloaders and video-to-URL converters exist on the web. AT rescue projects usually use yt-dlp.
YouTube annotations (speech bubbles and notes) are available as XML

http://www.youtube.com/api/reviews/y/read2?feat=TCS&video_id=

To transform this XML to SRT, use ann2srt

(Automatic) tubeup.py - Youtube Video IA Archiver

Note: When uploading to the Internet Archive, please avoid exposing the site to legal risk by adhering to their terms of service for blatantly copyrighted content. Unfortunately, they are subject to similar threats of DMCA takedowns as YouTube, so do use discretion.
Note: Be very careful dumping channels over 100 videos with this script. Let an admin know what you're doing, dump 50 videos, and have a collection created. Work is being started on adding a flag to specify a collection name instead of "Community Video" which is what it defaults to. Always try to create an item. For the time being the script will have to be hand edited to specify a different collection.

tubeup.py is an automated archival script that uses yt-dlp to download a Youtube video (or any other provider supported by yt-dlp), and then uploads it with all metadata to the Internet Archive.

This way, all metadata from the video, such as title, tags, categories, and description, are preserved in the corresponding Internet Archive item, without having to manually enter it.

It also creates a standardized Internet Archive item name format that makes it easy to find the video using the Youtube ID, and reduces duplication: https://archive.org/details/youtube-v9sGhNoSG3o

yt-dlp also works with many other video sites.

(Manual) Recommended way to archive YouTube videos

First, download the video/playlist/channel/user using yt-dlp:

yt-dlp --continue --retries 4 --write-info-json --write-description --write-thumbnail --write-annotations --all-subs --ignore-errors -f bestvideo+bestaudio URL

This can be simplified by running the script by emijrp and others, which also handles upload.

You need a recent (2014) ffmpeg or avconv for the bestvideo+bestaudio muxing to work.

Also, make sure you're using the most recent version of yt-dlp.

Then, upload it to https://archive.org/upload/ Make sure to upload not only the video itself (.mp4 and/or .mkv files), but also the metadata files created along with it (.info.json, .jpg, .annotations.xml and .description).

To capture a WARC using Warcprox, once Warcprox is running, the URL and port of the proxy can be passed to yt-dlp with the proxy parameter, for example: --proxy="localhost:8000"

Two other yt-dlp parameters for getting some more data are --write-all-thumbnails and --write-auto-sub.

Wayback Machine

Archiveteam1.png This section is a draft.

You can submit YouTube URLs to the Wayback Machine, and the video will be archived on the Wayback Machine. I haven't checked if this still works but it used to possible. Some YouTube videos are on the Wayback Machine.

However, many of those videos that were saved stopped being accessible at some point in time due to an internal mistake at the Internet Archive. This is reported to be fixed, but recently, for a video that did not load, it wasn't lost to the sands of time, I managed to access the complete video by using the right URL: https://web.archive.org/web/2oe_/http://wayback-fakeurl.archive.org/yt/VideoID

(For more detail, see YouTube videos that were once accessible on the Wayback Machine have since disappeared.)

Tools to check for regional restrictions

Annotations removal (January 2019)

YouTube Annotations
Youtube-annotations-example-fukkireta.jpg
Status Offline as of 2019-01-15 ~15:00 UTC
Archiving status Partially saved
Archiving type Unknown
Project source omarroth/archive
IRC channel #archiveteam-bs (on hackint)
Project lead /u/omarroth
Data[how to use] youtubeannotations and youtubeannotations.tar.zstd

Annotations[2] were notes that could be added to videos after the upload, they provided plenty customization options such as color, size and internal & external links. People used them to correct mistakes in videos, created mini-games within YouTube and abused them too: "Please like and subscribe", "Watch in HD!!!"

On 2017-05-02 YouTube disabled editing and creating annotations for videos. On 2019-01-16 they were removed completely from the site and API responses (~15:00 UTC, same time when they disabled editing).

An archiving project "YouTube Annotation Archive"[3][4][5] organized by u/omarroth that achieved to discover ~1.4 billion videos and download the annotations, if they had any. The project did not completely finish due to the deadline. The annotation archive is available at youtubeannotations, and a compressed copy can be found at youtubeannotations.tar.zstd. 16GB of just video IDs that were encompassed by the project can be downloaded here.

Liked lists (December 2019)

YouTube Liked Lists
Status Offline
Archiving status Partially saved
Archiving type Unknown
Project source youtube-likedlists-grab
Project tracker youtube-likedlists
IRC channel #down-the-tube (on hackint)
Project lead marked
Data[how to use] archiveteam_youtubelikes

On 2019-12-05, the playlists containing users' liked videos were made private.[6]

Playlist notes (October 2020)

YouTube playlist notes
Status Offline
Archiving status Partially saved
Archiving type Unknown
Project source youtube-playlistnotes-grab
Project tracker youtube-playlistnotes
IRC channel #down-the-tube (on hackint)
Data[how to use] archiveteam_youtube?query=playlist+notes

On 2020-10-08, playlist notes were removed because they were 'rarely used'.[7] While the official documentation only talks of 'private notes', these could be public as well. Although the notes are no longer displayed, they are still included in the Polymer page source code for playlists as of 2021-07-11 and 2023-11-11.

Older unlisted videos (July 2021)

YouTube Older Unlisted Videos
Status Offline except for users who opted their channels out or have made/will make individual videos public or unlisted again
Archiving status Partially saved - saved 5,739,754 videos in 360p (including watch pages, captions, comments, thumbnails), 201.98 TiB total
Archiving type Unknown
Project source youtube-grab
Project tracker youtube
IRC channel #down-the-tube (on hackint)
Data[how to use] archiveteam_youtube

On 2021-06-23, YouTube announced that they would make all unlisted videos uploaded before 2017-01-01 private in one month, on 2021-07-23, unless the uploader opted-out of the change[8][9]. Uploaders could also opt all of the videos on their channel out of this change by completing a form[10]. This change is likely because YouTube video IDs are believed to have previously been generated from an encrypted sequential integer instead of randomly, meaning that according to YouTube, the possibility of an unlisted URL being guessed was too significant. What makes the argument dubious is that the possibility is marginal to begin with, and even in the unlikely possibility of it happening, that would be one view of a random unlisted video from somewhere in the world, meaning the possibility of harm from guessed URLs did practically not exist to begin with. An HTTP 429 rate limit would have presumably been hit anyway well before any URL could be guessed through brute-forcing. Additionally, uploaders of unlisted videos consented from the beginning to the ever-so-slight possibility of their video's URL being guessed. And if this alleged security risk was known since 2017, it would mean that Google has withheld this information from users for four years.

It is expected that unlisted content from older, inactive channels will be lost because of this change.

Although the support article was updated to use past-tense language and the opt-out form was made unavailable approximately as scheduled (around the beginning of 2021-07-23), the videos were not made private until around the beginning of 2021-07-28 UTC. The change occurred gradually over a number of hours in alphabetical order based on the video ID, starting around 2021-07-27 22:00 UTC and finishing around 2021-07-29 03:00 UTC (give or take an hour or two at both ends).

Discussions/Channel Comments (October 2021)

YouTube Discussions
Status Offline on 2021-10-12 (UI), 2021-10-19 (InnerTube API), before 2021-11-09 (Public API)
Archiving status Saved! 257.96 million channels, 2.13TiB
Archiving type Unknown
Project source youtube-discussions-grab
Project tracker youtube-discussions
IRC channel #down-the-tube (on hackint)
Data[how to use] archiveteam_youtube?query=discussions

In addition, YouTube decided that their new “Community Post” feature can not co-exist with the “Discussion” feature[11], earlier known as “Channel Comments”. Therefore, all channels that reach a certain subscriber threshold (formerly 10000, 1500, 1000, now 500 as of 2021-10-12[12]), all channel comments will be permanently erased instead of being merged or co-existing.

The owner from the YouTube channel is still able to access comments from the discussion tab for 30 days after their channel has received the community tab[11] feature. For channel visitors, the “Discussion” tab immediately becomes inaccessible after it becomes replaced with the “Community” tab. Attempting to access the discussion tab via direct URL gets redirected to the main page of the channel.

As of 2021, the discussion tab on the mobile website now displays a message stating "The Discussion tab is no longer supported on Mobile Web. You can still access the Discussion tab on a computer or by using Desktop mode on a phone. Learn more". This change likely occurred at some point between 2020-07-15[13] and 2021-06-02. It appears that the mobile app never supported the discussion tab.

On 2021-09-09, YouTube announced that it would be removing all remaining Discussion tabs on 2021-10-12. This will affect channels with 1,000 or fewer subscribers as well as channels which have reached more than 1,000 subscribers within the past week. On that date YouTube will also decrease the minimum number of subscribers required for a Community tab from over 1,000 to over 500[14].

User:Tech234a wrote a simple Python script to download discussions without an API key: https://github.com/tech234a/youtube-discussions-archive

Discussion tabs were removed from the YouTube web interface as scheduled on 2021-10-12, with the earliest known report of removal occurring at 21:01 UTC. At some point afterwards (by 2021-10-15 06:49 UTC) the drop down menu for enabling/disabling Discussion tabs or setting a moderation level was removed from YouTube Studio settings. The InnerTube API endpoint used by the Archive Team project scripts continued to return Discussion tab data for approximately another week, with data becoming inaccessible using this method by around 2021-10-19 19:48 UTC. As of 2021-10-20 05:56 UTC, Discussion tab data is still retrievable using the public YouTube Data API. However, by 2021-11-09 05:54 UTC, the public YouTube Data API endpoint started returning empty responses (indicating 0 comments) for all channels. This change has not been documented in the official API changelog.

Archival Project

Archive Team ran a project from 2021-10-10 through 2021-10-19 to archive the InnerTube API data responses for Discussion tab data. In addition to retrieving data from channels with visible Discussion tabs, using the InnerTube API endpoint to retrieve data also allowed retrieval of Discussion tab comments from channels that had hidden their discussion tabs or had them replaced by community tabs. Note that this project did not archive the discussion pages themselves since on the Polymer YouTube layout those pages were just wrappers that contained no comments without first making a request to the InnerTube API used by this project.

Channels were discovered from various sources, including within the project by saving channels that posted comments on other channels.

The version of the script used during the first few hours of the project had a small bug that prevented archiving more than 10 replies for a given comment. This would have only affected a small number of comments on a small number of channels, and the first 10 replies were retrieved in all cases. The affected version of the script was 20211010.02. Unfortunately, these items were not retried.

Overall, the project successfully retrieved Discussion tabs for 257.96 million YouTube channels, with data totaling 2.13TiB, which will be made available in WARC format on the Internet Archive. Because all of the requests have the same two URLs differentiated by POST data, accessing this data using the Wayback Machine may be difficult.

Removal of public video dislikes (December 2021)

YouTube Dislikes
Status Offline except for the person that uploaded the video and whitelisted API clients
Archiving status Partially saved
Archiving type Unknown
Project source youtube-dislikes-grab
Project tracker youtube-dislikes
IRC channel #down-the-tube (on hackint)
Data[how to use] archiveteam_youtubedislikes

YouTube announced on 2021-11-10 that they would remove public display of the display count to 'help better protect creators from harassment, and reduce dislike attacks'. This change was rolled out gradually, and API access will be disabled on December 13th. Dislike counts continue to be visible to the person that uploaded the video as well as whitelisted API clients.[15]

Site reconnaissance

Little is known about its database, but according to data from 2006, it was 45TB and doubling every 4 months. At this rate it would be 660 Petabytes as of October 2014.

According to Leo Leung's calculations based on available information, an often updated Google spreadsheet estimates that in early 2015 YouTube's content reached 500 petabytes in size.

Ethan Zuckerman's blog post estimates that there are 13.3 billion videos on YouTube based on random sampling of search results, as of December 2023. The data is available on TubeStats.

FYI, all of Google Video was about 45TB, and the Archive Team's current biggest project, URLs is 5.5PB. The Internet Archive's total capacity is 150PB as of December 2023[16]. So let's hope YouTube stays healthy, because the Archive Team may have finally met its match.

Vital signs

Will be living off Google for a long time if nothing changes.

Advertising policies

Around early 2017, numerous content creators have expressed concerns about recent changes with YouTube's advertising policies, and many have also noticed sharp drops in ad revenue as a result, with some creators like Casey Neistat and h3h3Productions expressing existential fears. While not necessarily a cause for imminent alarm, the situation should be watched closely in the event that a positive feedback loop was to begin with a creator exodus.

Annotations

On 2018-11-27, YouTube updated its <span title="Contents: “Update: We will stop showing existing annotations to viewers starting 2019-01-15. All existing annotations will be removed.”">help page[IAWcite.todayMemWeb] to include that all annotations (which had been disabled for new videos and replaced with "cards" early May 2017, but old annotations remained visible) will be removed from videos hosted on the platform on 2019-01-15.

Discussions (Channel Comments)

See Discussions/Channel Comments (October 2021).

Removal of AutoShare

AutoShare was a YouTube feature, introduced in 2009[17], that allowed users to automatically share the following actions on YouTube to Twitter, Google Plus and formerly Facebook[17] automatically:

  • Liking a video.
  • Publishing a video.
  • Adding video to playlist (AutoShare did not share name and URL of the target playlist.)
  • Favourising a video (removed YouTube feature, existing favourite videos automatically got moved to “Favourites” playlist.)


The tweet or Google+ post posted by AutoShare included the title and the URL to the target video.
An AutoShared tweet's metadata source tag[18] indicates “Google”.

Example tweet:[IAWcite.todayMemWeb]

I liked a @YouTube video https://youtu.be/By_Cn5ixYLg?a YouTube Rewind 2018 but it's actually good

At some point, YouTube AutoShare tweets also included the Twitter handle[IAWcite.todayMemWeb] of the YouTube channel the video was shared from, if the owner of the YouTube channel had also linked his Twitter account to his YouTube channel and publicly listed it on the channel.


Twitter's tweet search feature allowed finding out the URL or the title of a deleted video from each other, or the original video URL and full title from knowing the approximate title.
It could also help to trace back changes of a video's title.

This information was useful to find out more about lost videos, e.g. what caused them to be removed, captures by Wayback/Archive.Today, online discussions and potential re-uploads.


On 2019-01-10, YouTube announced[19][20][21] that they would get rid of that feature on 2019-01-31.

After the removal of AutoShare, people still tweeted “I liked a @YouTube video […]” manually.

Video Statistics

Without any clear warning in advance, YouTube has also entirely removed the public “video statistics” feature. The only warning sign was that their new “Polymer” website layout lacked the statistics feature, which could originally be found in the “More” menu below a video.

Only their “One” website layout that they introduced in 2013 and revised in late 2014[22], then barely changed for years, co-existing with “Polymer” since late 2017, included the statistics feature.
When YouTube released their “One” website layout, the number of parameters displayed by the video statistics was also reduced.[23]

The video statistics were also inaccessible from the “One” website since late 2018, and the API endpoints were removed as well.[24].

Parameters (legacy)[25]

  • Total view count + graph.
  • Total comments count + graph.
  • Total favourited count + graph (removed YouTube feature).
  • Total rating counts (no graph)
  • Like and dislike count (no graph)
  • Referral sources with date of initial referral (exact dates available as of 2007-11-30).
    • View counts from each referral source.
      • First featured video view
      • First referral from related videos (separately mentioned)
      • First view from embededd video
      • First view from embededd video (specified website)
      • First referral from YouTube search (for separate search terms)
  • Most popular audiences (age and gender)
  • World map that highlights countries in which the video is more popular.
    • Countries with higher popularity are highlighted in darker green.
  • Honour badges
    • Total honour counts
    • List of honours
      • Rank: Most viewed of all time
      • This list is incomplete yet.

Parameters (recent)

  • Views
  • Total Watch time
    • Average watch time per user.
  • Subscribers gained from said video
  • Share counter: How often the video was shared.

Each of these parameters could be viewed as total cumulative count or counts per day, of which the latter was the default setting.

“One” channel layout

In March 2013, YouTube enforced all channels to change to the previously optional new channel layout called “One” channel layout. [26]

Good bye, unified page.

“One” channel layout crippled the Wayback Machine's ability to crawl channel pages.
The Wayback Machine often just captures the home page of a YouTube channel and misses the channel subpage tabs.

Since “One” channel layout was introduced in March 2013[26], the information of YouTube channels is no longer unified onto a single page, but has been split up onto different, separate subpages:

  • /featured (possibly formerly /home)
  • /videos
  • /playlists
  • /community
  • /store
  • /channels
  • /discussion
  • /about
    • Does include channel views, total view count, subscriber count and channel creation date.
    • Includes user-added information such as E-Mail address (ReCaptcha needed to access due to spam prevention), channel description, user-specified country, link farm.
  • /search
  • /feed (likely discontinued)
    • (Includes “shared thoughts”[IAWcite.todayMemWeb]).
    • Thoughts shared via the “Share your thoughts” feature (not to be confused with discussions or community) can only be written with the legacy channel layout (accessible via ?disable_polymer=1 URL parameter) but read via the new polymer channel layout too.
    • This feature is potentially endangered due to the lack of support from polymer layout, and depreciation of purpose. Existing thoughts shared via this feature could be deleted.

Some channels have the custom layout[27] feature turned off. In that case, the channel description shows on the channel's home page[28], but the discussion page (usually not enabled by default in the customized channel layout mode) is enabled and accessible via the URL.

Because all this information is no longer on a single page, if the Wayback Machine captures a channel page, it often only captures the main (home) page of the channel, dismissing the information from the other channel tabs because they are on different pages instead of all information on one page.

Custom channel layouts

Due to the lack of customizability of the “One Channel Layout”, as of March 2013, all custom creative channel designs have been deleted. Also other information such as channel view counts are no longer shown, but channel view counts can still be accessed via the YouTube API.[29]

Video counter

Prior to that layout change, when YouTube video title stood above instead of below the video[30], there was a menu where one could quickly see other videos uploaded by the channel.[31] If this feature existed today, it could have helped chromebot save information about the 20 most recent uploads of a channel the video was archived from.

From March 2013 to August 2014[22], the video counter was still visible next the channel name,[32] although clicking on it just refers to the /videos page of the uploaders channel instead of directly listing the recent uploads.[31]

YouTube's August 2014 redesign[22], which has slightly changed throughout the years, co-existed with the polymer design and was fully accessible until 2020-06-16 by appending the &disable_polymer=1 or &disable_polymer=true parameter to the URL. This parameter later worked only on selected pages. In late 2020, this design required special headers to access, but as of 2020-12-14 the design became inaccessible entirely. As of 2021-06-02, only two YouTube pages are known to still use this design: /attribution?v=[video ID] and /advanced_settings?ar=[current/recent Unix timestamp in milliseconds]. It is expected that these pages will be updated or removed soon. As of approximately 2021-05-27 or 2021-05-28, the one known remaining feature of /advanced_settings was made obsolete and the link to that page was removed from the new YouTube Studio. On June 10 2021, the /advanced_settings page was removed and then started returning 404 not found. As of 2022, it now redirets to a channel page. The information on /attribution has been broken/missing for several years for many videos so it may also go away soon, though no timeline can be predicted for this.
Since that August 2014 redesign, the number of publicly available videos is also no longer visible directly from the watch page[22]. And also not from the channel's “about” page, but only from the search results.

However, SocialBlade[IAWcite.todayMemWeb] tracks the statistics of as many YouTube channels as possible, storing historical data for three years[33].

Comment loading

Since approximately July 9th, 2013,[34] soon after YouTube's “One Channel Layout” redesign, comments “extraload” using AJAX, which means that they do not get loaded within the page itself, but only start loading after scrolling down towards the comments. This made the comment section inaccessible to the Wayback Machine. However, there was a page called youtube.com/all_comments?v=<video ID>, which actually loaded the first few comments without AJAX (included in HTML source code), but loading more comments required AJAX. But that one was discontinued and started redirecting to the main /watch?v= page since January of 2016. However, archive.today still was able to archive YouTube comments until late 2017. As of April 2019, archiving YouTube comments using http://archive.today/ is still possible by linking directly to a comment using YouTube's lc URL parameter.

As of June 2019 YouTube's google_comment_id URL parameter does the same thing as lc (maybe it was created as a replacement for lc which might not do anything in the future, speculation of course):

Comments on YouTube can be sorted by Top Comments or by Newest Comments. The uploader of a video can specify which way of sorting is used for the video by default, but it can be adjusted manually by the user. The default preset is Top Comments. However, no known URL parameter is able to select the sorting methods for the comments so far. Therefore, crawlers that can access the comments can only crawl them in the preselected way of sorting.

Trivia:

  • The Top Comments are not necessarily the comments with the highest number of upvotes, probably because YouTube does not want always the same comments to stay on top for too long. Older comments get pushed down despite having a high rating.
  • At some point, YouTube started hiding negative comment ratings and only shows how often a comment has been rated positively. Rating a comment negatively is still possible however. It's effect is pushing comments further down from the Top Comments.

Video Responses

“Video responses” was a feature where users could respond to videos with new or existing videos that were listed above the comment sections.

After YouTube removed that feature, the existing videos that were posted as response were not removed. But the links between the original video and the video used as response were removed.

Video reactions

YouTube's “Video Reactions” feature (2011) is similar to Facebook's extended liking feature[IAWcite.todayMemWeb][35] since 2016.

The possible reactions were: “LOL, OMG, EPIC, CUTE, WTF or FAIL”.[36][37][38]

To view reaction counts, one needed to be logged in.

But the feature got removed within just months after it's initial release, erasing all existing reactions.

Reasons for video deletions

When a video gets removed for any reason (e.g. manually removed by uploader; uploader closed channel; guideline issues; channel terminated) or not available in a specific region, YouTube usually displays the reason for why the video can not be played.

However, at some point (estimatedly 2013 or 2014), YouTube started purging unavailability reasons for manually deleted videos and videos and channel pages of closed channels, meaning that YouTube shows “This video does not exist.” or “This video is unavailable.”, as if the video never existed in first place.

Since then, reasons for videos that have been permanently erased will only be displayed for a very limited time, after which the video reads as unavailable instead of removed by user or closed channel.

Example:

YouTube does not differentiate between these videos after the deletion reason gets purged.


Since 2017, a private video will also indicate as “not available” (same as a video that never existed in first place) immediately after privated, while playlists that contain the unavailable videos still indicate the distinction between private and deleted videos.

YouTu.be

YouTu.be used to be an image hoster back in 2006. In late 2015, they created a robots.txt file that disallows all crawlers (or “User-agent: * Disallow:/”).


yt.be

YouTube also operates a URL shortener at yt.be. It uses Firebase Dynamic Links.


Bulletins

YouTube used to have a feature called “Bulletins” in it's earlier years, likely introduced in 2006.[39]

After co-existing with “Channel Comments” that would later become “Discussions”, that feature got removed at some point, erasing all existing Bulletin posts.


das_captcha

From approximately 2010 to 2012, a whole generation of Wayback Machine captures of YouTube pages were redirected to /das_captcha (example)[IAWcite.todayMemWeb] by HTTP 303 request, rendering these Wayback captures next to useless.

Inactive accounts policy

YouTube has an Inactive Accounts Policy[IAWcite.todayMemWeb] which has text dating back to at least 2009 (it was originally included as part of the Username Squatting Policy, though it was moved and renamed to its current name around 2013 or 2014). This policy states that accounts that have not logged in for 6 months, or have not uploaded any videos, or are not actively watching/commenting on videos or channels may be "reclaimed". As of 2021, this policy is not in active use[IAWcite.todayMemWeb]. YouTube would likely notify users via email if they decided to start enforcing this policy again in the future. This policy was noticed by archivists in December 2020.

Attributions

On 2021-08-18, YouTube announced video attribution pages would removed "after September"[40]. Video attribution pages were believed to be the final remaining page on YouTube to use the old, pre-polymer layout. As of some time between 2021-09-20 05:19 UTC[41] and 2021-09-20 20:25 UTC, the video attributions page was removed and now returns a blank page with HTTP error code 410. As of this point, links to attribution pages still appear below video descriptions.

Comment Removals

COPPA comment removal

  • In early 2020, all comments for kids videos were removed to keep in touch with USA's COPPA privacy act and European Union's Rule.

Pedophilia comment removal

  • YouTube removed comments of videos that contain kids in it to prevent pedophilia.

Music Comment Removal

  • In late 2020, YouTube deactivated comments under auto-generated art tracks, though they later reverted that decision in 2023.[42][43][44]

Video Dislikes

See Removal of public video dislikes (December 2021).

Stories

“Stories” was a feature of YouTube[45][46][47][48] which works in a similar way to , and . It can exclusively be accessed from the YouTube application for mobile devices (not YouTube's mobile website or desktop website). It allows users, momentarily exclusively with channels that have over 10000 subscribers, to publish vertical videos, usually filmed from their mobile devices, and optionally edited with e.g. text, using their basic editing features.

Creation of new Stories was discontinued in favor of Community Posts and Shorts on 2023-06-26. All remaining Stories expired within 7 days[49].

Volatility

“Story” videos automatically got deleted in 7 days (= 1 week) after being posted.

Accessibility

“Stories” could be accessed from the “Stories” tab on YouTube channels when viewed in the mobile application.

Removed or blocked channels

Trivia

  • All ytimg servers (where YouTube saved images, stylesheets and the .swf file of the flash-based YouTube video player) used to be intoxicated by robots.txt (“user-agent:* disallow:/ noindex:/”). When browsing YouTube layouts starting circa 2008 using the Wayback Machine, the website could only be viewed as black text on white background, just with different sizes (e.g. <h1> video title), due to the missing images and stylesheet information. Only information visible in the HTML source code of the page itself could be rendered. On 2012-02-29, the robots.txt file vanished off the ytimg servers, lifting the restrictions and making YouTube more properly browse-able through the Wayback Machine.
  • Yahoo and Bing Video results might contain titles, upload dates and thumbnails of unavailable YouTube videos (i.e. after privated, deleted or channel terminated). Please help saving them to Archive.Today and ArchiveBot if found, before they become purged from the results.

Finding Archived Videos

The YouTube Video Finder can be used to search several different YouTube archives for individual YouTube video IDs.

References

  1. YCombinator user sen, June 24, 2021, on "Older unlisted videos will be made private unless you opt out"
  2. https://youtube.googleblog.com/2008/06/new-beta-feature-video-annotations.html
  3. https://old.reddit.com/r/DataHoarder/comments/aa6czg/youtube_annotation_archive/[IAWcite.todayMemWeb]
  4. https://old.reddit.com/r/DataHoarder/comments/al7exa/youtube_annotation_archive_update_and_preview/[IAWcite.todayMemWeb]
  5. https://old.reddit.com/r/DataHoarder/comments/b7imx9/youtube_annotation_archive_annotation_data_from/[IAWcite.todayMemWeb]
  6. https://support.google.com/youtube/answer/6083270[IAWcite.todayMemWeb]
  7. https://web.archive.org/web/20200926110942/https://support.google.com/youtube/answer/6083634
  8. https://blog.youtube/news-and-events/update-youtube-unlisted-links/
  9. https://support.google.com/youtube/answer/9230970
  10. https://support.google.com/youtube/contact/older_unlisted_update
  11. 11.0 11.1 YouTube Help article: “Engage with creators on Community posts” mentions Note: As creators get the Community tab, it will replace the Discussion tab. You can access or delete any comments you left on the Discussion tab for 30 days after creators receive the Community tab. Follow the instructions below.”[IAWcite.todayMemWeb]
  12. Learn about Community posts - YouTube Help[IAWcite.todayMemWeb]
  13. Sample archive of mobile web discussion tab on 2020-07-15[IAWcite.todayMemWeb]
  14. https://support.google.com/youtube/thread/125117248/expanding-community-posts-to-channels-with-500-subscribers-and-removing-the-discussion-tab[IAWcite.todayMemWeb]
  15. https://support.google.com/youtube/thread/134791097/update-to-youtube-dislike-counts?hl=en[IAWcite.todayMemWeb]
  16. Used and Free Paired Space on https://archive.org/~tracey/stats
  17. 17.0 17.1 Article “Youtube Adds AutoSharing – YouTwitFace Is Now Real” by TubularInsights.com, formerly “ReelSEO”, written by Mark R Robertson.[IAWcite.todayMemWeb]
  18. Twitter tweet object metadata documentation.[IAWcite.todayMemWeb]
  19. “YouTube is apparently removing the ability to auto-share videos to Twitter” – influencerupdate.biz[IAWcite.todayMemWeb]
  20. YouTube to Remove Automatic Sharing to Twitter – Here is How to Fix It” article by TechWiser.com[IAWcite.todayMemWeb]
  21. Tweet by @TeamYouTube: “After Jan 31st, we're saying goodbye 👋 to automated tweets like the one below. You can still share your YouTube activity with your followers in more customized posts via the Share button. Full update →[IAWcite.todayMemWeb] https://goo.gl/ef8Vc3”
  22. 22.0 22.1 22.2 22.3 About YouTube's August 2014 layout that is still optionally accessible as of May 2019[IAWcite.todayMemWeb]
  23. What public YouTube video statistics looked like in 2008[IAWcite.todayMemWeb], 2010[IAWcite.todayMemWeb] and shortly before discontinued[IAWcite.todayMemWeb].
  24. Tweet by @Omarroth1: “YouTube actually removed any publicly available analytics maybe a couple months ago (the "analytics" tab that used to be available in the "report" and "transcription" tab. The endpoint is also no longer available. I'll see if I can write something 1/2”[IAWcite.todayMemWeb] “Unfortunately the internal endpoint has been completely removed. It used to be `/insights_ajax`, now returns 404.”[IAWcite.todayMemWeb]
  25. YouTube legacy video statistics (removed feature) as of June 2010[IAWcite.todayMemWeb]
  26. 26.0 26.1 YouTube “One Channel” layout propaganda. 📅 Uploaded on 2013-03-08 01:06:18 UTC, 👁 309346 views, 👍 1125 likes, 👎 8648 dislikes, 💬4139 comments, mostly negative comments.[IAWcite.todayMemWeb]
  27. YouTube Help article: “Customize channel layout”[IAWcite.todayMemWeb]
  28. Channel with “customization” turned off, found via [1]. Original URL: https://www.youtube.com/channel/UCUQe-1KNzrCyJG9Lld07DZQ[IAWcite.todayMemWeb]
  29. YouTube API: channel statistics[IAWcite.todayMemWeb].
  30. YouTube 2012 watch page screenshot[IAWcite.todayMemWeb]
  31. 31.0 31.1 (Couldn't find a screenshot of this former feature in action (quickly see recent videos of uploader without needing to leave watch page, by clicking on video counter). Whoever finds a screenshot of the feature in action should please add it to this reference.)
  32. PSY's “Gangnam Style” hits 2 Billion views. YouTube UI below video as of 2013-2014.[IAWcite.todayMemWeb]
  33. How far back in time can I look up stats? (FAQ)[IAWcite.todayMemWeb]
  34. Watch page on July 8, 2013; July 10, 2013
  35. TechCrunch article from 2016-02-24: “Facebook Enhances Everyone’s Like With Love, Haha, Wow, Sad, Angry Buttons”.[IAWcite.todayMemWeb]
  36. AdWeek article from 2011-08-05: “YouTube Trades In LOL, OMG, WTF Buttons For Drop Down Reactions Menu”.[IAWcite.todayMemWeb]
  37. MarketingHits.com article “YouTube Testing New “Reaction” Buttons: OMG, Epic, LOL, Fail, WTF, & Cute”,[IAWcite.todayMemWeb] originally from ReelSEO[IAWcite.todayMemWeb].
  38. 20110602 article by “Google Operating System” (unofficial blog): “YouTube Reactions”.[IAWcite.todayMemWeb]
  39. Video: “How To Use YouTube's New Post Bulletin Feature”[IAWcite.todayMemWeb]
  40. https://support.google.com/youtube/answer/2797468[IAWcite.todayMemWeb]
  41. {{URL|https://web.archive.org/web/20210920210730/https://webcache.googleusercontent.com/search?q=cache%3ATKqcRDlg7U8J%3Ahttps%3A%2F%2Fwww.youtube.com%2Fattribution%3Fv%3DsYv6XRqRwDg+&cd=15&hl=en&ct=clnk&gl=us
  42. Auto-generated Art Tracks and YouTube Comments - 2020-12-15 (also see its description)
  43. Learn why comments are disabled - YouTube help
  44. YouTube mass disables all comments on “Art Track” music videos, angering communities - 2020-12-18
  45. 201811291200 TheVerge article “YouTube is rolling out its Instagram-like Stories feature to more creators”[IAWcite.todayMemWeb]
  46. YouTube Help article “Watch YouTube Stories”[IAWcite.todayMemWeb]
  47. YouTube Help article: “YouTube Stories for creators”[IAWcite.todayMemWeb]
  48. YouTube Creator Academy: Lessons: Stories.[IAWcite.todayMemWeb]
  49. YouTube Stories are Going Away on 6/26/2023[IAWcite.todayMemWeb]

See also

External links

v · t · e         YouTube

GLAM · Governments · Local TV News · VHS