Difference between revisions of "Reddit"

From Archiveteam
Jump to navigation Jump to search
m
m (Add project tracker and source code)
(33 intermediate revisions by 10 users not shown)
Line 1: Line 1:
{{Infobox project
{{Infobox project
| title = reddit
| title = reddit
| image = Reddit logo.png
| logo = Reddit logo.png
| image = Reddit home page 2013-03-26.png
| image = Reddit home page - 2019-12-14.png
| description = reddit home page as seen on March 26, 2013
| description = reddit home page as seen on December 14, 2019
| URL = http://www.reddit.com/
| URL = https://www.reddit.com/<br />https://old.reddit.com/<br />https://i.reddit.com/
| project_status = {{online}}
| project_status = {{online}}
| archiving_status = {{Partiallysaved}}
| archiving_status = {{Partiallysaved}}
| irc = deaddit
| source = [https://github.com/ArchiveTeam/reddit-grab reddit-grab]
| tracker = [https://tracker.archiveteam.org/reddit/ reddit]
| irc = shreddit
| irc_network = hackint
| source = [https://github.com/ArchiveTeam/reddit-grab reddit-grab]
}}
}}


'''reddit''' is a content aggregator and social bookmarking service similar to the likes of Digg. Users can submit links, submit text posts, vote and comment on submissions in communities called "subreddits". It received considerable attention from its twelve hour SOPA blackout early in January of 2012.
'''Reddit''' is a content aggregator and social bookmarking service similar to the likes of Digg. Users can submit links, text posts, images and videos, vote and comment on submissions in communities called "subreddits". It received considerable attention from its twelve-hour SOPA blackout early in January 2012.
 
Reddit "quarantines" some controversial subreddits. Many of such quarantine subreddits have been deleted, and to date no quarantined subreddit has ever emerged unscathed, so it is important to make backups of them. [https://www.reddit.com/r/thequarantinelist/ Here is a list of quarantined reddits.]
 
It contains some subredits devoted to similar goals as [https://www.reddit.com/r/ArchiveTeam ArchiveTeam], including [https://www.reddit.com/r/AbandonedWebsites /r/AbandonedWebsites], [https://www.reddit.com/r/ForgottenWebsites /r/ForgottenWebsites], & [https://www.reddit.com/r/DataHoarder /r/DataHoarder], which are worth checking for material to be added to [[ArchiveBot]] or otherwise benefit from the attention of the team.


== Vital signs ==
== Vital signs ==


* <s>Appears stable, though a small to medium size team is a concern.</s>
* <s>Appears stable, though a small to medium size team is a concern.</s>
* '''Update (6/10/15)''': the admins carried out bannings of several subreddits claiming they were harassing people, the most notable of which was /r/fatpeoplehate. This has instilled some fear, uncertainty, and doubt in some part of the userbase, with a few claiming that reddit will soon become what Digg is now: nearly dead.
* 2015-10-06: The admins carried out bannings of several subreddits claiming they were harassing people, the most notable of which was /r/fatpeoplehate. This has instilled some fear, uncertainty, and doubt in some part of the userbase, with a few claiming that reddit will soon become what Digg is now: nearly dead.
* ''<s>Extremely endangered</s> - many subreddits were picketing after the firing of a reddit employee named Victoria by turning themselves private or restricting submissions.''
* ''<s>Extremely endangered</s> - many subreddits were picketing after the firing of a reddit employee named Victoria by turning themselves private or restricting submissions.''
* ''''Caution'''' - Reddit seems to have calmed down and returned to normal functionality after Ellen Pao's firing, and the Reddit team is making serious reforms (reducing shadowbanning, more mod tools). However, the revolt left unresolved issues and sour grapes within the community, and it seems Reddit was only saved by the lack of a practical alternative (Voat.co was crushed and went offline due to floods of refugees). '''It would be wise to preemptively archive the site''' before another crisis occurs.
* ''''Caution'''' - Reddit seems to have calmed down and returned to normal functionality after Ellen Pao's firing, and the Reddit team is making serious reforms (reducing shadowbanning, more mod tools). However, the revolt left unresolved issues and sour grapes within the community, and it seems Reddit was only saved by the lack of a practical alternative (Voat.co was crushed and went offline due to floods of refugees). '''It would be wise to preemptively archive the site''' before another crisis occurs.
* On July 3rd, 2015, Reddit user Stuck_in_the_Matrix '''completed his 14-month effort to archive Reddit's entire publicly available dataset''', just in time before the onset of the Reddit revolt. [https://archive.org/details/2015_reddit_comments_corpus It has been uploaded to the Internet Archive] in it's entirety.
* On July 3rd, 2015, Jason Baumgartner '''completed his 14-month effort to archive Reddit's entire publicly available textual content''', just in time before the onset of the Reddit revolt. The archive is still updated monthly. '''[http://files.pushshift.io/reddit/ The files are available here.]''' However, images and videos hosted by Reddit are not archived.
* '''As of November 9, 2015, it is stable once again.'''
* In 2017-2018, Reddit has carried out bannings of several subreddits including r/incels and r/maleforeveralone, which had tens of thousands of subscribers each. Other subreddits including r/Braincels, r/foreveralone, r/TheRedPill and r/MGTOW are endangered. Discussions and [https://www.thepetitionsite.com/takeaction/308/200/042/?TAP=1007&cid=causes_petition_postinfo petitions] about banning those subreddits are currently taking place.[https://babe.net/2018/03/07/incel-40474][https://www.reddit.com/r/IncelTears/comments/83irsc/why_isnt_rbraincels_banned_yet/]
* In 2018, a new, redesigned website became the default version of Reddit. This redesigned version has numerous usability issues. It heavily relies on JS and is essentially uncrawlable without dedicated code. The pre-redesign version of Reddit continues to be available at [https://old.reddit.com/ old.reddit.com].
* In March 2019, /r/watchpeopledie, /r/Gore, and some other subs were banned after the Christchurch shooting – this was clearly not due to the video recording of that shooting getting shared (that was forbidden on WPD at least) but due to the negative press coverage, just like for previous bans.
* Also in March 2019, /r/Piracy got threatened by Reddit's legal team with a ban due to the mods allegedly doing too little against copyright infringement.<ref>{{URL|https://old.reddit.com/r/Piracy/comments/b28d9q/rpiracy_has_received_a_notice_of_multiple/}}</ref>
* Reddit has quarantined manosphere subreddits including /r/Braincels, /r/TheRedPill. /r/Braincels was banned on October 30, 2019.
* Users began to spot in December 2019 that comment threads, at least on the "new" version of the site, were being locked behind a registration wall in an apparent A/B test.<ref>{{URL|https://news.ycombinator.com/item?id=21780092}}</ref><ref>{{URL|https://old.reddit.com/r/mobileweb/comments/e7yivg/join_reddit_to_keep_reading_an_account_is_now/?sort=top}}</ref>


== October 2007 - May 2015 (Nearly) Complete Reddit Archive ==
== Textual Archive (Without Images or Videos) ==


On July 3rd, 2015, Reddit user Stuck_in_the_Matrix '''completed his 14-month effort to archive Reddit's entire publicly available dataset''', just in time before the onset of the Reddit revolt. [https://archive.org/details/2015_reddit_comments_corpus It has been uploaded to the Internet Archive] in it's entirety.
On July 3rd, 2015, Jason Baumgartner completed his 14-month effort to archive Reddit's entire publicly available textual content, just in time before the onset of the Reddit revolt. The archive is still being updated monthly. '''[http://files.pushshift.io/reddit/ The files are available here.]'''


* Reddit JSON API output
* Does not include images and videos hosted by Reddit
* ~1.7 billion comments
* Reddit JSON API output. Posts are archived incrementally in real-time.
* ~300,000 comments not accessible due to private subreddits or comment deletion or other API issues
* Some comments not accessible due to private subreddits or comment deletion or other API issues
* 250GB compressed (bzip2)
* [https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment/ Reddit /r/datasets - I have every publicly available Reddit comment for research. ~ 1.7 billion comments @ 250 GB compressed. Any interest in this?]
* [https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment/ Reddit /r/datasets - I have every publicly available Reddit comment for research. ~ 1.7 billion comments @ 250 GB compressed. Any interest in this?]
* [https://archive.org/details/2015_reddit_comments_corpus Internet Archive - 2015 Reddit Comments Corpus]
* [https://www.reddit.com/r/bigquery/comments/3cej2b/17_billion_reddit_comments_loaded_on_bigquery/ Google BigQuery Analysis of Reddit]
* [https://www.reddit.com/r/bigquery/comments/3cej2b/17_billion_reddit_comments_loaded_on_bigquery/ Google BigQuery Analysis of Reddit]


The scripts used to generate this API dump were not made public, but it likely used PRAW, and it would probably be better to rewrite from scratch.
The scripts used to generate this API dump were not made public, but it likely used PRAW, and it would probably be better to rewrite from scratch.


Also, this only preserves comments. All sidebar, wiki, and live thread data was not retrieved, so it should be scraped in an expansion pack. It is unknown if Thread OPs fall under the umbrella of "comment", but they probably do.
Also, this only preserves textual submissions and comments. All images and videos hosted on Reddit are not archived. All sidebar, wiki, and live thread data are not retrieved, so these should be scraped in an expansion pack.
 
===API===
Jason Baumgartner also provides an API for accessing Reddit's textual archive available [https://github.com/pushshift/api here]. The archive is updated in real-time. This API does not have the limitations of Reddit's API. For example, it does not impose limits on the number of submissions or comments that are retrieved.
 
To search for submissions of a subreddit (500 limit):
 
https://api.pushshift.io/reddit/search/submission/?subreddit=Archiveteam&size=500
 
To retrieve ''all'' comments for a submission (with tens of thousands of comments):
 
https://api.pushshift.io/reddit/submission/comment_ids/6uey5x
 
Note that posts are archived in real-time after they are created. Newer versions of edited posts are not archived. One may have to re-fetch the content on Reddit's site to get the latest revision of an edited post.
 
Also, one may also have to fetch the images and videos as they are not archived by the API.


== Data liberation ==
== Data liberation ==
Line 41: Line 67:
As of March 26, 2013, users can only see up to 1,000 posts and comments on a profile page. However, it was stated by admin "spladug" [http://www.reddit.com/r/ideasfortheadmins/comments/10tai6/ever_wondered_the_data_liberation_policy_of_reddit/c6gicdf that older comments and posts are still in the database]. spladug also states that the team is in favor for retrieving dumps of a user's data, but that the task would be taxing on the servers. <s>Since this comment was posted, there appears to have been no progress on a dump system.</s> Archiving would be nearly impossible using the old-fashioned way (without wget) if things do wind up FUBAR in the future because of this limitation.
As of March 26, 2013, users can only see up to 1,000 posts and comments on a profile page. However, it was stated by admin "spladug" [http://www.reddit.com/r/ideasfortheadmins/comments/10tai6/ever_wondered_the_data_liberation_policy_of_reddit/c6gicdf that older comments and posts are still in the database]. spladug also states that the team is in favor for retrieving dumps of a user's data, but that the task would be taxing on the servers. <s>Since this comment was posted, there appears to have been no progress on a dump system.</s> Archiving would be nearly impossible using the old-fashioned way (without wget) if things do wind up FUBAR in the future because of this limitation.


Instead, any archival methods should scrape from the Reddit API (which would have to run over several months). The API provides all nested comments that are not noticed by HTML. In addition, it significantly reduces server load.  
Instead, any archival methods should scrape from the Reddit API (which would have to run over several months). The API provides all nested comments that are not noticed by HTML. In addition, it significantly reduces server load.
 
Because of EU GDPR, progress was forcibly made to be compliant and the site [https://www.reddit.com/settings/data-request now has a request form]. Users can specify that they want a copy of all of their data, or data from specific date ranges. The site says requests may take up to 30 days to be processed.
 
== Gallery ==
 
<gallery>
File:Reddit home page 2013-03-26.png|reddit home page as seen on March 26, 2013, using the "old" version of the site still available today when logged in or through old.reddit.com
</gallery>
 
== Lists ==
 
* [[List of Reddit subs by country and territory]]
* [[List of Reddit subs by language]]
 
== Potentially endangered subreddits ==
 
* https://old.reddit.com/r/WatchRedditDie/ | Anti-Reddit
* https://old.reddit.com/r/opendirectories/ | Piracy
* https://old.reddit.com/r/DeadorVegetable/ | Gore and death
* https://old.reddit.com/r/FiftyFifty/ | Gore and death
* https://old.reddit.com/r/Piracy/ | Piracy
 
== References ==
 
<references/>


<!--
<!--
Line 56: Line 107:


-->
-->
== External Links ==
* {{url|1=http://www.reddit.com|2=reddit}}


{{Navigation box}}
{{Navigation box}}

Revision as of 08:28, 26 July 2020

reddit
Reddit logo
reddit home page as seen on December 14, 2019
reddit home page as seen on December 14, 2019
URL https://www.reddit.com/
https://old.reddit.com/
https://i.reddit.com/
Status Online!
Archiving status Partially saved
Archiving type Unknown
Project source reddit-grab
Project tracker reddit
IRC channel #shreddit (on hackint)

Reddit is a content aggregator and social bookmarking service similar to the likes of Digg. Users can submit links, text posts, images and videos, vote and comment on submissions in communities called "subreddits". It received considerable attention from its twelve-hour SOPA blackout early in January 2012.

Reddit "quarantines" some controversial subreddits. Many of such quarantine subreddits have been deleted, and to date no quarantined subreddit has ever emerged unscathed, so it is important to make backups of them. Here is a list of quarantined reddits.

It contains some subredits devoted to similar goals as ArchiveTeam, including /r/AbandonedWebsites, /r/ForgottenWebsites, & /r/DataHoarder, which are worth checking for material to be added to ArchiveBot or otherwise benefit from the attention of the team.

Vital signs

  • Appears stable, though a small to medium size team is a concern.
  • 2015-10-06: The admins carried out bannings of several subreddits claiming they were harassing people, the most notable of which was /r/fatpeoplehate. This has instilled some fear, uncertainty, and doubt in some part of the userbase, with a few claiming that reddit will soon become what Digg is now: nearly dead.
  • Extremely endangered - many subreddits were picketing after the firing of a reddit employee named Victoria by turning themselves private or restricting submissions.
  • 'Caution' - Reddit seems to have calmed down and returned to normal functionality after Ellen Pao's firing, and the Reddit team is making serious reforms (reducing shadowbanning, more mod tools). However, the revolt left unresolved issues and sour grapes within the community, and it seems Reddit was only saved by the lack of a practical alternative (Voat.co was crushed and went offline due to floods of refugees). It would be wise to preemptively archive the site before another crisis occurs.
  • On July 3rd, 2015, Jason Baumgartner completed his 14-month effort to archive Reddit's entire publicly available textual content, just in time before the onset of the Reddit revolt. The archive is still updated monthly. The files are available here. However, images and videos hosted by Reddit are not archived.
  • In 2017-2018, Reddit has carried out bannings of several subreddits including r/incels and r/maleforeveralone, which had tens of thousands of subscribers each. Other subreddits including r/Braincels, r/foreveralone, r/TheRedPill and r/MGTOW are endangered. Discussions and petitions about banning those subreddits are currently taking place.[1][2]
  • In 2018, a new, redesigned website became the default version of Reddit. This redesigned version has numerous usability issues. It heavily relies on JS and is essentially uncrawlable without dedicated code. The pre-redesign version of Reddit continues to be available at old.reddit.com.
  • In March 2019, /r/watchpeopledie, /r/Gore, and some other subs were banned after the Christchurch shooting – this was clearly not due to the video recording of that shooting getting shared (that was forbidden on WPD at least) but due to the negative press coverage, just like for previous bans.
  • Also in March 2019, /r/Piracy got threatened by Reddit's legal team with a ban due to the mods allegedly doing too little against copyright infringement.[1]
  • Reddit has quarantined manosphere subreddits including /r/Braincels, /r/TheRedPill. /r/Braincels was banned on October 30, 2019.
  • Users began to spot in December 2019 that comment threads, at least on the "new" version of the site, were being locked behind a registration wall in an apparent A/B test.[2][3]

Textual Archive (Without Images or Videos)

On July 3rd, 2015, Jason Baumgartner completed his 14-month effort to archive Reddit's entire publicly available textual content, just in time before the onset of the Reddit revolt. The archive is still being updated monthly. The files are available here.

The scripts used to generate this API dump were not made public, but it likely used PRAW, and it would probably be better to rewrite from scratch.

Also, this only preserves textual submissions and comments. All images and videos hosted on Reddit are not archived. All sidebar, wiki, and live thread data are not retrieved, so these should be scraped in an expansion pack.

API

Jason Baumgartner also provides an API for accessing Reddit's textual archive available here. The archive is updated in real-time. This API does not have the limitations of Reddit's API. For example, it does not impose limits on the number of submissions or comments that are retrieved.

To search for submissions of a subreddit (500 limit):

https://api.pushshift.io/reddit/search/submission/?subreddit=Archiveteam&size=500

To retrieve all comments for a submission (with tens of thousands of comments):

https://api.pushshift.io/reddit/submission/comment_ids/6uey5x

Note that posts are archived in real-time after they are created. Newer versions of edited posts are not archived. One may have to re-fetch the content on Reddit's site to get the latest revision of an edited post.

Also, one may also have to fetch the images and videos as they are not archived by the API.

Data liberation

As of March 26, 2013, users can only see up to 1,000 posts and comments on a profile page. However, it was stated by admin "spladug" that older comments and posts are still in the database. spladug also states that the team is in favor for retrieving dumps of a user's data, but that the task would be taxing on the servers. Since this comment was posted, there appears to have been no progress on a dump system. Archiving would be nearly impossible using the old-fashioned way (without wget) if things do wind up FUBAR in the future because of this limitation.

Instead, any archival methods should scrape from the Reddit API (which would have to run over several months). The API provides all nested comments that are not noticed by HTML. In addition, it significantly reduces server load.

Because of EU GDPR, progress was forcibly made to be compliant and the site now has a request form. Users can specify that they want a copy of all of their data, or data from specific date ranges. The site says requests may take up to 30 days to be processed.

Gallery

Lists

Potentially endangered subreddits

References

  1. https://old.reddit.com/r/Piracy/comments/b28d9q/rpiracy_has_received_a_notice_of_multiple/[IAWcite.todayMemWeb]
  2. Example[IAWcite.todayMemWeb]URL not specified; if the URL contains an = please prefix it with 1= so it is not treated as a named template parameter
  3. Example[IAWcite.todayMemWeb]URL not specified; if the URL contains an = please prefix it with 1= so it is not treated as a named template parameter