Difference between revisions of "Google Reader"

From Archiveteam
Jump to navigation Jump to search
(it's ready; non-Warrior instructions)
(add youtube; low priority)
Line 174: Line 174:
** Popular: http://del.icio.us/rss/popular
** Popular: http://del.icio.us/rss/popular
** Popular tags: http://del.icio.us/rss/popular/TAGNAME
** Popular tags: http://del.icio.us/rss/popular/TAGNAME
* http://youtube.com/user/*
** http://www.youtube.com/rss/user/USERNAME/videos.rss (old feed)
** http://gdata.youtube.com/feeds/api/users/USERNAME/uploads
** https://gdata.youtube.com/feeds/api/users/USERNAME/uploads
** http://gdata.youtube.com/feeds/api/users/USERNAME/uploads?max-results=50
** http://gdata.youtube.com/feeds/api/users/USERNAME/uploads?alt=rss&max-results=50
** http://gdata.youtube.com/feeds/base/users/USERNAME/uploads?alt=rss&v=2&client=ytapi-youtube-profile
** http://gdata.youtube.com/feeds/base/users/USERNAME/uploads?alt=rss&v=2&orderby=published&client=ytapi-youtube-profile
** http://gdata.youtube.com/feeds/base/users/USERNAME/uploads?alt=rss&client=ytapi-youtube-rss-redirect&v=2&orderby=updated (redirect from old feed)
* ... and many more (please add them above!)
* ... and many more (please add them above!)
** http://taimoorsultan.com/list-of-25-blogging-platforms/
** http://taimoorsultan.com/list-of-25-blogging-platforms/

Revision as of 22:48, 30 May 2013

Google Reader
URL http://www.google.com/reader/[IAWcite.todayMemWeb]
Status Online!
Archiving status
Archiving type Unknown
Project source https://github.com/ArchiveTeam/greader-grab
Project tracker N/A
IRC channel #donereading (on hackint)

Shutdown notification

On the March 13, Google announced that they'll "spring clean" Google Reader at Official Google Reader Blog:

we will soon retire Google Reader (the actual date is July 1, 2013)

Backing up your own data

Backing up the historical feed data

Google Reader acts as a cache for RSS/Atom feed content, keeping deleted posts and deleted blogs accessible (if you can recreate the RSS/Atom feed URL). After the Reader shutdown, this data might still be available[1] via the Feeds API, but we'd like to grab most of this data before July 1 through the much more straightforward /reader/ API.

Your help is needed

Give us your feed URLs

We need to discover as many feed URLs as possible. Not all of them can be discovered through crawling, so we need your OPML files. (Though if you have any private or passworded feeds, please strip them out.)

Upload OPML files and lists of URLs to:

http://allyourfeed.ludios.org:8080/

Install the ArchiveTeam Warrior, or run the pipeline on your Linux machine

Install the ArchiveTeam Warrior and have it run ArchiveTeam's Choice:

http://www.archiveteam.org/index.php?title=ArchiveTeam_Warrior

Google Reader will (probably) soon become the primary job.

If you cannot use the Warrior, follow the instructions on https://github.com/ArchiveTeam/greader-grab

Crawl websites to discover blogs and usernames

We need to discover millions of blog/username URLs on popular blogging platforms (which we'll turn into feed URLs).

Join #donereading and #archiveteam on efnet if you'd like to help with this.

Tools for URL discovery

git clone https://github.com/trivio/common_crawl_index
cd common_crawl_index
pip install --user boto
PYTHONPATH=. python bin/index_lookup_remote 'com.blogspot'

Crawl Google Reader itself for feeds

https://www.google.com/reader/directory/search?q=keyword-here

https://www.google.com/reader/directory/search?q=keyword-here&start=10

Add gzip support to wget-lua

It would be quite helpful to have a wget-lua that supports gzip content encoding (vanilla wget doesn't support it either.) This will speed up downloads and save a lot of bandwidth.

There have already been some attempts at making wget support gzip:

https://github.com/kravietz/wget-gzip (Windows-only; needs to work on Linux)

https://github.com/ptolts/wget-with-gzip-compression (based on a wget from 2003?)