Difference between revisions of "Talk:ESPN Forums"
Mithrandir (talk | contribs) |
Mithrandir (talk | contribs) (i accidentally a word) |
||
| Line 12: | Line 12: | ||
They do have some rather restrictive limits on how many requests "the customer" can make per minute and | They do have some rather restrictive limits on how many requests "the customer" can make per minute and | ||
"The rate limits are per API endpoint and per API key." --[[User:Mithrandir|Mithrandir]] 16:15, 4 July 2013 (EDT) | "The rate limits are per API endpoint and per API key." --[[User:Mithrandir|Mithrandir]] 16:15, 4 July 2013 (EDT) | ||
: Looks they've locked the forums, so it could go offline at anytime. That script is still buggy, but I've managed to get >100 forums so far. Here they are on [https://www.dropbox.com/sh/6i1o3hkhvfxo62r/UT2KG7EbrB Dropbox]. It's just tarred and [https://en.wikipedia.org/wiki/Lzip lzipped] directories of JSON files (each file being one thread.) They should be pretty complete, maybe 99% up-to-date or so. --[[User:Mithrandir|Mithrandir]] 19:56, 20 July 2013 (EDT) | : Looks like they've locked the forums, so it could go offline at anytime. That script is still buggy, but I've managed to get >100 forums so far with it. Here they are on [https://www.dropbox.com/sh/6i1o3hkhvfxo62r/UT2KG7EbrB Dropbox]. It's just tarred and [https://en.wikipedia.org/wiki/Lzip lzipped] directories of JSON files (each file being one thread.) They should be pretty complete, maybe 99% up-to-date or so. --[[User:Mithrandir|Mithrandir]] 19:56, 20 July 2013 (EDT) | ||
Revision as of 23:57, 20 July 2013
Backup Notes
Capturing the webpage with the usual tools won't work well, as loading comments requires Javascript. So the best thing to do is to download the back-end JSON (or use PhantomJS, but that's too slow IMO.)
The API URL for the comments/topics is
http://api.echoenabled.com/v1/search?q=...&appkey=dev.espn.go.com
and the documentation for the '...' part is here
I wrote a small Python script that loops through the topics list, downloads the title/description and comments for each topic, and stores it a file. It's a bit buggy right now though, so I won't post it just yet.
They do have some rather restrictive limits on how many requests "the customer" can make per minute and "The rate limits are per API endpoint and per API key." --Mithrandir 16:15, 4 July 2013 (EDT)
- Looks like they've locked the forums, so it could go offline at anytime. That script is still buggy, but I've managed to get >100 forums so far with it. Here they are on Dropbox. It's just tarred and lzipped directories of JSON files (each file being one thread.) They should be pretty complete, maybe 99% up-to-date or so. --Mithrandir 19:56, 20 July 2013 (EDT)