Difference between revisions of "Ispygames"

From Archiveteam
Jump to navigation Jump to search
Line 50: Line 50:
  
 
== IGN domains ==
 
== IGN domains ==
* beacon.snowball.com  
+
* http://beacon.snowball.com - already closed
* cars.ign.com
+
* http://codesmedia.ign.com  
* codesmedia.ign.com  
+
* http://dcmedia.ign.com  
* dcmedia.ign.com  
+
* http://entertainmentmedia.ign.com  
* entertainmentmedia.ign.com  
+
* http://faqsmedia.ign.com  
* faqsmedia.ign.com  
+
* http://faqsmovies.ign.com  
* faqsmovies.ign.com  
+
* http://ffmedia.ign.com  
* ffmedia.ign.com
 
* guides.ign.com  
 
 
* http://911.ign.com
 
* http://911.ign.com
 
* http://aavault.ign.com
 
* http://aavault.ign.com
Line 76: Line 74:
 
* http://aivlev.dev.m.uk.ign.com
 
* http://aivlev.dev.m.uk.ign.com
 
* http://alli.ign.com
 
* http://alli.ign.com
* http://answers.ign.com
 
 
* http://antis.ign.com
 
* http://antis.ign.com
 
* http://aovault.ign.com
 
* http://aovault.ign.com
Line 133: Line 130:
 
* http://faqsmovies.ign.com
 
* http://faqsmovies.ign.com
 
* http://feeds.ign.com
 
* http://feeds.ign.com
* http://ffmovies.ign.com - redirects to ffmedia.ign.com
 
 
* http://ffvault.ign.com
 
* http://ffvault.ign.com
 
* http://findit.ign.com
 
* http://findit.ign.com
Line 305: Line 301:
 
* http://vault.ign.com
 
* http://vault.ign.com
 
* http://vaultmedia.ign.com
 
* http://vaultmedia.ign.com
* http://ve3dboards.ign.com - redirects to http://www.ign.com/boards/
 
 
* http://ve3d.ign.com - grabbing
 
* http://ve3d.ign.com - grabbing
 
* http://vgu.stg.www.ign.com
 
* http://vgu.stg.www.ign.com
Line 332: Line 327:
 
* http://xboxmedia.ign.com
 
* http://xboxmedia.ign.com
 
* http://xboxmovies.ign.com
 
* http://xboxmovies.ign.com
* insdermedia.ign.com  
+
* http://insdermedia.ign.com  
* mac.ign.com  
+
* http://mac.ign.com  
* media.ign.com  
+
* http://media.ign.com  
* memoviedia.ign.com  
+
* http://memoviedia.ign.com  
* mobile.ign.com  
+
* http://mobile.ign.com  
* pcmedia.ign.com  
+
* http://pcmedia.ign.com  
* pocketmedia.ign.com  
+
* http://pocketmedia.ign.com  
* ps2media.ign.com  
+
* http://ps2media.ign.com  
* psxmedia.ign.com  
+
* http://psxmedia.ign.com  
* psxmovies.ign.com  
+
* http://psxmovies.ign.com  
* scifimedia.ign.com  
+
* http://scifimedia.ign.com  
* share.affiliation.com  
+
* http://share.affiliation.com  
* vaultmedia.ign.com  
+
* http://vaultmedia.ign.com  
* wiremedia.ign.com  
+
* http://wiremedia.ign.com  
* wiremovies.ign.com  
+
* http://wiremovies.ign.com  
* www.championshipgamingseries.com  
+
* http://www.championshipgamingseries.com  
* xboxlive.ign.com
+
* http://xboxlive.ign.com
 +
 
 +
=== Redirects ===
 +
 
 +
* http://cars.ign.com -> http://www.ign.com/
 +
* http://guides.ign.com -> http://www.ign.com/wikis
 +
* http://answers.ign.com -> http://www.ign.com/boards/
 +
* http://ve3dboards.ign.com -> http://www.ign.com/boards/
 +
* http://ffmovies.ign.com -> http://ffmedia.ign.com
  
 
== Gamespy Domains ==
 
== Gamespy Domains ==

Revision as of 19:28, 24 March 2013

The News

IGN hit with layoffs, 1UP, UGO and GameSpy shutting down
1UP, UGO and GameSpy to be shut down

The Problems

  • Once you start digging around these sites you find it to be a mess of inconsistent url schemes and content everywhere.
  • Some files are being hosted on MediaFire.
  • Based on tests the larger and older a site is the more that is missed by a wget crawl due to the url scheme.

What we know

  • We already have a list of almost all the domains involved
  • A clean list with dups and bad domains is already being process and will be posted here when complete.
  • Most of the sites are not that big, but a few are huge.

The plan

  • Save the sites and related content
  • Backup the twitter feeds for any associated accounts. All my tweets just takes a username and returns the max tweets possible.


wget test command

This if for the gamespy sites.

USER_AGENT="Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)"
SAVE_HOST="http://planetdoom.gamespy.com"
WARC_NAME="warc_name"

wget -e robots=off --mirror --page-requisites \ 
--waitretry 5 --timeout 60 --tries 5 --wait 2 \
--warc-header "operator: Archive Team" --warc-cdx --warc-file="$SAVE_HOST" \
-U "$USER_AGENT" "$SAVE_HOST" \
--span-hosts --domains=$SAVE_HOST,pcmedia.gamespy.com,pnmedia.gamespy.com,pspmedia.gamespy.com,oystatic.ignimgs.com

Try this for the ign, ugo sites.

USER_AGENT="Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)"
SAVE_HOST="http://ve3d.ign.com"
WARC_NAME="warc_name"

wget -e robots=off --mirror --page-requisites \ 
--waitretry 5 --timeout 60 --tries 5 --wait 2 \
--warc-header "operator: Archive Team" --warc-cdx --warc-file="$SAVE_HOST" \
-U "$USER_AGENT" "$SAVE_HOST"

IGN domains

Redirects

Gamespy Domains