Difference between revisions of "Ispygames"

From Archiveteam
Jump to navigation Jump to search
(All the check domains I found so far)
Line 32: Line 32:
--span-hosts --domains=$SAVE_HOST,pcmedia.gamespy.com,pnmedia.gamespy.com,pspmedia.gamespy.com,oystatic.ignimgs.com
--span-hosts --domains=$SAVE_HOST,pcmedia.gamespy.com,pnmedia.gamespy.com,pspmedia.gamespy.com,oystatic.ignimgs.com
</pre>
</pre>
== IGN domains ==
* beacon.snowball.com
* cars.ign.com
* codesmedia.ign.com
* cubemedia.ign.com
* dcmedia.ign.com
* entertainmentmedia.ign.com
* faqsmedia.ign.com
* faqsmovies.ign.com
* ffmedia.ign.com
* guides.ign.com
* http://911.ign.com
* http://aavault.ign.com
* http://ac2vault.ign.com
* http://actionunleashed.ign.com
* http://acvault.ign.com
* http://adtools.ign.com
* http://aevans.dev.m.au.ign.com
* http://aevans.dev.m.ca.ign.com
* http://aevans.dev.m.ie.ign.com
* http://aevans.dev.m.ign.com
* http://aevans.dev.m.uk.ign.com
* http://aion.ign.com
* http://aivlev.dev.m.au.ign.com
* http://aivlev.dev.m.ie.ign.com
* http://aivlev.dev.m.ign.com
* http://aivlev.dev.m.uk.ign.com
* http://alli.ign.com
* http://answers.ign.com
* http://antis.ign.com
* http://aovault.ign.com
* http://apassey.dev.m.ign.com
* http://atvault.ign.com
* http://au.bestof.ign.com
* http://au.media.ign.com
* http://au.microsites.ign.com
* http://au.retro.ign.com
* http://au.sports.ign.com
* http://au.top100.ign.com
* http://au.video.ign.com
* http://beacon.ign.com
* http://beaterator.ign.com
* http://bestofe3.ign.com
* http://bestof.ign.com
* http://bigworldvault.ign.com
* http://blockbuster.ign.com
* http://bombergirl.ign.com
* http://broadband.ign.com
* http://browserthemes.ign.com
* http://carsmedia.ign.com
* http://carsmovies.ign.com
* http://cdyi.dev.m.ign.com
* http://ces2009.ign.com
* http://c.ffmovies.ign.com
* http://championshipgamingseries.ign.com
* http://championsonline.ign.com
* http://code.ign.com
* http://codesmedia.ign.com
* http://cohvault.ign.com
* http://comiccon.ign.com
* http://corp.ign.com
* http://covvault.ign.com
* http://crose.dev.m.ca.ign.com
* http://crose.dev.m.ign.com
* http://crossassault.ign.com
* http://cubemedia.ign.com
* http://dcmedia.ign.com
* http://ddovault.ign.com
* http://design.ign.com
* http://dndvault.ign.com
* http://doa.ign.com
* http://download.ign.com
* http://downloads.ign.com
* http://dragonica.ign.com
* http://dsi.ign.com
* http://dsvault.ign.com
* http://emailpreferences.ign.com
* http://entertainmentmedia.ign.com
* http://eq2vault.ign.com
* http://eqvault.ign.com
* http://esports.ign.com
* http://evevault.ign.com
* http://faqsmedia.ign.com
* http://faqsmovies.ign.com
* http://feeds.ign.com
* http://ffmovies.ign.com
* http://ffvault.ign.com
* http://findit.ign.com
* http://flashpoint.ign.com
* http://formenmedia.ign.com
* http://gamechanger.ign.com
* http://gamesites.ign.com
* http://gamestore.ign.com
* http://gamingworldrecord.ign.com
* http://gbartone.dev.m.au.ign.com
* http://gbartone.dev.m.ca.ign.com
* http://gbartone.dev.m.ie.ign.com
* http://gbartone.dev.m.ign.com
* http://gbartone.dev.m.uk.ign.com
* http://grandtheftautohood.ign.com
* http://grandtheftauto.ign.com
* http://gtahood.ign.com
* http://gta.ign.com
* http://guidesarchive.ign.com
* http://guidesmedia.ign.com
* http://guitarhero3.ign.com
* http://gwvault.ign.com
* http://halo.ign.com
* http://hjvault.ign.com
* http://hls.gbartone.dev.m.ign.com
* http://horizonsvault.ign.com
* http://ie.bestof.ign.com
* http://ie.media.ign.com
* http://ie.top100.ign.com
* http://insiderdownloads.ign.com
* http://insidermedia.ign.com
* http://ipl.ign.com
* http://iplstore.ign.com
* http://jloijens.dev.m.ign.com
* http://jtai.dev.m.ign.com
* http://kaneandlynch.ign.com
* http://kjaniak.dev.m.ign.com
* http://l2vault.ign.com
* http://labs.ign.com
* http://lanoire.ign.com
* http://links.em.ign.com
* http://littlebigplanet.ign.com
* http://live.ign.com
* http://lotrovault.ign.com
* http://macmedia.ign.com
* http://macmovies.ign.com
* http://mag.ign.com
* http://mail2.ign.com
* http://m.au.ign.com
* http://m.ca.ign.com
* http://mcraft.ign.com
* http://media.ign.com
* http://mevault.ign.com
* http://m.ie.ign.com
* http://m.ign.com
* http://minecraft.ign.com
* http://mobile.ign.com
* http://moviemedia.ign.com
* http://moviesmovies.ign.com
* http://m.uk.ign.com
* http://musichub.ign.com
* http://mxovault.ign.com
* http://nchandra.dev.m.au.ign.com
* http://nchandra.dev.m.ie.ign.com
* http://nchandra.dev.m.ign.com
* http://nchandra.dev.m.uk.ign.com
* http://nchandra.dev.www.ign.com
* http://niboppub.ign.com
* http://nwvault.ign.com
* http://o.cubemedia.ign.com
* http://o.faqsmedia.ign.com
* http://o.faqsmovies.ign.com
* http://o.ffmovies.ign.com
* http://o.guidesarchive.ign.com
* http://o.guidesmedia.ign.com
* http://o.insidermedia.ign.com
* http://o.media.ign.com
* http://o.mobile.ign.com
* http://o.moviesmovies.ign.com
* http://o.pcmedia.ign.com
* http://open.em.ign.com
* http://o.pocketmedia.ign.com
* http://o.ps2media.ign.com
* http://opt-out.emailpreferences.ign.com
* http://o.rpgvaultarchive.ign.com
* http://overlord.ign.com
* http://o.xboxmedia.ign.com
* http://pawong.dev.www.ign.com
* http://pcmedia.ign.com
* http://planetelderscrolls.ign.com
* http://play.ign.com
* http://pocketmedia.ign.com
* http://podcast.ign.com
* http://podcasts.ign.com
* http://potbsvault.ign.com
* http://primeblog.ign.com
* http://promotions.ign.com
* http://promotools.ign.com
* http://ps2media.ign.com
* http://psxmedia.ign.com
* http://psxmovies.ign.com
* http://publish.ign.com
* http://rift.ign.com
* http://rmcadams.dev.m.ca.ign.com
* http://rmcadams.dev.m.ign.com
* http://rmcadams.dev.www.ign.com
* http://rotavault.ign.com
* http://rpgvaultarchive.ign.com
* http://rsullivan.dev.m.ca.ign.com
* http://rsullivan.dev.m.ign.com
* http://rsullivan.dev.www.ign.com
* http://ryzomvault.ign.com
* http://sbvault.ign.com
* http://scifimedia.ign.com
* http://s.faqsmovies.ign.com
* http://s.ffmovies.ign.com
* http://share.ign.com
* http://shootmania.ign.com
* http://s.insiderdownloads.ign.com
* http://skate2.ign.com
* http://smashbros.ign.com
* http://smcnabb.dev.www.ign.com
* http://s.moviesmovies.ign.com
* http://sovault.ign.com
* http://speedtv.ign.com
* http://s.psxmovies.ign.com
* http://sslvpn.ign.com
* http://staging-api.ign.com
* http://starcraft2.ign.com
* http://starcraft.ign.com
* http://store.ign.com
* http://strangleholdcentral.ign.com
* http://strategyvault.ign.com
* http://sts.ign.com
* http://studio.ign.com
* http://supersmashbros.ign.com
* http://swgvault.ign.com
* http://swvault.ign.com
* http://s.xboxmovies.ign.com
* http://tabularasavault.ign.com
* http://tdu.ign.com
* http://tford.dev.m.ign.com
* http://tford.dev.www.ign.com
* http://thhe.ign.com
* http://ticket.ign.com
* http://tickets.ign.com
* http://titanquestvault.ign.com
* http://tjohnson.dev.au.ign.com
* http://tjohnson.dev.ca.ign.com
* http://tjohnson.dev.ie.ign.com
* http://tjohnson.dev.uk.ign.com
* http://tjohnson.dev.www.ign.com
* http://top100.ign.com
* http://touch.ign.com
* http://tqvault.ign.com
* http://trvault.ign.com
* http://twoworldsvault.ign.com
* http://uk.bestof.ign.com
* http://uk.corp.ign.com
* http://uk.media.ign.com
* http://uk.retro.ign.com
* http://uk.sports.ign.com
* http://uk.top100.ign.com
* http://uk.video.ign.com
* http://uovault.ign.com
* http://v3-api.stg.ie.ign.com
* http://v3-api.stg.m.au.ign.com
* http://v3-api.stg.m.ie.ign.com
* http://v3-api.stg.www.ign.com
* http://vanguardvault.ign.com
* http://vault.ign.com
* http://vaultmedia.ign.com
* http://ve3dboards.ign.com
* http://ve3d.ign.com
* http://vgu.stg.www.ign.com
* http://viashoka.dev.m.ign.com
* http://viashoka.dev.www.ign.com
* http://videopatch1.atio.dev.uk.ign.com
* http://videoplayer.phantom.stg.www.ign.com
* http://video.stg.www.ign.com
* http://vnboards.ign.com
* http://warhammervault.ign.com
* http://wikihub.stg.www.ign.com
* http://wiki.stg.www.ign.com
* http://wiremedia.ign.com
* http://wiremovies.ign.com
* http://wishvault.ign.com
* http://witchervault.ign.com
* http://wowvault.ign.com
* http://www.antis.ign.com
* http://www.blockbuster.ign.com
* http://www.ipl.ign.com
* http://www.kaneandlynch.ign.com
* http://www.mevault.ign.com
* http://www.supersmashbros.ign.com
* http://xboxlivemedia.ign.com
* http://xboxlivemovies.ign.com
* http://xboxmedia.ign.com
* http://xboxmovies.ign.com
* insdermedia.ign.com
* mac.ign.com
* media.ign.com
* memoviedia.ign.com
* mobile.ign.com
* pcmedia.ign.com
* pocketmedia.ign.com
* ps2media.ign.com
* psxmedia.ign.com
* psxmovies.ign.com
* scifimedia.ign.com
* share.affiliation.com
* vaultmedia.ign.com
* wiremedia.ign.com
* wiremovies.ign.com
* www.championshipgamingseries.com
* xboxlive.ign.com
== Gamespy Domains ==
* http://planetcnc.gamespy.com
* http://planetthesims.gamespy.com
* http://norad.gamespy.com
* http://planetfrontlines.gamespy.com
* http://planetcivilization.gamespy.com
* http://planethalflife.gamespy.com
* http://bf2142portalservices.gamespy.com
* http://arena.gamespy.com
* http://planettransformers.gamespy.com
* http://www.gamespy.com
* http://planetcoh.gamespy.com
* http://planetbattlefield.gamespy.com
* http://planetcrysis.gamespy.com
* http://planetresidentevil.gamespy.com
* http://planetxmen.gamespy.com
* http://bugsubmit.gamespy.com
* http://planetquake.gamespy.com
* http://planetgrandtheftauto.gamespy.com
* http://planetthemovies.gamespy.com
* http://forumplanet.gamespy.com
* http://planetfarcry.gamespy.com
* http://planetelderscrolls.gamespy.com
* http://ps3.gamespy.com
* http://pc.gamespy.com
* http://planettonyhawk.gamespy.com
* http://planetunreal.gamespy.com
* http://planetmedalofhonor.gamespy.com
* http://xbox360.gamespy.com
* http://sslvpn.gamespy.com
* http://planetfallout.gamespy.com
* http://planetageofempires.gamespy.com
* http://planetgearsofwar.gamespy.com
* http://lanoirepc.d2gstore.gamespy.com
* http://planetcallofduty.gamespy.com
* http://ds.gamespy.com
* http://classicgaming.gamespy.com
* http://planetdoom.gamespy.com
* http://planetwwe.gamespy.com
* http://psp.gamespy.com
* http://forums.gamespy.com

Revision as of 12:57, 27 February 2013

The News

IGN hit with layoffs, 1UP, UGO and GameSpy shutting down
1UP, UGO and GameSpy to be shut down

The Problems

  • Once you start digging around these sites you find it to be a mess of inconsistent url schemes and content everywhere.
  • Some files are being hosted on MediaFire.
  • Based on tests the larger and older a site is the more that is missed by a wget crawl due to the url scheme.

What we know

  • We already have a list of almost all the domains involved
  • A clean list with dups and bad domains is already being process and will be posted here when complete.
  • Most of the sites are not that big, but a few are huge.

The plan

  • Save the sites and related content
  • Backup the twitter feeds for any associated accounts. All my tweets just takes a username and returns the max tweets possible.


wget test command

USER_AGENT="Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)"
SAVE_HOST="planetdoom.gamespy.com"

wget -e robots=off --mirror --page-requisites --waitretry 5 --timeout 60 \
--tries 10 --warc-header "operator: Archive Team" --warc-cdx \
"$SAVE_HOST" --warc-file="$SAVE_HOST" --wait 2 -U "$USER_AGENT" \
--span-hosts --domains=$SAVE_HOST,pcmedia.gamespy.com,pnmedia.gamespy.com,pspmedia.gamespy.com,oystatic.ignimgs.com

IGN domains


Gamespy Domains