From Archiveteam
Revision as of 06:52, 29 November 2015 by Start (talk | contribs) (→‎Public HTTP/FTP Server List)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

I like preserving the web.

I also go by Start+Select and Pressstart.


Website Crawls

Public HTTP/FTP Server List

Searching intitle:"index of /" inurl:"ftp" on Google gives millions of results.

blah blah blah ignore


  • TODO: Scrape Google
  • TODO: Scrape Bing
  • TODO: Scrape DuckDuckGo
  • TODO: Scrape Twitter
  • TODO: Scrape Reddit
  • TODO: Scrape links from MediaWiki wikis
  • TODO: Scrape the Open Directory Project
  • TODO: Scrape the Common Crawl Index
  • TODO: Scrape the Wayback Machine
  • TODO: Scrape URLTeam dumps
  • TODO: Scrape a list of subdomains from (if applicable)