"Here at DNS History we have been crawling DNS records since 2009, our database currently contains over 500 million domains and over 2 billion DNS records."
|URL||• • •|
|Archiving status||In progress...|
|IRC channel||(on hackint)|
It was scheduled to close on July 10th, 2016, due to "funding issues".
According to a Twitter entry dated 2016-06-26:
- "dnshistory.org/ will be closing down in a couple of weeks - it has been a fun project over the years thanks folks."
Also, on the main page, on the right:
- "This site will shut down on the 10th July 2016 due to funding issues."
A scrape of all TLDs which are crawled by DNS History currently is available here.
Out of the 1365 TLDs,
- 1114 do have pages
- 1110 are here
- .info: 251346 pages
- .org: 288405 pages
- .net: 422569 pages
- .com: unknown (expected to be millions of pages)
- 251 don't have a single domain: they
After ArchiveTeam started the project (2016-07-04), dnshistory.org staff shortly activated CloudFlare in order to block our access to the site.
user:bzc6p contacted them asking for cooperation, but they replied that "there is no way you can archive the >billion pages on the site and trying to do so is causing issues for other users on the site."
Since then CloudFlare seems to have been deactivated, but the site was broken for a month, giving 500 errors. However, as of 2016-08-17, the site is up, with various queries, including search, still functioning, with the closure notice still up.
On 2016-08-20, ArchiveTeam restarted its efforts. On 08-21, although the site seemed to deal with the load, and the site has officially been closed for 42 days, they started to ban our crawlers again.
How can I help?
Running a Warrior
You can start up a Warrior and there select DNS History. (If you don't really care what you are archiving, select ArchiveTeam's Choice instead, as at some points ArchiveTeam may priorize another project.)
Running the script manually
If you use Linux and you're a bit familiar with it, you can try running the script directly.
The instructions can be found at github.com/ArchiveTeam/dnshistory-grab.
|Some additional information|
|Don't forget to replace YOURNICKHERE with your nickname.
The number after
If you want to stop the script, please do it gracefully if possible. To do so, create an empty file named STOP in the folder of the script (terminal command:
If you see "Project code is out of date", kill the script, go to its folder (
Donating to the Internet Archive
Content downloaded by the ArchiveTeam will be uploaded to the Internet Archive, where it will be stored and be available – hopefully – forever. However, storing it costs thousands of dollars in the long run. So, if you can afford, please consider donating to the Internet Archive, so that this piece of history can be kept for us all. http://archive.org/donate
Do you like our cause?
If you want to help in other projects, want to learn more about ArchiveTeam, or even help in development in general, navigate to the Main Page of this wiki, from there you can reach a lot of information. The Team consists of volunteers working on the projects in their free time, so helping hands (and resources) are always welcome.