Difference between revisions of "DNS History"

From Archiveteam
Jump to navigation Jump to search
m (→‎Discovery results: .org result)
m (started)
Line 6: Line 6:
| description = "Here at DNS History we have been crawling DNS records since 2009, our database currently contains over 500 million domains and over 2 billion DNS records."
| description = "Here at DNS History we have been crawling DNS records since 2009, our database currently contains over 500 million domains and over 2 billion DNS records."
| project_status = {{closing}}
| project_status = {{closing}}
| archiving_status = {{upcoming}}
| archiving_status = {{inprogress}}
<!-- | tracker = [http://tracker.archiveteam.org/dnshistory dnshistory] -->
| tracker = [http://tracker.archiveteam.org/dnshistory dnshistory]
<!-- | source = [https://github.com/ArchiveTeam/dnshistory dnshistory] -->
| source = [https://github.com/ArchiveTeam/dnshistory-grab dnshistory-grab]
| irc = greatlookup
| irc = greatlookup
}}
}}
Line 36: Line 36:
* 251 don't have a single domain: [http://pastebin.com/raw/JVXRDhru they]
* 251 don't have a single domain: [http://pastebin.com/raw/JVXRDhru they]


<!-- SOON
== How can I help? ==
== How can I help? ==


Line 68: Line 67:


If you want to help in other projects, want to learn more about ArchiveTeam, or even help in development in general, navigate to the [[Main Page]] of this wiki, from there you can reach a lot of information. The Team consists of volunteers working on the projects in their free time, so helping hands (and resources) are always welcome.
If you want to help in other projects, want to learn more about ArchiveTeam, or even help in development in general, navigate to the [[Main Page]] of this wiki, from there you can reach a lot of information. The Team consists of volunteers working on the projects in their free time, so helping hands (and resources) are always welcome.
-->
 
<!--
== References ==
<references/>
-->
{{Navigation box}}
{{Navigation box}}

Revision as of 18:12, 4 July 2016

DNS History
DNS History logo
"Here at DNS History we have been crawling DNS records since 2009, our database currently contains over 500 million domains and over 2 billion DNS records."
"Here at DNS History we have been crawling DNS records since 2009, our database currently contains over 500 million domains and over 2 billion DNS records."
URL dnshistory.org[IAWcite.todayMemWeb]
Status Closing
Archiving status In progress...
Archiving type Unknown
Project source dnshistory-grab
Project tracker dnshistory
IRC channel #greatlookup (on hackint)

DNS History is a DNS record archive provided by 8086 Consulting.

On July 10th, 2016, DNS History will be shut down due to "funding issues".

Announcement

On the main page, on the right:

"This site will shut down on the 10th July 2016 due to funding issues."

There has been no further annoucement which we have been able to find at this time, the operators have not replied, and the forums are inaccessible.

Discovery results

A scrape of all TLDs which are crawled by DNS History currently is available here.

Out of the 1365 TLDs,

  • 1114 do have pages
    • 1110 are here
    • .info: 251346 pages
    • .org: 288405 pages
    • .net: 422569 pages
    • .com: discovery still running (expected to be millions of pages)
  • 251 don't have a single domain: they

How can I help?

Running a Warrior

You can start up a Warrior and there select DNSHistory. (If you don't really care what you are archiving, select ArchiveTeam's Choice instead, as at some points ArchiveTeam may priorize another project.)

Running the script manually

If you use Linux and you're a bit familiar with it, you can try running the script directly. - When it's written!

The instructions can be found at github.com/ArchiveTeam/coursera-grab.

Some additional information
Don't forget to replace YOURNICKHERE with your nickname.

The number after --concurrent determines how many threads run at the same time. You can increase this number if your resources (RAM, CPU, bandwidth) are sufficient. However, if you constantly see messages about rate limiting, there is no need to increase the concurrency.

If you want to stop the script, please do it gracefully if possible. To do so, create an empty file named STOP in the folder of the script (terminal command: touch STOP). The script finishes the current item(s) and stops only after that. (If you kill the script immediately, the items get broken, and they will need to be reassigned to another user.) – Before starting the script again, don't forget to remove the STOP file.

If you see "Project code is out of date", kill the script, go to its folder (cd coursera-grab) and issue git pull https://github.com/ArchiveTeam/coursera-grab. After the updating has finished, re-launch the script.

Donating to the Internet Archive

Content downloaded by the ArchiveTeam will be uploaded to the Internet Archive, where it will be stored and be available – hopefully – forever. However, storing it costs thousands of dollars in the long run. So, if you can afford, please consider donating to the Internet Archive, so that this piece of history can be kept for us all. http://archive.org/donate

Do you like our cause?

If you want to help in other projects, want to learn more about ArchiveTeam, or even help in development in general, navigate to the Main Page of this wiki, from there you can reach a lot of information. The Team consists of volunteers working on the projects in their free time, so helping hands (and resources) are always welcome.