Difference between revisions of "DNS History"

From Archiveteam
Jump to navigation Jump to search
m (→‎Final announcement: Add year for clarity)
(Update infobox)
(3 intermediate revisions by 3 users not shown)
Line 5: Line 5:
| URL = {{url|https://dnshistory.org/|dnshistory.org}}
| URL = {{url|https://dnshistory.org/|dnshistory.org}}
| description = "Here at DNS History we have been crawling DNS records since 2009, our database currently contains over 500 million domains and over 2 billion DNS records."
| description = "Here at DNS History we have been crawling DNS records since 2009, our database currently contains over 500 million domains and over 2 billion DNS records."
| project_status = {{closing}}
| project_status = {{endangered}}
| archiving_status = {{notsavedyet}}
| archiving_status = {{notsavedyet}}
| tracker = [http://tracker.archiveteam.org/dnshistory dnshistory]
| tracker = [http://tracker.archiveteam.org/dnshistory dnshistory]
| source = [https://github.com/ArchiveTeam/dnshistory-grab dnshistory-grab]
| source = [https://github.com/ArchiveTeam/dnshistory-grab dnshistory-grab]
| irc = greatlookup
| irc = greatlookup
| irc_network = EFnet
| irc_abandoned = true
}}
}}


'''DNS History''' is a [http://en.wikipedia.org/wiki/Domain_Name_System DNS] record archive provided by [http://8086.net 8086 Consultancy].
'''DNS History''' is a [[wikipedia:Domain Name System|DNS]] record archive provided by {{URL|http://8086.net|2=8086 Consultancy}}.


It was scheduled to close on July 10, 2016 due to "funding issues".  
It was scheduled to close on July 10, 2016 due to "funding issues".  
Line 18: Line 20:
== Announcement ==
== Announcement ==


According to a [https://twitter.com/dnshistory/status/747021758075703300 Twitter entry] dated 2016-06-26:
According to a {{URL|https://twitter.com/dnshistory/status/747021758075703300|2=Twitter entry}} dated 2016-06-26:


:"dnshistory.org will be closing down in a couple of weeks - it has been a fun project over the years thanks folks."
:"dnshistory.org will be closing down in a couple of weeks - it has been a fun project over the years thanks folks."


Also, on the [http://dnshistory.org main page], on the right:
Also, on the {{URL|http://dnshistory.org|2=main page}}, on the right:


:"This site will shut down on the 10th July 2016 due to funding issues."
:"This site will shut down on the 10th July 2016 due to funding issues."
Line 34: Line 36:
Out of the 1365 TLDs,
Out of the 1365 TLDs,
* 1114 do have pages
* 1114 do have pages
** 1110 are [http://pastebin.com/raw/hQ7bXcem here]
** 1110 are {{URL|https://pastebin.com/raw/hQ7bXcem|2=here}}
** .info: 251346 pages
** .info: 251346 pages
** .org: 288405 pages
** .org: 288405 pages
** .net: 422569 pages
** .net: 422569 pages
** .com: unknown (expected to be millions of pages)
** .com: unknown (expected to be millions of pages)
* 251 don't have a single domain: [http://pastebin.com/raw/JVXRDhru they]
* 251 don't have a single domain: {{URL|https://pastebin.com/raw/JVXRDhru|2=they}}


== Archiving ==
== Archiving ==
Line 45: Line 47:
After ArchiveTeam started the project (2016-07-04), dnshistory.org staff shortly activated CloudFlare in order to block our access to the site.
After ArchiveTeam started the project (2016-07-04), dnshistory.org staff shortly activated CloudFlare in order to block our access to the site.


[[user:bzc6p]] contacted them asking for cooperation, but they [https://secure.8086.net/portal/viewticket.php?tid=NTM-405143&c=9JIYc46J replied] that ''"there is no way you can archive the >billion pages on the site and trying to do so is causing issues for other users on the site."''
[[user:bzc6p|bzc6p]] contacted them asking for cooperation, but they {{URL|1=https://secure.8086.net/portal/viewticket.php?tid=NTM-405143&c=9JIYc46J|2=replied}} that ''"there is no way you can archive the >billion pages on the site and trying to do so is causing issues for other users on the site."''


Later CloudFlare seemed to have been deactivated, but the site was broken for a month, giving 500 errors. However, as of 2016-08-17, the site is up, with various queries, including search, still functioning, with the closure notice still up.
Later CloudFlare seemed to have been deactivated, but the site was broken for a month, giving 500 errors. However, as of 2016-08-17, the site is up, with various queries, including search, still functioning, with the closure notice still up.
Line 54: Line 56:


ArchiveTeam was hardly able to save anything.
ArchiveTeam was hardly able to save anything.
Since at least 2018, the site has had numerous prolonged downtimes returning blank HTTP 500 responses on all pages except the homepage.


{{Navigation box}}
{{Navigation box}}

Revision as of 16:45, 14 November 2021

DNS History
DNS History logo
"Here at DNS History we have been crawling DNS records since 2009, our database currently contains over 500 million domains and over 2 billion DNS records."
"Here at DNS History we have been crawling DNS records since 2009, our database currently contains over 500 million domains and over 2 billion DNS records."
URL dnshistory.org[IAWcite.todayMemWeb]
Status Endangered
Archiving status Not saved yet
Archiving type Unknown
Project source dnshistory-grab
Project tracker dnshistory
IRC channel #archiveteam-bs (on hackint)
(formerly #greatlookup (on EFnet))

DNS History is a DNS record archive provided by 8086 Consultancy[IAWcite.todayMemWeb].

It was scheduled to close on July 10, 2016 due to "funding issues".

Announcement

According to a Twitter entry[IAWcite.todayMemWeb] dated 2016-06-26:

"dnshistory.org will be closing down in a couple of weeks - it has been a fun project over the years thanks folks."

Also, on the main page[IAWcite.todayMemWeb], on the right:

"This site will shut down on the 10th July 2016 due to funding issues."

Final announcement

On the main page, appeared on August 23, 2016:

"DNS History has now shutdown, any updates are for my personal requirements only - the site may go up/down as I need the resources it uses. The number of servers dedicated to the DNS History project is now ZERO. Due to the ArchiveTeams self-righteous attitude CloudFlares DDoS protection has been enabled. They are quite open about their attitude of ignoring robots.txt files and work around blocks on their user agent - this as abuse."

Discovery results

Out of the 1365 TLDs,

  • 1114 do have pages
    • 1110 are here[IAWcite.todayMemWeb]
    • .info: 251346 pages
    • .org: 288405 pages
    • .net: 422569 pages
    • .com: unknown (expected to be millions of pages)
  • 251 don't have a single domain: they[IAWcite.todayMemWeb]

Archiving

After ArchiveTeam started the project (2016-07-04), dnshistory.org staff shortly activated CloudFlare in order to block our access to the site.

bzc6p contacted them asking for cooperation, but they replied[IAWcite.todayMemWeb] that "there is no way you can archive the >billion pages on the site and trying to do so is causing issues for other users on the site."

Later CloudFlare seemed to have been deactivated, but the site was broken for a month, giving 500 errors. However, as of 2016-08-17, the site is up, with various queries, including search, still functioning, with the closure notice still up.

On 2016-08-20, ArchiveTeam restarted its efforts. On 08-21, although the site seemed to deal with the load, and the site has officially been closed for 42 days, they started to ban our crawlers. On 08-22, they deployed their impenetrable Cloudflare browser checker again.

On August 23, a final shutdown notice appeared (see above).

ArchiveTeam was hardly able to save anything.

Since at least 2018, the site has had numerous prolonged downtimes returning blank HTTP 500 responses on all pages except the homepage.