Difference between revisions of "Project Sonar"

From Archiveteam
Jump to navigation Jump to search
(Add size estimate)
(Update)
Line 9: Line 9:


== Datasets ==
== Datasets ==
As of 2018-11-13, the following data is available through Project Sonar:
As of 2021-06-29, the following data is available through Project Sonar:


{| class="wikitable"
{| class="wikitable"
Line 15: Line 15:
! Dataset !! Description !! Scan frequency !! Data range !! Available without registration !! Size !! Size growth per month
! Dataset !! Description !! Scan frequency !! Data range !! Available without registration !! Size !! Size growth per month
|-
|-
| [https://opendata.rapid7.com/sonar.fdns_v2/ Forward DNS (FDNS)] || "DNS 'ANY', 'A', 'AAAA', 'TXT', 'MX', and 'CNAME' responses for known forward DNS names" || Weekly || Feb 2017 to present || Third most recent scan || 2.8 TiB || 220 GiB
| [https://opendata.rapid7.com/sonar.fdns_v2/ Forward DNS (FDNS)] || "DNS 'ANY', 'A', 'AAAA', 'TXT', 'MX', and 'CNAME' responses for known forward DNS names" || Weekly || Feb 2017 to present || Third most recent scan || 11.1 TiB || 393 GiB
|-
|-
| [https://opendata.rapid7.com/sonar.rdns_v2/ Reverse DNS (RDNS)] || "DNS IPv4 PTR responses" || Weekly || Feb 2017 to present || Third most recent scan || 1.6 TiB || 90 GiB
| [https://opendata.rapid7.com/sonar.rdns_v2/ Reverse DNS (RDNS)] || "DNS IPv4 PTR responses" || Weekly || Feb 2017 to present || Third most recent scan || 3.14 TiB || 58 GiB
|-
|-
| [https://opendata.rapid7.com/sonar.http/ HTTP GET Responses] || "Responses to HTTP/1.1 GET requests against various HTTP ports" || Bi-weekly || Oct 2013 to present || Second most recent scan || 8.6 TiB || 155 GiB
| [https://opendata.rapid7.com/sonar.http/ HTTP GET Responses] || "Responses to HTTP/1.1 GET requests against various HTTP ports" || Bi-weekly || Oct 2013 to present || Second most recent scan || 15.3 TiB || 265 GiB
|-
|-
| [https://opendata.rapid7.com/sonar.https/ HTTPS GET Responses] || "Responses to HTTP/1.1 GET requests against various HTTPS ports" || Bi-weekly || July 2017 to present || Second most recent scan || 4.2 TiB || 125 GiB
| [https://opendata.rapid7.com/sonar.https/ HTTPS GET Responses] || "Responses to HTTP/1.1 GET requests against various HTTPS ports" || Bi-weekly || July 2017 to present || Second most recent scan || 11.2 TiB || 299 GiB
|-
|-
| [https://opendata.rapid7.com/sonar.ssl/ SSL Certificates] || "X.509 certificate metadata observed when communicating with HTTPS endpoints" || Bi-weekly || Oct 2013 to present || Second most recent scan || 380 GiB || 9 GiB
| [https://opendata.rapid7.com/sonar.ssl/ SSL Certificates] || "X.509 certificate metadata observed when communicating with HTTPS endpoints" || Bi-weekly || Oct 2013 to present || Second most recent scan || 1.24 TiB || 61 GiB
|-
|-
| [https://opendata.rapid7.com/sonar.moressl/ More SSL Certificates (non-443)] || "X.509 certificate metadata observed when communicating with miscellaneous non-HTTPS endpoints, such as IMAPS, POP3S, etc." || Bi-weekly || June 2014 to present || Second most recent scan || 130 GiB || 5 GiB
| [https://opendata.rapid7.com/sonar.moressl/ More SSL Certificates (non-443)] || "X.509 certificate metadata observed when communicating with miscellaneous non-HTTPS endpoints, such as IMAPS, POP3S, etc." || Bi-weekly || June 2014 to present || Second most recent scan || 450 GiB || 14.6 GiB
|-
|-
| [https://opendata.rapid7.com/sonar.udp/ UDP Scans] || "UDP scan results for common UDP services across all of IPv4" || Monthly || June 2014 to present || Second most recent scan || 295 GiB || 2 GiB
| [https://opendata.rapid7.com/sonar.udp/ UDP Scans] || "UDP scan results for common UDP services across all of IPv4" || Monthly || June 2014 to present || Second most recent scan || 340 GiB || 1 GiB
|-
|-
| [https://opendata.rapid7.com/sonar.tcp/ TCP Scans] || "SYN scan results for common TCP services across all of IPv4" || Weekly || Mar 2017 to present || Second most recent scan || 140 GiB || 11 GiB
| [https://opendata.rapid7.com/sonar.tcp/ TCP Scans] || "SYN scan results for common TCP services across all of IPv4" || Weekly || Mar 2017 to present || Second most recent scan || 680 GiB || 19 GiB
|-
|-
| [https://opendata.rapid7.com/sonar.national_exposure/ National Exposure Scans] || "Open port results for Rapid7's National Exposure reports" || N/A || Apr 2016 to Apr 2018 || None || 13 GiB || N/A
| [https://opendata.rapid7.com/sonar.national_exposure/ National Exposure Scans] || "Open port results for Rapid7's National Exposure reports" || N/A || Apr 2016 to Apr 2018 || None || 13 GiB || N/A
|-
|-
| [https://opendata.rapid7.com/heisenberg.cowrie/ Rapid7 Heisenberg Cloud Honeypot cowrie Logs] || "Rapid7 Heisenberg Cloud Honeypot cowrie Logs" || N/A || Nov 2016 || None || 520 MiB || N/A
| [https://opendata.rapid7.com/sonar.fdns/ Forward DNS (FDNS) -- ANY 2014-2017] || "DNS 'ANY' responses for known forward DNS names from 2014-2017" || Weekly || Mar 2014 to Jan 2017 || None || 1.1 TiB || N/A
|-
|-
| [https://opendata.rapid7.com/sonar.fdns/ Forward DNS (FDNS) -- ANY 2014-2017] || "DNS 'ANY' responses for known forward DNS names from 2014-2017" || Weekly || Mar 2014 to Jan 2017 || None || 1.1 TiB || N/A
| [https://opendata.rapid7.com/sonar.rdns/ Reverse DNS (RDNS) -- 2013-2017] || "DNS IPv4 PTR responses from 2013-2017" || Weekly || Sept 2013 to Feb 2017 || None || 650 GiB || N/A
|-
|-
| [https://opendata.rapid7.com/sonar.rdns/ Reverse DNS (RDNS) -- 2013-2017] || "DNS IPv4 PTR responses from 2013-2017" || Weekly || Sept 2013 to Feb 2017 || None || 650 GiB || N/A
| [https://opendata.rapid7.com/sonar.cio/ Critical.IO Service Fingerprints] || "The Critical.IO project was designed to uncover large-scale vulnerabilities on the global IPv4 internet and scanned a number of ports across between May 2012 and March 2013." || Monthly || May 2012 to Mar 2013 || None || 355 GiB || N/A
|-
| [https://opendata.rapid7.com/sonar.cio/ Critical.IO Service Fingerprints] || "The Critical.IO project was designed to uncover large-scale vulnerabilities on the global IPv4 internet and scanned a number of ports across between May 2012 and March 2013." || Monthly || May 2012 to Mar 2013 || None || 355 GiB || N/A
|-
|-
! scope="row" colspan="5"|Total
! scope="row" colspan="5"|Total
| 20.2 TiB
| 46.3 TiB
| 620 GiB
| 1.08 TiB
|}
|}
(Note: The website uses the wrong units on the files. 'MB' on their list actually means MiB.)
=== No longer available (lost?) ===
* {{URL|https://opendata.rapid7.com/heisenberg.cowrie/|Rapid7 Heisenberg Cloud Honeypot cowrie Logs}} from Nov 2016. A single file of 520 MiB was available as of 2018-11-13 and has vanished as of 2021-06-29. The file was publicly accessible but is not available in the Wayback Machine and presumed {{lost}}.

Revision as of 22:01, 29 June 2021

Project Sonar
URL Description[IAWcite.todayMemWeb]
Datasets[IAWcite.todayMemWeb]
Status Online!
Archiving status Not saved yet
Archiving type Unknown
IRC channel #archiveteam-bs (on hackint)

Project Sonar by security company Rapid7 is a collection of internet-wide network scans with the aim of investigating vulnerabilities. The data is available for free, though many files (historical and most current ones) require an account.

Datasets

As of 2021-06-29, the following data is available through Project Sonar:

Dataset Description Scan frequency Data range Available without registration Size Size growth per month
Forward DNS (FDNS) "DNS 'ANY', 'A', 'AAAA', 'TXT', 'MX', and 'CNAME' responses for known forward DNS names" Weekly Feb 2017 to present Third most recent scan 11.1 TiB 393 GiB
Reverse DNS (RDNS) "DNS IPv4 PTR responses" Weekly Feb 2017 to present Third most recent scan 3.14 TiB 58 GiB
HTTP GET Responses "Responses to HTTP/1.1 GET requests against various HTTP ports" Bi-weekly Oct 2013 to present Second most recent scan 15.3 TiB 265 GiB
HTTPS GET Responses "Responses to HTTP/1.1 GET requests against various HTTPS ports" Bi-weekly July 2017 to present Second most recent scan 11.2 TiB 299 GiB
SSL Certificates "X.509 certificate metadata observed when communicating with HTTPS endpoints" Bi-weekly Oct 2013 to present Second most recent scan 1.24 TiB 61 GiB
More SSL Certificates (non-443) "X.509 certificate metadata observed when communicating with miscellaneous non-HTTPS endpoints, such as IMAPS, POP3S, etc." Bi-weekly June 2014 to present Second most recent scan 450 GiB 14.6 GiB
UDP Scans "UDP scan results for common UDP services across all of IPv4" Monthly June 2014 to present Second most recent scan 340 GiB 1 GiB
TCP Scans "SYN scan results for common TCP services across all of IPv4" Weekly Mar 2017 to present Second most recent scan 680 GiB 19 GiB
National Exposure Scans "Open port results for Rapid7's National Exposure reports" N/A Apr 2016 to Apr 2018 None 13 GiB N/A
Forward DNS (FDNS) -- ANY 2014-2017 "DNS 'ANY' responses for known forward DNS names from 2014-2017" Weekly Mar 2014 to Jan 2017 None 1.1 TiB N/A
Reverse DNS (RDNS) -- 2013-2017 "DNS IPv4 PTR responses from 2013-2017" Weekly Sept 2013 to Feb 2017 None 650 GiB N/A
Critical.IO Service Fingerprints "The Critical.IO project was designed to uncover large-scale vulnerabilities on the global IPv4 internet and scanned a number of ports across between May 2012 and March 2013." Monthly May 2012 to Mar 2013 None 355 GiB N/A
Total 46.3 TiB 1.08 TiB

(Note: The website uses the wrong units on the files. 'MB' on their list actually means MiB.)

No longer available (lost?)