Difference between revisions of "Project Sonar"

From Archiveteam
Jump to navigation Jump to search
(Created page with "{{Infobox project | title = Project Sonar | URL = {{URL|https://www.rapid7.com/research/project-sonar/|Description}}<br />{{URL|https://opendata.rapid7.com/|Datasets}} | proje...")
 
(Add size estimate)
Line 9: Line 9:
  
 
== Datasets ==
 
== Datasets ==
As of November 2018, the following data is available through Project Sonar:
+
As of 2018-11-13, the following data is available through Project Sonar:
  
 
{| class="wikitable"
 
{| class="wikitable"
 
|-
 
|-
! Dataset !! Description !! Scan frequency !! Data range !! Available without registration
+
! Dataset !! Description !! Scan frequency !! Data range !! Available without registration !! Size !! Size growth per month
 
|-
 
|-
| [https://opendata.rapid7.com/sonar.fdns_v2/ Forward DNS (FDNS)] || "DNS 'ANY', 'A', 'AAAA', 'TXT', 'MX', and 'CNAME' responses for known forward DNS names" || Weekly || Feb 2017 to present || Third most recent scan
+
| [https://opendata.rapid7.com/sonar.fdns_v2/ Forward DNS (FDNS)] || "DNS 'ANY', 'A', 'AAAA', 'TXT', 'MX', and 'CNAME' responses for known forward DNS names" || Weekly || Feb 2017 to present || Third most recent scan || 2.8 TiB || 220 GiB
 
|-
 
|-
| [https://opendata.rapid7.com/sonar.rdns_v2/ Reverse DNS (RDNS)] || "DNS IPv4 PTR responses" || Weekly || Feb 2017 to present || Third most recent scan
+
| [https://opendata.rapid7.com/sonar.rdns_v2/ Reverse DNS (RDNS)] || "DNS IPv4 PTR responses" || Weekly || Feb 2017 to present || Third most recent scan || 1.6 TiB || 90 GiB
 
|-
 
|-
| [https://opendata.rapid7.com/sonar.http/ HTTP GET Responses] || "Responses to HTTP/1.1 GET requests against various HTTP ports" || Bi-weekly || Oct 2013 to present || Second most recent scan
+
| [https://opendata.rapid7.com/sonar.http/ HTTP GET Responses] || "Responses to HTTP/1.1 GET requests against various HTTP ports" || Bi-weekly || Oct 2013 to present || Second most recent scan || 8.6 TiB || 155 GiB
 
|-
 
|-
| [https://opendata.rapid7.com/sonar.https/ HTTPS GET Responses] || "Responses to HTTP/1.1 GET requests against various HTTPS ports" || Bi-weekly || July 2017 to present || Second most recent scan
+
| [https://opendata.rapid7.com/sonar.https/ HTTPS GET Responses] || "Responses to HTTP/1.1 GET requests against various HTTPS ports" || Bi-weekly || July 2017 to present || Second most recent scan || 4.2 TiB || 125 GiB
 
|-
 
|-
| [https://opendata.rapid7.com/sonar.ssl/ SSL Certificates] || "X.509 certificate metadata observed when communicating with HTTPS endpoints" || Bi-weekly || Oct 2013 to present || Second most recent scan
+
| [https://opendata.rapid7.com/sonar.ssl/ SSL Certificates] || "X.509 certificate metadata observed when communicating with HTTPS endpoints" || Bi-weekly || Oct 2013 to present || Second most recent scan || 380 GiB || 9 GiB
 
|-
 
|-
| [https://opendata.rapid7.com/sonar.moressl/ More SSL Certificates (non-443)] || "X.509 certificate metadata observed when communicating with miscellaneous non-HTTPS endpoints, such as IMAPS, POP3S, etc." || Bi-weekly || June 2014 to present || Second most recent scan
+
| [https://opendata.rapid7.com/sonar.moressl/ More SSL Certificates (non-443)] || "X.509 certificate metadata observed when communicating with miscellaneous non-HTTPS endpoints, such as IMAPS, POP3S, etc." || Bi-weekly || June 2014 to present || Second most recent scan || 130 GiB || 5 GiB
 
|-
 
|-
| [https://opendata.rapid7.com/sonar.udp/ UDP Scans] || "UDP scan results for common UDP services across all of IPv4" || Monthly || June 2014 to present || Second most recent scan
+
| [https://opendata.rapid7.com/sonar.udp/ UDP Scans] || "UDP scan results for common UDP services across all of IPv4" || Monthly || June 2014 to present || Second most recent scan || 295 GiB || 2 GiB
 
|-
 
|-
| [https://opendata.rapid7.com/sonar.tcp/ TCP Scans] || "SYN scan results for common TCP services across all of IPv4" || Weekly || Mar 2017 to present || Second most recent scan
+
| [https://opendata.rapid7.com/sonar.tcp/ TCP Scans] || "SYN scan results for common TCP services across all of IPv4" || Weekly || Mar 2017 to present || Second most recent scan || 140 GiB || 11 GiB
 
|-
 
|-
| [https://opendata.rapid7.com/sonar.national_exposure/ National Exposure Scans] || "Open port results for Rapid7's National Exposure reports" || N/A || Apr 2016 to Apr 2018 || None
+
| [https://opendata.rapid7.com/sonar.national_exposure/ National Exposure Scans] || "Open port results for Rapid7's National Exposure reports" || N/A || Apr 2016 to Apr 2018 || None || 13 GiB || N/A
 
|-
 
|-
| [https://opendata.rapid7.com/heisenberg.cowrie/ Rapid7 Heisenberg Cloud Honeypot cowrie Logs] || "Rapid7 Heisenberg Cloud Honeypot cowrie Logs" || N/A || Nov 2016 || None
+
| [https://opendata.rapid7.com/heisenberg.cowrie/ Rapid7 Heisenberg Cloud Honeypot cowrie Logs] || "Rapid7 Heisenberg Cloud Honeypot cowrie Logs" || N/A || Nov 2016 || None || 520 MiB || N/A
 
|-
 
|-
| [https://opendata.rapid7.com/sonar.fdns/ Forward DNS (FDNS) -- ANY 2014-2017] || "DNS 'ANY' responses for known forward DNS names from 2014-2017" || Weekly || Mar 2014 to Jan 2017 || None
+
| [https://opendata.rapid7.com/sonar.fdns/ Forward DNS (FDNS) -- ANY 2014-2017] || "DNS 'ANY' responses for known forward DNS names from 2014-2017" || Weekly || Mar 2014 to Jan 2017 || None || 1.1 TiB || N/A
 
|-
 
|-
| [https://opendata.rapid7.com/sonar.rdns/ Reverse DNS (RDNS) -- 2013-2017] || "DNS IPv4 PTR responses from 2013-2017" || Weekly || Sept 2013 to Feb 2017 || None
+
| [https://opendata.rapid7.com/sonar.rdns/ Reverse DNS (RDNS) -- 2013-2017] || "DNS IPv4 PTR responses from 2013-2017" || Weekly || Sept 2013 to Feb 2017 || None || 650 GiB || N/A
 
|-
 
|-
| [https://opendata.rapid7.com/sonar.cio/ Critical.IO Service Fingerprints] || "The Critical.IO project was designed to uncover large-scale vulnerabilities on the global IPv4 internet and scanned a number of ports across between May 2012 and March 2013." || Monthly || May 2012 to Mar 2013 || None
+
| [https://opendata.rapid7.com/sonar.cio/ Critical.IO Service Fingerprints] || "The Critical.IO project was designed to uncover large-scale vulnerabilities on the global IPv4 internet and scanned a number of ports across between May 2012 and March 2013." || Monthly || May 2012 to Mar 2013 || None || 355 GiB || N/A
 +
|-
 +
! scope="row" colspan="5"|Total
 +
| 20.2 TiB
 +
| 620 GiB
 
|}
 
|}

Revision as of 19:16, 13 November 2018

Project Sonar
URL Description[IAWcite.todayMemWeb]
Datasets[IAWcite.todayMemWeb]
Status Online!
Archiving status Not saved yet
Archiving type Unknown
IRC channel #archiveteam-bs (on hackint)

Project Sonar by security company Rapid7 is a collection of internet-wide network scans with the aim of investigating vulnerabilities. The data is available for free, though many files (historical and most current ones) require an account.

Datasets

As of 2018-11-13, the following data is available through Project Sonar:

Dataset Description Scan frequency Data range Available without registration Size Size growth per month
Forward DNS (FDNS) "DNS 'ANY', 'A', 'AAAA', 'TXT', 'MX', and 'CNAME' responses for known forward DNS names" Weekly Feb 2017 to present Third most recent scan 2.8 TiB 220 GiB
Reverse DNS (RDNS) "DNS IPv4 PTR responses" Weekly Feb 2017 to present Third most recent scan 1.6 TiB 90 GiB
HTTP GET Responses "Responses to HTTP/1.1 GET requests against various HTTP ports" Bi-weekly Oct 2013 to present Second most recent scan 8.6 TiB 155 GiB
HTTPS GET Responses "Responses to HTTP/1.1 GET requests against various HTTPS ports" Bi-weekly July 2017 to present Second most recent scan 4.2 TiB 125 GiB
SSL Certificates "X.509 certificate metadata observed when communicating with HTTPS endpoints" Bi-weekly Oct 2013 to present Second most recent scan 380 GiB 9 GiB
More SSL Certificates (non-443) "X.509 certificate metadata observed when communicating with miscellaneous non-HTTPS endpoints, such as IMAPS, POP3S, etc." Bi-weekly June 2014 to present Second most recent scan 130 GiB 5 GiB
UDP Scans "UDP scan results for common UDP services across all of IPv4" Monthly June 2014 to present Second most recent scan 295 GiB 2 GiB
TCP Scans "SYN scan results for common TCP services across all of IPv4" Weekly Mar 2017 to present Second most recent scan 140 GiB 11 GiB
National Exposure Scans "Open port results for Rapid7's National Exposure reports" N/A Apr 2016 to Apr 2018 None 13 GiB N/A
Rapid7 Heisenberg Cloud Honeypot cowrie Logs "Rapid7 Heisenberg Cloud Honeypot cowrie Logs" N/A Nov 2016 None 520 MiB N/A
Forward DNS (FDNS) -- ANY 2014-2017 "DNS 'ANY' responses for known forward DNS names from 2014-2017" Weekly Mar 2014 to Jan 2017 None 1.1 TiB N/A
Reverse DNS (RDNS) -- 2013-2017 "DNS IPv4 PTR responses from 2013-2017" Weekly Sept 2013 to Feb 2017 None 650 GiB N/A
Critical.IO Service Fingerprints "The Critical.IO project was designed to uncover large-scale vulnerabilities on the global IPv4 internet and scanned a number of ports across between May 2012 and March 2013." Monthly May 2012 to Mar 2013 None 355 GiB N/A
Total 20.2 TiB 620 GiB