Difference between revisions of "Project Sonar"

From Archiveteam
Jump to navigation Jump to search
(Created page with "{{Infobox project | title = Project Sonar | URL = {{URL|https://www.rapid7.com/research/project-sonar/|Description}}<br />{{URL|https://opendata.rapid7.com/|Datasets}} | proje...")
 
m (Seems to affect other files as well)
 
(6 intermediate revisions by 3 users not shown)
Line 2: Line 2:
| title = Project Sonar
| title = Project Sonar
| URL = {{URL|https://www.rapid7.com/research/project-sonar/|Description}}<br />{{URL|https://opendata.rapid7.com/|Datasets}}
| URL = {{URL|https://www.rapid7.com/research/project-sonar/|Description}}<br />{{URL|https://opendata.rapid7.com/|Datasets}}
| project_status = {{online}}
| project_status = {{online}} (Closed to public)
| archiving_status = {{nosavedyet}}
| archiving_status = {{nosavedyet}}
}}
}}
Line 9: Line 9:


== Datasets ==
== Datasets ==
As of November 2018, the following data is available through Project Sonar:
As of 2021-06-29, the following data is available through Project Sonar:


{| class="wikitable"
{| class="wikitable"
|-
|-
! Dataset !! Description !! Scan frequency !! Data range !! Available without registration
! Dataset !! Description !! Scan frequency !! Data range !! Available without registration !! Size !! Size growth per month
|-
|-
| [https://opendata.rapid7.com/sonar.fdns_v2/ Forward DNS (FDNS)] || "DNS 'ANY', 'A', 'AAAA', 'TXT', 'MX', and 'CNAME' responses for known forward DNS names" || Weekly || Feb 2017 to present || Third most recent scan
| [https://opendata.rapid7.com/sonar.fdns_v2/ Forward DNS (FDNS)] || "DNS 'ANY', 'A', 'AAAA', 'TXT', 'MX', and 'CNAME' responses for known forward DNS names" || Weekly || Feb 2017 to present || Third most recent scan || 11.1&nbsp;TiB || 393&nbsp;GiB
|-
|-
| [https://opendata.rapid7.com/sonar.rdns_v2/ Reverse DNS (RDNS)] || "DNS IPv4 PTR responses" || Weekly || Feb 2017 to present || Third most recent scan
| [https://opendata.rapid7.com/sonar.rdns_v2/ Reverse DNS (RDNS)] || "DNS IPv4 PTR responses" || Weekly || Feb 2017 to present || Third most recent scan || 3.14&nbsp;TiB || 58&nbsp;GiB
|-
|-
| [https://opendata.rapid7.com/sonar.http/ HTTP GET Responses] || "Responses to HTTP/1.1 GET requests against various HTTP ports" || Bi-weekly || Oct 2013 to present || Second most recent scan
| [https://opendata.rapid7.com/sonar.http/ HTTP GET Responses] || "Responses to HTTP/1.1 GET requests against various HTTP ports" || Bi-weekly || Oct 2013 to present || Second most recent scan || 15.3&nbsp;TiB || 265&nbsp;GiB
|-
|-
| [https://opendata.rapid7.com/sonar.https/ HTTPS GET Responses] || "Responses to HTTP/1.1 GET requests against various HTTPS ports" || Bi-weekly || July 2017 to present || Second most recent scan
| [https://opendata.rapid7.com/sonar.https/ HTTPS GET Responses] || "Responses to HTTP/1.1 GET requests against various HTTPS ports" || Bi-weekly || July 2017 to present || Second most recent scan || 11.2&nbsp;TiB || 299&nbsp;GiB
|-
|-
| [https://opendata.rapid7.com/sonar.ssl/ SSL Certificates] || "X.509 certificate metadata observed when communicating with HTTPS endpoints" || Bi-weekly || Oct 2013 to present || Second most recent scan
| [https://opendata.rapid7.com/sonar.ssl/ SSL Certificates] || "X.509 certificate metadata observed when communicating with HTTPS endpoints" || Bi-weekly || Oct 2013 to present || Second most recent scan || 1.24&nbsp;TiB || 61&nbsp;GiB
|-
|-
| [https://opendata.rapid7.com/sonar.moressl/ More SSL Certificates (non-443)] || "X.509 certificate metadata observed when communicating with miscellaneous non-HTTPS endpoints, such as IMAPS, POP3S, etc." || Bi-weekly || June 2014 to present || Second most recent scan
| [https://opendata.rapid7.com/sonar.moressl/ More SSL Certificates (non-443)] || "X.509 certificate metadata observed when communicating with miscellaneous non-HTTPS endpoints, such as IMAPS, POP3S, etc." || Bi-weekly || June 2014 to present || Second most recent scan || 450&nbsp;GiB || 14.6&nbsp;GiB
|-
|-
| [https://opendata.rapid7.com/sonar.udp/ UDP Scans] || "UDP scan results for common UDP services across all of IPv4" || Monthly || June 2014 to present || Second most recent scan
| [https://opendata.rapid7.com/sonar.udp/ UDP Scans] || "UDP scan results for common UDP services across all of IPv4" || Monthly || June 2014 to present || Second most recent scan || 340&nbsp;GiB || 1&nbsp;GiB
|-
|-
| [https://opendata.rapid7.com/sonar.tcp/ TCP Scans] || "SYN scan results for common TCP services across all of IPv4" || Weekly || Mar 2017 to present || Second most recent scan
| [https://opendata.rapid7.com/sonar.tcp/ TCP Scans] || "SYN scan results for common TCP services across all of IPv4" || Weekly || Mar 2017 to present || Second most recent scan || 680&nbsp;GiB || 19&nbsp;GiB
|-
|-
| [https://opendata.rapid7.com/sonar.national_exposure/ National Exposure Scans] || "Open port results for Rapid7's National Exposure reports" || N/A || Apr 2016 to Apr 2018 || None
| [https://opendata.rapid7.com/sonar.national_exposure/ National Exposure Scans] || "Open port results for Rapid7's National Exposure reports" || N/A || Apr 2016 to Apr 2018 || None || 13&nbsp;GiB || N/A
|-
|-
| [https://opendata.rapid7.com/heisenberg.cowrie/ Rapid7 Heisenberg Cloud Honeypot cowrie Logs] || "Rapid7 Heisenberg Cloud Honeypot cowrie Logs" || N/A || Nov 2016 || None
| [https://opendata.rapid7.com/sonar.fdns/ Forward DNS (FDNS) -- ANY 2014-2017] || "DNS 'ANY' responses for known forward DNS names from 2014-2017" || Weekly || Mar 2014 to Jan 2017 || None || 1.1&nbsp;TiB || N/A
|-
|-
| [https://opendata.rapid7.com/sonar.fdns/ Forward DNS (FDNS) -- ANY 2014-2017] || "DNS 'ANY' responses for known forward DNS names from 2014-2017" || Weekly || Mar 2014 to Jan 2017 || None
| [https://opendata.rapid7.com/sonar.rdns/ Reverse DNS (RDNS) -- 2013-2017] || "DNS IPv4 PTR responses from 2013-2017" || Weekly || Sept 2013 to Feb 2017 || None || 650&nbsp;GiB || N/A
|-
|-
| [https://opendata.rapid7.com/sonar.rdns/ Reverse DNS (RDNS) -- 2013-2017] || "DNS IPv4 PTR responses from 2013-2017" || Weekly || Sept 2013 to Feb 2017 || None
| [https://opendata.rapid7.com/sonar.cio/ Critical.IO Service Fingerprints] || "The Critical.IO project was designed to uncover large-scale vulnerabilities on the global IPv4 internet and scanned a number of ports across between May 2012 and March 2013." || Monthly || May 2012 to Mar 2013 || None || 355&nbsp;GiB || N/A
|-
|-
| [https://opendata.rapid7.com/sonar.cio/ Critical.IO Service Fingerprints] || "The Critical.IO project was designed to uncover large-scale vulnerabilities on the global IPv4 internet and scanned a number of ports across between May 2012 and March 2013." || Monthly || May 2012 to Mar 2013 || None
! scope="row" colspan="5"|Total
| 46.3&nbsp;TiB
| 1.08&nbsp;TiB
|}
|}
(Note: The website uses the wrong units on the files. 'MB' on their list actually means MiB.)
=== No longer available (lost?) ===
* {{URL|https://opendata.rapid7.com/heisenberg.cowrie/|Rapid7 Heisenberg Cloud Honeypot cowrie Logs}} from Nov 2016. A single file of 520 MiB was available as of 2018-11-13 and has vanished as of 2021-06-29. The file was publicly accessible but is not available in the Wayback Machine and presumed {{lost}}.
==Closure to Public==
On February 10th, 2022, Rapid7 decided to close public access to Project Sonar. The project was closed due to "changes ... triggered by the evolving regulatory landscape"<ref>https://www.rapid7.com/blog/post/2022/02/10/evolving-how-we-share-rapid7-research-data-2/</ref>,  concerning the heavy presence of IP address in their data. Further, the API that made this data searchable (<strike>https://sonar.omnisint.io/</strike>) <ref> https://github.com/Cgboal/SonarSearch </ref> has been removed.
Some past data that was previously released to the public remains available in the [https://web.archive.org/web/*/https://f002.backblazeb2.com/file/rapid7-opendata/* Wayback Machine], with the latest available files being from 2021-09-24, though many of them appear to be truncated at 2GB.
== References ==
<references/>

Latest revision as of 07:28, 3 December 2023

Project Sonar
URL Description[IAWcite.todayMemWeb]
Datasets[IAWcite.todayMemWeb]
Status Online! (Closed to public)
Archiving status Not saved yet
Archiving type Unknown
IRC channel #archiveteam-bs (on hackint)

Project Sonar by security company Rapid7 is a collection of internet-wide network scans with the aim of investigating vulnerabilities. The data is available for free, though many files (historical and most current ones) require an account.

Datasets

As of 2021-06-29, the following data is available through Project Sonar:

Dataset Description Scan frequency Data range Available without registration Size Size growth per month
Forward DNS (FDNS) "DNS 'ANY', 'A', 'AAAA', 'TXT', 'MX', and 'CNAME' responses for known forward DNS names" Weekly Feb 2017 to present Third most recent scan 11.1 TiB 393 GiB
Reverse DNS (RDNS) "DNS IPv4 PTR responses" Weekly Feb 2017 to present Third most recent scan 3.14 TiB 58 GiB
HTTP GET Responses "Responses to HTTP/1.1 GET requests against various HTTP ports" Bi-weekly Oct 2013 to present Second most recent scan 15.3 TiB 265 GiB
HTTPS GET Responses "Responses to HTTP/1.1 GET requests against various HTTPS ports" Bi-weekly July 2017 to present Second most recent scan 11.2 TiB 299 GiB
SSL Certificates "X.509 certificate metadata observed when communicating with HTTPS endpoints" Bi-weekly Oct 2013 to present Second most recent scan 1.24 TiB 61 GiB
More SSL Certificates (non-443) "X.509 certificate metadata observed when communicating with miscellaneous non-HTTPS endpoints, such as IMAPS, POP3S, etc." Bi-weekly June 2014 to present Second most recent scan 450 GiB 14.6 GiB
UDP Scans "UDP scan results for common UDP services across all of IPv4" Monthly June 2014 to present Second most recent scan 340 GiB 1 GiB
TCP Scans "SYN scan results for common TCP services across all of IPv4" Weekly Mar 2017 to present Second most recent scan 680 GiB 19 GiB
National Exposure Scans "Open port results for Rapid7's National Exposure reports" N/A Apr 2016 to Apr 2018 None 13 GiB N/A
Forward DNS (FDNS) -- ANY 2014-2017 "DNS 'ANY' responses for known forward DNS names from 2014-2017" Weekly Mar 2014 to Jan 2017 None 1.1 TiB N/A
Reverse DNS (RDNS) -- 2013-2017 "DNS IPv4 PTR responses from 2013-2017" Weekly Sept 2013 to Feb 2017 None 650 GiB N/A
Critical.IO Service Fingerprints "The Critical.IO project was designed to uncover large-scale vulnerabilities on the global IPv4 internet and scanned a number of ports across between May 2012 and March 2013." Monthly May 2012 to Mar 2013 None 355 GiB N/A
Total 46.3 TiB 1.08 TiB

(Note: The website uses the wrong units on the files. 'MB' on their list actually means MiB.)

No longer available (lost?)

Closure to Public

On February 10th, 2022, Rapid7 decided to close public access to Project Sonar. The project was closed due to "changes ... triggered by the evolving regulatory landscape"[1], concerning the heavy presence of IP address in their data. Further, the API that made this data searchable (https://sonar.omnisint.io/) [2] has been removed.

Some past data that was previously released to the public remains available in the Wayback Machine, with the latest available files being from 2021-09-24, though many of them appear to be truncated at 2GB.

References