Difference between revisions of "The WARC Ecosystem"

From Archiveteam
Jump to navigation Jump to search
(→‎CDX File Format: fix long line)
(Add a summary of what WARC is)
(24 intermediate revisions by 12 users not shown)
Line 1: Line 1:
Everything about the WARC format and the tools that support it.
Everything about the WARC format and the tools that support it.
WARC is a file format for accurately storing Web traffic.


== Information ==
== Information ==
* https://en.wikipedia.org/wiki/Web_ARChive
* [[wikipedia:Web_ARChive]]
* https://webarchive.jira.com/wiki/pages/viewpage.action?pageId=4817 - Contains examples of WARC records
* {{URL|https://webarchive.jira.com/wiki/pages/viewpage.action?pageId=4817}} - Contains examples of WARC records
* [http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf ISO 28500 - The WARC File Format]
* {{URL|http://bibnum.bnf.fr/WARC/|The WARC File Format (ISO 28500) - Information, Maintenance, Drafts}}
* http://archive-access.sourceforge.net/warc/ - WARC ISO docs
* {{URL|http://archive-access.sourceforge.net/warc/}} - WARC ISO docs
* http://www.digitalpreservation.gov/formats/fdd/fdd000236.shtml
* {{URL|https://www.loc.gov/preservation/digital/formats/fdd/fdd000236.shtml}}
* http://www.netpreserve.org/resources/warc-implementation-guidelines-v1 http://www.netpreserve.org/sites/default/files/resources/WARC_Guidelines_v1.pdf
* {{URL|https://netpreserve.org/resources/warc-implementation-guidelines-v1/}}
* http://commoncrawl.org/navigating-the-warc-file-format/
* {{URL|https://netpreserve.org/resources/WARC_Guidelines_v1.pdf}}
* {{URL|https://commoncrawl.org/2014/04/navigating-the-warc-file-format/}}
* {{URL|https://www.taricorp.net/2016/web-history-warc}}
* {{URL|https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.0/|WARC/1.0 specification}}
* {{URL|https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/|WARC/1.1 specification}}
* {{URL|https://github.com/iipc/warc-specifications|GitHub repository coordinating the specification}}


== Tools ==
== Tools ==


{|class="wikitable"
{|class="wikitable"
! name
! Name
! license
! License
! programming language
! Language
! test suite
! Testing
! has documentation
! Documentation
! # of authors
! Author count
! description
! Description
|-
|-
| [https://www.gnu.org/software/wget/ wget v1.14+]
| [https://www.gnu.org/software/wget/ wget v1.14+]
Line 30: Line 37:
|-
|-
| InternetArchive's [https://github.com/internetarchive/warc warc python library]
| InternetArchive's [https://github.com/internetarchive/warc warc python library]
| GPL v2 || Python
| GPL v2 || Python 2
| looks to have a test suite - https://github.com/internetarchive/warc/blob/master/warc/tests/test_warc.py
| looks to have a test suite - https://github.com/internetarchive/warc/blob/master/warc/tests/test_warc.py
| A readme with examples online at http://warc.readthedocs.org/en/latest/
| README with examples online at https://warc.readthedocs.io/en/latest/
| 3 commiters on github
| 3 commiters on github
| library to work with WARC files
| library to work with WARC files
Line 39: Line 46:
| ISC || Python
| ISC || Python
| Not enough tests
| Not enough tests
| A readme file + [http://scrapy.org/ Scrapy docs]
| README + [https://scrapy.org/ Scrapy docs]
| 1 author
| 1 author
| Mirrors websites and saves the results to a WARC file
| Mirrors websites and saves the results to a WARC file
Line 46: Line 53:
| ISC || Python
| ISC || Python
| NO TEST SUITE
| NO TEST SUITE
| A readme file
| README
| 1 author
| 1 author
| a simple HTTP proxy that saves all HTTP traffic to a file
| a simple HTTP proxy that saves all HTTP traffic to a file
Line 54: Line 61:
| Python
| Python
| NO TEST SUITE
| NO TEST SUITE
| A readme file
| README
| 1 author
| 1 author
| HTTPS proxy that saves traffic to a WARC file
| HTTPS proxy that saves traffic to a WARC file
|-
|-
| [https://github.com/internetarchive/warctools/ warc-tools]  
| [https://github.com/internetarchive/warctools warc-tools]  
| MIT License
| MIT License
| Python 2.6
| Python 2.6
| NO TEST SUITE
| NO TEST SUITE
| A readme file
| README
| 4 commiters
| 4 commiters
| warc validator, dump, search, index, convert arc to warc
| warc validator, dump, search, index, convert arc to warc


The previous versions can be found at https://code.google.com/p/warc-tools/ and http://code.hanzoarchives.com/warc-tools .
The previous versions can be found at https://code.google.com/p/warc-tools/ and https://bitbucket.org/hanzo/warc-tools
 
old: http://code.hanzoarchives.com/warc-tools/src/6e1d36297688/hanzo/warcextract.py<br />
new (untested): http://code.hanzoarchives.com/warc-tools/src/fd3b49a7ee22fe4eee0d51dc841af40d4b9d2e1e/warcunpack_ia.py?at=default
|-
|-
| [https://github.com/alard/warc-proxy WARC viewer]  
| [https://github.com/alard/warc-proxy WARC viewer]  
Line 75: Line 79:
| Python
| Python
| NO TEST SUITE
| NO TEST SUITE
| A readme file
| README
| 1 author
| 1 author
| WARC viewer for browsing the contents of a WARC file.
| WARC viewer for browsing the contents of a WARC file.
Line 83: Line 87:
| Python
| Python
| NO TEST SUITE
| NO TEST SUITE
| A readme file
| README
| 1 author
| 1 author
| Merge many small warcs into a large one
| Merge many small warcs into a large one
Line 91: Line 95:
| [https://github.com/alard/warctozip-service warc to zip]  
| [https://github.com/alard/warctozip-service warc to zip]  
| no license information
| no license information
| python
| Python
| NO TEST SUITE
| NO TEST SUITE
| A readme file
| README
| 1 author
| 1 author
| An HTTP-based warc-to-zip converter
| An HTTP-based warc-to-zip converter
Line 101: Line 105:
| Python 3
| Python 3
| yes
| yes
| A readme file.
| README
| 1 author
| 1 author
| warcat concat, extract, list, pass, split, verify warc files
| warcat concat, extract, list, pass, split, verify warc files
Line 116: Line 120:
| Bash shell scripting
| Bash shell scripting
| NO TEST SUITE
| NO TEST SUITE
| A readme file.
| README
| 1 author
| 1 author
| Generates 50gb warc files from existing warc files
| Generates 50gb warc files from existing warc files
Uploads to archive.org
Uploads to archive.org
|-
|-
| [https://github.com/rajbot/CDX-Writer CDX Writer]  
| [https://github.com/rajbot/CDX-Writer CDX Writer]  
| no license information
| no license information
| python
| Python
| Has a test suite
| Has a test suite
| A readme file.
| README
| 1 author
| 1 author
| Create CDX index files from WARC files.
| Create CDX index files from WARC files.
|-
|-
| [https://webarchive.jira.com/wiki/display/Heritrix/Heritrix Heritrix]  
| [https://webarchive.jira.com/wiki/spaces/Heritrix/overview Heritrix]  
| Apache v2.0
| Apache v2.0
| java
| Java
| Has a test suite
| Has a test suite
| javadoc, website
| javadoc, website
Line 142: Line 145:
| A library for writing Heritrix 3 output directly to Cassandra as records.
| A library for writing Heritrix 3 output directly to Cassandra as records.
|-
|-
| [http://sourceforge.net/projects/deduplicator/ DeDuplicator (Heritrix add-on)]
| [https://landsbokasafn.github.io/DeDuplicator/ DeDuplicator (Heritrix add-on)]
| ? || ? || ? || ? || ?
| GPL v2.1
| Java
| Very few tests
| [https://landsbokasafn.github.io/DeDuplicator/started.html Getting Started] page.
| 1 author
| The DeDuplicator is an add-on module (plug-in) for the web crawler Heritrix. It offers a means to reduce the amount of duplicate data collected in a series of snapshot crawls.
| The DeDuplicator is an add-on module (plug-in) for the web crawler Heritrix. It offers a means to reduce the amount of duplicate data collected in a series of snapshot crawls.
|-
|-
Line 150: Line 157:
| A simple wrapper around the Heritrix 3.x API. Developed in April 2012 against Heritrix 3.1.0 at GWU Libraries in Washington, DC, USA.
| A simple wrapper around the Heritrix 3.x API. Developed in April 2012 against Heritrix 3.1.0 at GWU Libraries in Washington, DC, USA.
|-
|-
| [http://warcreate.com/ Chrome/Chromium plugin WARCreate]  
| [https://warcreate.com/ WARCreate (Chrome/Chromium extension)]
| GPL v3
| MIT
| javascript
| JavaScript
| ???
| ???
| none
| none
| 1 author
| 1 author
| WARCreate is a Google Chrome extension that allows a user to create a Web ARChive (WARC) file from any browseable webpage. [https://github.com/machawk1/warcreate code repo]
| WARCreate is a Google Chrome extension that allows a user to create a WARC file from any browseable webpage. [https://github.com/machawk1/warcreate code repo]
|-
|-
| [https://sbforge.org/display/JWAT/JWAT Java Web Archive Toolkit]  
| [https://sbforge.org/display/JWAT/JWAT Java Web Archive Toolkit]  
Line 168: Line 175:
[https://bitbucket.org/nclarkekb/jwat/overview code repo]
[https://bitbucket.org/nclarkekb/jwat/overview code repo]
|-
|-
| [http://matkelly.com/wail/ WAIL]  
| [https://machawk1.github.io/wail/ Web Archiving Integration Layer (WAIL)]  
| CC-BY-SA
| MIT
| Python, JS
| Python
| ???
| ???
| Online
| Online
| 1
| 1 author
| Web Archiving Integration Layer (WAIL) is a graphical user interface (GUI) atop multiple web archiving tools intended to be used as an easy way for anyone to preserve and replay web pages.
| Web Archiving Integration Layer (WAIL) is a graphical user interface (GUI) atop multiple web archiving tools intended to be used as an easy way for anyone to preserve and replay web pages.
Tools included and accessible through the GUI are Heritrix 3.1.2, Wayback 1.7, and warc-proxy. Support packages include Apache Tomcat, phantomjs and pyinstaller.
Tools included and accessible through the GUI are Heritrix 3.2.0 and OpenWayback 2.4.0.  


[https://github.com/machawk1/wail code repo]
[https://github.com/machawk1/wail code repo]
Line 185: Line 192:
| ?
| ?
| 1 author
| 1 author
CDX support
|CDX support
Written by odie5533 which frequents #archiveteam, as another independant WARC library for Python.
Another independent WARC library for Python.
|-
|-
| [https://github.com/chfoo/wpull Wpull]  
| [https://github.com/ArchiveTeam/wpull Wpull]  
| GPL version 3
| GPL version 3
| Python 3
| Python 3
| many unit tests (Travis CI registered), simple experimental fuzzer
| many unit tests (Travis CI registered), simple experimental fuzzer
| a quick start readme, brief usage overview, good docstrings coverage
| a quick start README, brief usage overview, good docstrings coverage
| 1 core author
| 1 core author
| Wget-compatible web downloader.
| Wget-compatible web downloader.
Beta quality. Lua/Python scripting. PhantomJS (experimental). Used by [[ArchiveBot]].
Beta quality. Lua/Python scripting. PhantomJS (experimental). Used by [[ArchiveBot]].
|-
|-
| [https://github.com/ludios/grab-site grab-site]  
| [https://github.com/ArchiveTeam/grab-site grab-site]  
| MIT
| MIT
| Python 3
| Python 3
| no
| no
| readme
| README
| 1 core author
| 1 core author
| wpull launcher with the dashboard and ignore patterns from ArchiveBot
| wpull launcher with the dashboard and ignore patterns from ArchiveBot
Line 210: Line 216:
| Python 2
| Python 2
| yes
| yes
| readme and wiki
| README and wiki
| 1 core author
| 1 core author
| A full-fledged Python reimplementation of Wayback Machine web archive replay capabilities. Also provides a live rewriting proxy.
| A full-fledged Python reimplementation of Wayback Machine web archive replay capabilities. Also provides a live rewriting proxy.
|-
| [https://github.com/helgeho/ArchiveSpark ArchiveSpark]
| MIT License
| Scala
| ?
| ?
| 2 authors
| Apache Spark framework that facilitates access to Web Archives
|-
|-
| [https://github.com/ikreymer/pywb-webrecorder pywb-webrecorder]
| [https://github.com/webrecorder/webrecorder-player Webrecorder Player]
| MIT
| Apache License 2.0
| Python 2
| JavaScript
| no
| ?
| readme
| ?
| 1 core author
| ?
| An experimental/demo integration of pywb + warcprox to allow live recording to WARC. Allows instant replay of recorded content from WARC.
| Desktop app for viewing high-fidelity web archives (WARC, HAR and ARC) on a local machine, no internet connection required. Particularly useful for social media, dynamic content. Supports OSX, Windows and Linux (experimental). Related to https://webrecorder.io/
|-
| [https://github.com/webrecorder/warcio warcio]
| Apache 2.0
| Python 2.7+/3.3+
| yes
| README
| 7 contributors
| WARC writer library
|-
| [https://github.com/internetarchive/warcprox warcprox]
| GPL v2+ || Python 3.4+
| yes
| README
| 1 core author, 11 contributors
| MITM proxy for capturing to WARC. See also [https://github.com/internetarchive/brozzler brozzler], a crawler based on headless Chromium and warcprox.
|-
|-
| [https://github.com/ikreymer/webarchiveplayer webarchiveplayer]
! Name
| GPL version 3
! License
| Python 2
! Language
| not yet, though most testable functionality in pywb
! Testing
| readme
! Documentation
| 1 core author
! Author count
| Point-and-click wrapper for Windows and OS X for browsing WARC files. Shows a basic file open dialog to select a WARC(s), then
! Description
starts a server and opens a browser. Also determines HTML pages within a WARC. Built on top of pywb. In beta at the moment (early 2015).
|}
|}


== Deprecated ==
== Deprecated ==
* https://code.google.com/p/warc-tools/ - Old, discontinued shit
* https://github.com/internetarchive/archive-commons - split into 2 new repos: ia-web-commons & ia-hadoop-tools
* https://github.com/internetarchive/archive-commons - split into 2 new repos: ia-web-commons & ia-hadoop-tools
* https://github.com/ikreymer/pywb-webrecorder
* https://code.google.com/p/warc-tools/
* https://github.com/lintool/warcbase
* [https://github.com/ikreymer/webarchiveplayer WebArchivePlayer]


== The WARC format ==
== The WARC format ==
Line 268: Line 299:
* Field values may contain any UTF-8 character.
* Field values may contain any UTF-8 character.
* The 'encoded-word' mechanism of [RFC2047] may also be used when writing WARC fields and shall also be understood by WARC reading software.
* The 'encoded-word' mechanism of [RFC2047] may also be used when writing WARC fields and shall also be understood by WARC reading software.
==== Defined field names ====
; WARC-Type : ''required'', can be one of 'warcinfo', 'response', 'resource', 'request', 'metadata', 'revisit', 'conversion', or 'continuation'
; WARC-Record-ID : ''required'', unique ID, as a URI
; WARC-Date : ''required''
; Content-Length : ''required''
; Content-Type : mime type
; WARC-Concurrent-To : ''repeatable'', WARC-Record-IDs associated with this one
; WARC-Block-Digest : ''optional'', hash of the whole record
; WARC-Payload-Digest : ''optional'', hash of the just the payload
; WARC-IP-Address : where the record was gotten from
; WARC-Refers-To : previous WARC-Record-ID this relates to
; WARC-Target-URI : the URL asked for
; WARC-Truncated  : why only part of the content was gotten
; WARC-Warcinfo-ID : WARC-Record-ID of the associated high-level metadata record
; WARC-Filename :                ''warcinfo only'', the expected name of the file containing this record
; WARC-Profile :                ''revisit only'', the way revisiting was handled, as a URI
; WARC-Identified-Payload-Type : a independently verified mime type of the payload (i.e. not just what it claims to be)
; WARC-Segment-Origin-ID :      ''continuation only''
; WARC-Segment-Number :
; WARC-Segment-Total-Length :    ''continuation only''


=== WARC content block ===
=== WARC content block ===
Part (zero or more octets) of a WARC record that follows the header and that forms the main body of a
Part (zero or more octets) of a WARC record that follows the header and that forms the main body of a WARC record.
WARC record.
 
== ArchiveBot job output ==
The [[ArchiveBot]] produces three types of files:
; .meta.warc.gz : The log of the job, listing all the files requested and downloaded, as well as any errors.
; .json : Some brief metadata about the job.
; -0000.warc.gz, -0001.warc.gz, ... : The actual requests and responses, in full.


== CDX File Format ==
== CDX File Format ==


* http://archive.org/web/researcher/cdx_legend.php
* https://archive.org/web/researcher/cdx_legend.php
* https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server -- How to query IA's CDX server


Example of generating a list of URLs in a MegaWARC:
Example of generating a list of URLs in a MegaWARC:
  curl -sL 'https://archive.org/download/archiveteam_zapd_20131016071259/zapd_20131016071259.megawarc.warc.os.cdx.gz' \
  curl -sL 'https://archive.org/download/archiveteam_zapd_20131016071259/zapd_20131016071259.megawarc.warc.os.cdx.gz' \
  | gunzip -c | cut -f3 -d' '
  | gunzip -c | cut -f3 -d' '
Example of getting a list of all the URLs in the Wayback Machine with a given prefix:
curl 'https://web.archive.org/cdx/search/cdx?fl=statuscode,timestamp,original&collapse=urlkey&matchType=prefix&url=http://www.conchord.org'


[[Category:Tools]]
[[Category:Tools]]


{{Navigation box}}
{{Navigation box}}

Revision as of 01:49, 5 May 2020

Everything about the WARC format and the tools that support it.

WARC is a file format for accurately storing Web traffic.

Information

Tools

Name License Language Testing Documentation Author count Description
wget v1.14+ GPL v3+ C Has a test suite but does not test any warc functionality Man pages, website, blog posts all over the net 2+ according to the changelog A non-interactive network downloader. wget also generates duplicate record ids in warc files.

More information about flags can be found on the Wget with WARC output page.

InternetArchive's warc python library GPL v2 Python 2 looks to have a test suite - https://github.com/internetarchive/warc/blob/master/warc/tests/test_warc.py README with examples online at https://warc.readthedocs.io/en/latest/ 3 commiters on github library to work with WARC files
WarcMiddleware ISC Python Not enough tests README + Scrapy docs 1 author Mirrors websites and saves the results to a WARC file
WarcProxy ISC Python NO TEST SUITE README 1 author a simple HTTP proxy that saves all HTTP traffic to a file
WarcMITMProxy ISC Python NO TEST SUITE README 1 author HTTPS proxy that saves traffic to a WARC file
warc-tools MIT License Python 2.6 NO TEST SUITE README 4 commiters warc validator, dump, search, index, convert arc to warc

The previous versions can be found at https://code.google.com/p/warc-tools/ and https://bitbucket.org/hanzo/warc-tools

WARC viewer no license information Python NO TEST SUITE README 1 author WARC viewer for browsing the contents of a WARC file.
Megawarc no license information Python NO TEST SUITE README 1 author Merge many small warcs into a large one

Checks if WARC files can be un-gzipped before adding them to the megawarc. Does not check anything else.

warc to zip no license information Python NO TEST SUITE README 1 author An HTTP-based warc-to-zip converter
warcat GPL v3 Python 3 yes README 1 author warcat concat, extract, list, pass, split, verify warc files

Install: pip-3 install warcat
Run: python3 -m warcat verify mysite.warc.gz

https://github.com/internetarchive/ia-web-commons 
https://github.com/internetarchive/ia-hadoop-tools 
Archive Team megawarc factory no license information Bash shell scripting NO TEST SUITE README 1 author Generates 50gb warc files from existing warc files

Uploads to archive.org

CDX Writer no license information Python Has a test suite README 1 author Create CDX index files from WARC files.
Heritrix Apache v2.0 Java Has a test suite javadoc, website many authors Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
Heritrix-Cassandra ? ? ? ? ? A library for writing Heritrix 3 output directly to Cassandra as records.
DeDuplicator (Heritrix add-on) GPL v2.1 Java Very few tests Getting Started page. 1 author The DeDuplicator is an add-on module (plug-in) for the web crawler Heritrix. It offers a means to reduce the amount of duplicate data collected in a series of snapshot crawls.
python-heritrix ? ? ? ? ? A simple wrapper around the Heritrix 3.x API. Developed in April 2012 against Heritrix 3.1.0 at GWU Libraries in Washington, DC, USA.
WARCreate (Chrome/Chromium extension) MIT JavaScript ??? none 1 author WARCreate is a Google Chrome extension that allows a user to create a WARC file from any browseable webpage. code repo
Java Web Archive Toolkit Apache 2.0 Java Partial Test Suite (check coverage profile) Online 1 author jwattools arc2warc, cdx, compress, decompress, extract, interval, pathindex, test, unpack

code repo

Web Archiving Integration Layer (WAIL) MIT Python ??? Online 1 author Web Archiving Integration Layer (WAIL) is a graphical user interface (GUI) atop multiple web archiving tools intended to be used as an easy way for anyone to preserve and replay web pages.

Tools included and accessible through the GUI are Heritrix 3.2.0 and OpenWayback 2.4.0.

code repo

pylibwarc ISC License Python ? ? 1 author CDX support

Another independent WARC library for Python.

Wpull GPL version 3 Python 3 many unit tests (Travis CI registered), simple experimental fuzzer a quick start README, brief usage overview, good docstrings coverage 1 core author Wget-compatible web downloader.

Beta quality. Lua/Python scripting. PhantomJS (experimental). Used by ArchiveBot.

grab-site MIT Python 3 no README 1 core author wpull launcher with the dashboard and ignore patterns from ArchiveBot
pywb GPL version 3 Python 2 yes README and wiki 1 core author A full-fledged Python reimplementation of Wayback Machine web archive replay capabilities. Also provides a live rewriting proxy.
ArchiveSpark MIT License Scala ? ? 2 authors Apache Spark framework that facilitates access to Web Archives
Webrecorder Player Apache License 2.0 JavaScript ? ? ? Desktop app for viewing high-fidelity web archives (WARC, HAR and ARC) on a local machine, no internet connection required. Particularly useful for social media, dynamic content. Supports OSX, Windows and Linux (experimental). Related to https://webrecorder.io/
warcio Apache 2.0 Python 2.7+/3.3+ yes README 7 contributors WARC writer library
warcprox GPL v2+ Python 3.4+ yes README 1 core author, 11 contributors MITM proxy for capturing to WARC. See also brozzler, a crawler based on headless Chromium and warcprox.
Name License Language Testing Documentation Author count Description

Deprecated

The WARC format

A .warc file is usually a group of one or more WARC records. The first record usually describes the records to follow.

Compression is optional. If used, each record is compressed via gzip. A gzip file supports multiple "members"; compressed warcs end in .warc.gz. According to the guidelines, WARC files should top out at 1 gb.

WARC record

  • header
  • content block
  • two newlines

WARC record header

The beginning of a WARC record, consisting of one first line declaring the record to be in the WARC format with a given version number, followed by lines of named fields up to a blank line. The WARC record header format largely follows the tradition of HTTP/1.1 [RFC2616] and [RFC2822] headers, with one major exception, allowing UTF-8 [RFC3629].


Example of a 'request' record header:

 WARC/1.0
 WARC-Type: request
 WARC-Target-URI: http://xbox.gamespy.com/
 Content-Type: application/http;msgtype=request
 WARC-Date: 2013-04-02T16:12:40Z
 WARC-Record-ID: <urn:uuid:08d9edb9-0ab8-4352-ba56-6cbbd590f34f>
 WARC-IP-Address: 213.248.112.146
 WARC-Warcinfo-ID: <urn:uuid:2b6ad3f1-efab-4e37-8faa-fc8ad112692f>
 WARC-Block-Digest: sha1:T6PJSZTTP7HBNA6OFZACXAFK25GGLVT4
 Content-Length: 150

WARC named fields

  • A set of elements consisting of a name, a colon, and a value, with long values continued on indented lines.
  • Named fields may appear in any order.
  • Field values may contain any UTF-8 character.
  • The 'encoded-word' mechanism of [RFC2047] may also be used when writing WARC fields and shall also be understood by WARC reading software.

Defined field names

WARC-Type
required, can be one of 'warcinfo', 'response', 'resource', 'request', 'metadata', 'revisit', 'conversion', or 'continuation'
WARC-Record-ID
required, unique ID, as a URI
WARC-Date
required
Content-Length
required
Content-Type
mime type
WARC-Concurrent-To
repeatable, WARC-Record-IDs associated with this one
WARC-Block-Digest
optional, hash of the whole record
WARC-Payload-Digest
optional, hash of the just the payload
WARC-IP-Address
where the record was gotten from
WARC-Refers-To
previous WARC-Record-ID this relates to
WARC-Target-URI
the URL asked for
WARC-Truncated
why only part of the content was gotten
WARC-Warcinfo-ID
WARC-Record-ID of the associated high-level metadata record
WARC-Filename
warcinfo only, the expected name of the file containing this record
WARC-Profile
revisit only, the way revisiting was handled, as a URI
WARC-Identified-Payload-Type
a independently verified mime type of the payload (i.e. not just what it claims to be)
WARC-Segment-Origin-ID
continuation only
WARC-Segment-Number
WARC-Segment-Total-Length
continuation only

WARC content block

Part (zero or more octets) of a WARC record that follows the header and that forms the main body of a WARC record.

ArchiveBot job output

The ArchiveBot produces three types of files:

.meta.warc.gz
The log of the job, listing all the files requested and downloaded, as well as any errors.
.json
Some brief metadata about the job.
-0000.warc.gz, -0001.warc.gz, ...
The actual requests and responses, in full.

CDX File Format

Example of generating a list of URLs in a MegaWARC:

curl -sL 'https://archive.org/download/archiveteam_zapd_20131016071259/zapd_20131016071259.megawarc.warc.os.cdx.gz' \
| gunzip -c | cut -f3 -d' '

Example of getting a list of all the URLs in the Wayback Machine with a given prefix:

curl 'https://web.archive.org/cdx/search/cdx?fl=statuscode,timestamp,original&collapse=urlkey&matchType=prefix&url=http://www.conchord.org'