The WARC Ecosystem

Everything about the WARC format and the tools that support it.

WARC is a file format for accurately storing Web traffic.

Information

• wikipedia:Web_ARChive
• https://webarchive.jira.com/wiki/pages/viewpage.action?pageId=4817^{[IA•Wcite•.today•MemWeb]} - Contains examples of WARC records
• The WARC File Format (ISO 28500) - Information, Maintenance, Drafts^{[IA•Wcite•.today•MemWeb]}
• http://archive-access.sourceforge.net/warc/^{[IA•Wcite•.today•MemWeb]} - WARC ISO docs
• https://www.loc.gov/preservation/digital/formats/fdd/fdd000236.shtml^{[IA•Wcite•.today•MemWeb]}
• https://netpreserve.org/resources/warc-implementation-guidelines-v1/^{[IA•Wcite•.today•MemWeb]}
• https://netpreserve.org/resources/WARC_Guidelines_v1.pdf^{[IA•Wcite•.today•MemWeb]}
• https://commoncrawl.org/2014/04/navigating-the-warc-file-format/^{[IA•Wcite•.today•MemWeb]}
• https://www.taricorp.net/2016/web-history-warc^{[IA•Wcite•.today•MemWeb]}
• WARC/1.0 specification^{[IA•Wcite•.today•MemWeb]}
• WARC/1.1 specification^{[IA•Wcite•.today•MemWeb]}
• GitHub repository coordinating the specification^{[IA•Wcite•.today•MemWeb]}

Tools

Name	License	Language	Testing	Documentation	Author count	Description
wget v1.14+	GPL v3+	C	Has a test suite but does not test any warc functionality	Man pages, website, blog posts all over the net	2+ according to the changelog	A non-interactive network downloader. wget also generates duplicate record ids in warc files. More information about flags can be found on the Wget with WARC output page.
InternetArchive's warc python library	GPL v2	Python 2	looks to have a test suite - https://github.com/internetarchive/warc/blob/master/warc/tests/test_warc.py	README with examples online at https://warc.readthedocs.io/en/latest/	3 commiters on github	library to work with WARC files
WarcMiddleware	ISC	Python	Not enough tests	README + Scrapy docs	1 author	Mirrors websites and saves the results to a WARC file
WarcProxy	ISC	Python	NO TEST SUITE	README	1 author	a simple HTTP proxy that saves all HTTP traffic to a file
WarcMITMProxy	ISC	Python	NO TEST SUITE	README	1 author	HTTPS proxy that saves traffic to a WARC file
warc-tools	MIT License	Python 2.6	NO TEST SUITE	README	4 commiters	warc validator, dump, search, index, convert arc to warc The previous versions can be found at https://code.google.com/p/warc-tools/ and https://bitbucket.org/hanzo/warc-tools
WARC viewer	no license information	Python	NO TEST SUITE	README	1 author	WARC viewer for browsing the contents of a WARC file.
Megawarc	no license information	Python	NO TEST SUITE	README	1 author	Merge many small warcs into a large one Checks if WARC files can be un-gzipped before adding them to the megawarc. Does not check anything else.
warc to zip	no license information	Python	NO TEST SUITE	README	1 author	An HTTP-based warc-to-zip converter
warcat	GPL v3	Python 3	yes	README	1 author	warcat concat, extract, list, pass, split, verify warc files Install: pip-3 install warcat Run: python3 -m warcat verify mysite.warc.gz https://github.com/internetarchive/ia-web-commons https://github.com/internetarchive/ia-hadoop-tools
Archive Team megawarc factory	no license information	Bash shell scripting	NO TEST SUITE	README	1 author	Generates 50gb warc files from existing warc files Uploads to archive.org
CDX Writer	AGPL v3	Python	Has a test suite	README	1 author	Create CDX index files from WARC files.
Heritrix	Apache v2.0	Java	Has a test suite	javadoc, website	many authors	Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
Heritrix-Cassandra	LGPL v2.1	?	?	?	?	A library for writing Heritrix 3 output directly to Cassandra as records.
DeDuplicator (Heritrix add-on)	LGPL v2.1	Java	Very few tests	Getting Started page.	1 author	The DeDuplicator is an add-on module (plug-in) for the web crawler Heritrix. It offers a means to reduce the amount of duplicate data collected in a series of snapshot crawls.
python-heritrix	?	?	?	?	?	A simple wrapper around the Heritrix 3.x API. Developed in April 2012 against Heritrix 3.1.0 at GWU Libraries in Washington, DC, USA.
WARCreate (Chrome/Chromium extension)	MIT	JavaScript	???	none	1 author	WARCreate is a Google Chrome extension that allows a user to create a WARC file from any browseable webpage. code repo
Java Web Archive Toolkit	Apache 2.0	Java	Partial Test Suite (check coverage profile)	Online	1 author	jwattools arc2warc, cdx, compress, decompress, extract, interval, pathindex, test, unpack code repo
Web Archiving Integration Layer (WAIL)	MIT	Python	???	Online	1 author	Web Archiving Integration Layer (WAIL) is a graphical user interface (GUI) atop multiple web archiving tools intended to be used as an easy way for anyone to preserve and replay web pages. Tools included and accessible through the GUI are Heritrix 3.2.0 and OpenWayback 2.4.0. code repo
pylibwarc	ISC License	Python	?	?	1 author	CDX support Another independent WARC library for Python.
Wpull	GPL v3	Python 3	many unit tests (Travis CI registered), simple experimental fuzzer	a quick start README, brief usage overview, good docstrings coverage	1 core author	Wget-compatible web downloader. Beta quality. Lua/Python scripting. PhantomJS (experimental). Used by ArchiveBot.
grab-site	MIT	Python 3	no	README	1 core author	wpull launcher with the dashboard and ignore patterns from ArchiveBot
pywb	GPL v3	Python 2	yes	README and wiki	1 core author	A full-fledged Python reimplementation of Wayback Machine web archive replay capabilities. Also provides a live rewriting proxy.
ArchiveSpark	MIT License	Scala	?	?	2 authors	Apache Spark framework that facilitates access to Web Archives
Webrecorder Player	Apache License 2.0	JavaScript	?	?	?	Desktop app for viewing high-fidelity web archives (WARC, HAR and ARC) on a local machine, no internet connection required. Particularly useful for social media, dynamic content. Supports OSX, Windows and Linux (experimental). Related to https://webrecorder.io/
warcio	Apache 2.0	Python 2.7+/3.4+	yes	README	14 contributors	WARC writer library
warcprox	GPL v2+	Python 3.4+	yes	README	1 core author, 14 contributors	MITM proxy for capturing to WARC. See also brozzler, a crawler based on headless Chromium and warcprox.
Name	License	Language	Testing	Documentation	Author count	Description

Deprecated

• https://github.com/internetarchive/archive-commons - split into 2 new repos: ia-web-commons & ia-hadoop-tools
• https://github.com/ikreymer/pywb-webrecorder
• https://code.google.com/p/warc-tools/
• https://github.com/lintool/warcbase
• WebArchivePlayer

The WARC format

A .warc file is usually a group of one or more WARC records. The first record usually describes the records to follow.

Compression is optional. If used, each record is compressed via gzip. A gzip file supports multiple "members"; compressed warcs end in .warc.gz. According to the guidelines, WARC files should top out at 1 gb.

WARC record

• header
• content block
• two newlines

WARC record header

The beginning of a WARC record, consisting of one first line declaring the record to be in the WARC format with a given version number, followed by lines of named fields up to a blank line. The WARC record header format largely follows the tradition of HTTP/1.1 [RFC2616] and [RFC2822] headers, with one major exception, allowing UTF-8 [RFC3629].

Example of a 'request' record header:

 WARC/1.0
 WARC-Type: request
 WARC-Target-URI: http://xbox.gamespy.com/
 Content-Type: application/http;msgtype=request
 WARC-Date: 2013-04-02T16:12:40Z
 WARC-Record-ID: <urn:uuid:08d9edb9-0ab8-4352-ba56-6cbbd590f34f>
 WARC-IP-Address: 213.248.112.146
 WARC-Warcinfo-ID: <urn:uuid:2b6ad3f1-efab-4e37-8faa-fc8ad112692f>
 WARC-Block-Digest: sha1:T6PJSZTTP7HBNA6OFZACXAFK25GGLVT4
 Content-Length: 150

WARC named fields

• A set of elements consisting of a name, a colon, and a value, with long values continued on indented lines.
• Named fields may appear in any order.
• Field values may contain any UTF-8 character.
• The 'encoded-word' mechanism of [RFC2047] may also be used when writing WARC fields and shall also be understood by WARC reading software.

Defined field names

WARC-Type

required, can be one of 'warcinfo', 'response', 'resource', 'request', 'metadata', 'revisit', 'conversion', or 'continuation'

WARC-Record-ID

required, unique ID, as a URI

WARC-Date

required

Content-Length

required

Content-Type

mime type

WARC-Concurrent-To

repeatable, WARC-Record-IDs associated with this one

WARC-Block-Digest

optional, hash of the whole record

WARC-Payload-Digest

optional, hash of the just the payload

WARC-IP-Address

where the record was gotten from

WARC-Refers-To

previous WARC-Record-ID this relates to

WARC-Target-URI

the URL asked for

WARC-Truncated

why only part of the content was gotten

WARC-Warcinfo-ID

WARC-Record-ID of the associated high-level metadata record

WARC-Filename

warcinfo only, the expected name of the file containing this record

WARC-Profile

revisit only, the way revisiting was handled, as a URI

WARC-Identified-Payload-Type

a independently verified mime type of the payload (i.e. not just what it claims to be)

WARC-Segment-Origin-ID

continuation only

WARC-Segment-Number

WARC-Segment-Total-Length

continuation only

WARC content block

Part (zero or more octets) of a WARC record that follows the header and that forms the main body of a WARC record.

ArchiveBot job output

The ArchiveBot produces three types of files:

.meta.warc.gz

The log of the job, listing all the files requested and downloaded, as well as any errors.

.json

Some brief metadata about the job.

-0000.warc.gz, -0001.warc.gz, ...

The actual requests and responses, in full.

CDX File Format

• https://archive.org/web/researcher/cdx_legend.php
• https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server -- How to query IA's CDX server

Example of generating a list of URLs in a MegaWARC:

curl -sL 'https://archive.org/download/archiveteam_zapd_20131016071259/zapd_20131016071259.megawarc.warc.os.cdx.gz' \
| gunzip -c | cut -f3 -d' '

Example of getting a list of all the URLs in the Wayback Machine with a given prefix:

curl 'https://web.archive.org/cdx/search/cdx?fl=statuscode,timestamp,original&collapse=urlkey&matchType=prefix&url=http://www.conchord.org'