Heritrix
Jump to navigation
Jump to search
Heritrix is a WARC-writing web crawler created by the Internet Archive. It is written in Java and can be found on the IA's GitHub page.
Sample WARC segments in text format are available for inspection at https://gist.github.com/ibnesayeed/ddb17bb2958fcff37f3dafc9a3a86793, though note it was made in 2018-03-16.