Difference between revisions of "Working with ARCHIVE.ORG"

From Archiveteam
Jump to navigation Jump to search
m (add wikilink + grammar & dash)
(update, per joepie91 whining)
Line 1: Line 1:
[[Image:Archivesneedlove.jpg.jpg]]
[[Image:Archivesneedlove.jpg.jpg]]


The [[Internet Archive]] has enormous resources at its disposal, and shares many parallel goals with Archive Team. There are some advantages to making as much of Archive Team's data saves available to ARCHIVE.ORG, especially as regards the [http://wayback.archive.org/web/ Wayback Machine], a mechanism to be able to browse older web materials going back years. Where possible, Archive Team works with ARCHIVE.ORG, and get the love going.


The [[Internet Archive]] has enormous resources at its disposal, and shares many parallel goals with Archive Team. There are some advantages to making as much of Archive Team's data saves available to ARCHIVE.ORG, especially as regards the [http://wayback.archive.org/web/ Wayback Machine], a mechanism to be able to browse older web materials going back years. Where possible, Archive Team should try to work with ARCHIVE.ORG, and get the love going.
__TOC__
__TOC__
== Headers and Logs ==
== Headers and Logs ==


The most striking difference between Archive Team and ARCHIVE.ORG is that while Archiveteam traditionally considers header and logging information to be a nice hat trick if you have the time, ARCHIVE.ORG absolutely needs it to import into their Wayback Machine. With WGET, the best way to do this is to save off the log files, and use ''--save-headers'' to include the headers in the file.
The most striking difference between Archive Team and ARCHIVE.ORG is that while Archiveteam traditionally considers header and logging information to be a nice hat trick if you have the time, ARCHIVE.ORG absolutely needs it to import into their Wayback Machine.


(More research needs to be done to ensure these options are in scripts.)
How? See [[The WARC Ecosystem]].
 
Unfortunately, this produces files that are not, initially, usable for just dumping into a new location, so there is a chance either two copies should be made, or a script written that strips the headers out of the files for later use.
 
== Stripping the Headers ==
As an example, let's download this page:
<pre>wget -O headers.raw.html --save-headers "http://www.archiveteam.org/index.php?title=Working_with_ARCHIVE.ORG"</pre>
 
If we want to strip the headers, then we run:
<pre>cat headers.raw.html | perl -ne 'unless ($out) { $out=1 if $_ eq "\r\n"; next } print' > noheaders.raw.html</pre>


== The Archive.org "Archiveteam" Collection ==
== The Archive.org "Archiveteam" Collection ==


Archive.org currently has an [http://www.archive.org/details/archiveteam Archive Team Collection], which consists of site-grabs, older archives, and various rips/mass downloads from websites and Archive Team activities. Right now, that collection is primarily administered by Jason Scott &mdash; sending him e-mail at jason@textfiles.com with something you think that collection should have is probably the best way to go about things.
Archive.org currently has an [http://www.archive.org/details/archiveteam Archive Team Collection], which consists of site-grabs, older archives, and various rips/mass downloads from websites and Archive Team activities. Right now, that collection is primarily administered by Jason Scott &mdash; sending him e-mail at jason@textfiles.com with something you think that collection should have is probably the best way to go about things.
[[Category:Tools]]

Revision as of 08:11, 9 December 2014

Archivesneedlove.jpg.jpg

The Internet Archive has enormous resources at its disposal, and shares many parallel goals with Archive Team. There are some advantages to making as much of Archive Team's data saves available to ARCHIVE.ORG, especially as regards the Wayback Machine, a mechanism to be able to browse older web materials going back years. Where possible, Archive Team works with ARCHIVE.ORG, and get the love going.

Headers and Logs

The most striking difference between Archive Team and ARCHIVE.ORG is that while Archiveteam traditionally considers header and logging information to be a nice hat trick if you have the time, ARCHIVE.ORG absolutely needs it to import into their Wayback Machine.

How? See The WARC Ecosystem.

The Archive.org "Archiveteam" Collection

Archive.org currently has an Archive Team Collection, which consists of site-grabs, older archives, and various rips/mass downloads from websites and Archive Team activities. Right now, that collection is primarily administered by Jason Scott — sending him e-mail at jason@textfiles.com with something you think that collection should have is probably the best way to go about things.

Retrieved from "https://wiki.archiveteam.org/index.php?title=Working_with_ARCHIVE.ORG&oldid=20978"