Difference between revisions of "Internet Archive Census"
(Created page with "The '''Internet Archive Census''' is an attempt to count and indicate the files available on the Internet Archive, focusing on downloadable, public-facing files. The purpose o...") |
|||
| Line 7: | Line 7: | ||
The Census was called for as a stepping stone in the [[INTERNETARCHIVE.BAK]] project, an experiment and project to have Archive Team back up the Internet Archive. While officially, the Internet Archive has 21 petabytes of information in its data stores (as of March 2015), some of that data is related to system overhead, or are stream-only/not available. By having a full run-through of the entire collection of items at the Archive, the next phases of the INTERNETARCHIVE.BAK experiment (testing methodologies) can move forward. | The Census was called for as a stepping stone in the [[INTERNETARCHIVE.BAK]] project, an experiment and project to have Archive Team back up the Internet Archive. While officially, the Internet Archive has 21 petabytes of information in its data stores (as of March 2015), some of that data is related to system overhead, or are stream-only/not available. By having a full run-through of the entire collection of items at the Archive, the next phases of the INTERNETARCHIVE.BAK experiment (testing methodologies) can move forward. | ||
The data is also useful for talking about what the Internet Archive does, and what kinds of items are in the stacks - collections can be found with very large or manageable amounts of data, and audiences/researchers outside the backup experiment can do their own sets of data access and acquisition. | The data is also useful for talking about what the Internet Archive does, and what kinds of items are in the stacks - collections can be found with very large or manageable amounts of data, and audiences/researchers outside the backup experiment can do their own sets of data access and acquisition. Search engines can be experimented with, as well as data visualization. | ||
== Contents of the Census == | == Contents of the Census == | ||
Revision as of 20:14, 11 March 2015
The Internet Archive Census is an attempt to count and indicate the files available on the Internet Archive, focusing on downloadable, public-facing files. The purpose of this project is multi-fold, including determination of sizes of various collections, and determining priorities in backing up portions of the Internet Archive's data stores.
The first Census was conducted in March of 2015. Its results are on the Archive at https://archive.org/details/ia-bak-census_20150304.
Purpose of the Census
The Census was called for as a stepping stone in the INTERNETARCHIVE.BAK project, an experiment and project to have Archive Team back up the Internet Archive. While officially, the Internet Archive has 21 petabytes of information in its data stores (as of March 2015), some of that data is related to system overhead, or are stream-only/not available. By having a full run-through of the entire collection of items at the Archive, the next phases of the INTERNETARCHIVE.BAK experiment (testing methodologies) can move forward.
The data is also useful for talking about what the Internet Archive does, and what kinds of items are in the stacks - collections can be found with very large or manageable amounts of data, and audiences/researchers outside the backup experiment can do their own sets of data access and acquisition. Search engines can be experimented with, as well as data visualization.
Contents of the Census
The Census is a very large collection of JSON-formatted tables, returned by the use of the ia-mine utility by Jake Johnson of the Internet archive. Like all such projects, the data should not be considered perfect, although a large percentage should accurately reflect the site. As there is only one census so far, there is no comparable data in terms of growth or file change. (There are reports of total files or other activity, but not to the level of the JSON format material the Census provides).
Some Relevant Information from the Census
Based on the output of the Census:
- The size of the listed data is 14.23 petabytes.
- The census only contains "original" data, not derivations created by the system. (For example, if a .AVI file is uploaded, the census only counts the .AVI, and not a .MP4 or .GIF derived from the original file).