Difference between revisions of "Software"

From Archiveteam
Jump to navigation Jump to search
(adding snscrape, fixing dead links, moved freeyourstuff.cc to Hosted tools, consolidating WARC specific software onto The WARC Ecosystem)
(proposals)
Line 33: Line 33:


* [http://www.shlock.co.uk/Utils/OmniFlop/OmniFlop.htm OmniFlop]
* [http://www.shlock.co.uk/Utils/OmniFlop/OmniFlop.htm OmniFlop]
== Proposed ==
* [https://solidproject.org/ Solid project] attempts to make data portability a reality
* [https://datatransferproject.dev/ Data transfer project] is a (promise of) a quick implementation of [[wikipedia:GDPR|GDPR]] data portability by the [[wikipedia:GAFA|GAFA]] + Twitter


== Web scraping ==
== Web scraping ==

Revision as of 05:20, 4 December 2019

WARC Tools

The WARC Ecosystem has information on tools to create, read and process WARC files.

General Tools

  • GNU WGET
    • Backing up a Wordpress site: "wget --no-parent --no-clobber --html-extension --recursive --convert-links --page-requisites --user=<username> --password=<password> <path>"
  • cURL
  • HTTrack - HTTrack options
  • Pavuk -- a bit flaky, but very flexible
  • Warrick - Tool to recover lost websites using various online archives and caches.
  • Beautiful Soup - Python library for web scraping
  • Scrapy - Fast python library for web scraping
  • snscrape - Tool to scrape social networking services.
  • Splinter - Web app acceptance testing library for Python -- could be used along with a scraping lib to extract data from hard-to-reach places
  • WiLiSe WikiLink Search - Python script to get links to specific pages of a site through the search in a Wiki (MediaWiki-type) has the api.php accessible or extension LinkSearch enabled (the project is still very immature and at the moment the code is only available in this SVN repository).
  • Mobile Phone Applications -- some notes on preserving old versions of mobile apps

Hosted tools

  • Pinboard is a convenient social bookmarking service that will archive copies of all your bookmarks for online viewing. The catch is that it costs $11/year, or $25/year if you want the archival feature and you can only download archives of your 25 most recent bookmarks in a particular category. This may pose problems if you ever need to get your data out in a hurry.
  • freeyourstuff.cc -- Extensible open-source (source) Chrome plugin allowing users to export their own content (reviews, posts, etc.). Exports to JSON format, optionally publish to freeyourstuff.cc & mirrors under Creative Commons CC0 license. Supports Yelp, IMDB, TripAdvisor, Amazon, GoodReads, and Quora as of July 2019.

Site-Specific

Format Specific

Proposed

Web scraping

Why Back Up?SoftwareFormats