Difference between revisions of "Twitch.tv/Vinesauce"

From Archiveteam
Jump to navigation Jump to search
m (typos fixed: in it's → in its, youtube → YouTube)
(93 intermediate revisions by 5 users not shown)
Line 1: Line 1:
As you might all know already, Twitch will be deleting past streams, so, at the Twitch's new VOD policies thread, me and some of the members decided to save past streams. Even if it's not a bad idea, to have streams downloaded multiple times, it might be a good idea, to organize this, not with everyone, but at least with two-three people to cover these downloading as wide a s possible and save these streams as a nice memory.
{{Infobox project
| title = Twitch.tv - Vinesauce Streams
| URL = http://twitch.tv/vinesauce/profile
| image = Vinesauce_com.jpg
| logo = Vinesauce_Logo.png
| project_status = '''Deadline: 2014-08-27'''
| archiving_status = '''??? / 1714 Streams'''
| source = [http://vinesauce.com/vinetalk/index.php?topic=4321.0 Vinesauce Forums - Archiving Streams]
| irc = none
}}


== Checklist ==
'' '''Vinesauce''' is a team of streamers that play a variety of games, from retro to new; maybe some you've never even heard of.''


Vinesauce currently has 1714 videos in it's VOD backlog.
The Vinesauce team has used Twitch.tv since 2013, when they moved over from Livestream. Now, Twitch.tv gives a deadline of three weeks, starting from August 6th, 2014, before they will delete recorded broadcasts older than 14 days.  


The user Null has made an [http://vinesauce.com/vinetalk/index.php?topic=4316.msg81601#msg81601 HTML List] of all the links to download.
Most Vinesauce livestreams are not archived elsewhere, have no official highlights (or unofficial ones), and will be irrevocably lost upon deletion.


There will be an actual wiki-editable checklist, so we can see who is downloading what chunk, and who has finished.
In response, the Vinesauce community has banded together to archive almost 1,714 streams, totaling a minimum of 500GB. You can help us out by marking a stream for archival on The Checklist, and downloading them using the method described below.


== Standard Folder Name Format ==
''All Vinesauce streams collected will be uploaded to the [http://blogs.loc.gov/digitalpreservation/2014/02/saving-digital-mementos-from-virtual-worlds/ Internet Archive's "Archiving Virtual Worlds" Collection], serving as a major artifact of contemporary Internet Folklife.''


Each Twitch.tv stream is split into multiple 30-minute parts.
== The Process ==


After downloading all the parts from a single stream, group them together into a folder. The folder name should follow this format:
A basic overview of how the project will work:


<code>YYYY-MM-DD [streamer] title</code>
# Download as many streams as you can to your hard drives.
# When they're ready to be shared, upload them to [https://mega.co.nz Mega.co.nz] or the [http://vinesauce.com/vinetalk/index.php?topic=4321.msg81998#msg81998 /r/Vinesauce FTP Server].
# Put the download link for each stream on [[Twitch.tv/Vinesauce#The Checklist|The Checklist]].
# Once the three week deadline strikes, we will download the streams from the links, and upload them to the [http://blogs.loc.gov/digitalpreservation/2014/02/saving-digital-mementos-from-virtual-worlds/ Internet Archive's "Archiving Virtual Worlds" Collection] for permanent archival.
# The archived streams will be viewable using [https://archive.org/details/The4chanCup_2013AutumnFriendlies_Day3 the Internet Archive webplayer, like we did for The 4chan Cup.] You can also download the videos in their original format, as necessary.


For example:
== The Procedure ==


<code>2013-02-12 [Joel] Daikatana</code>
This is how a prospective volunteer (i.e. you) can download streams from Twitch.tv on your own.


This will make it easier for an automated script to process the videos.
=== Reserving a Chunk ===


=== Please don't rename or merge the video files! ===
# Create an account on the ArchiveTeam Wiki.
# Go to [[Twitch.tv/Vinesauce#The Checklist|The Checklist]] below and decide on which chunk you want to upload.
# Reserve the chunk: Edit the section, and copy/paste this wiki markup into it (edit accordingly).
#:<pre>
#:: * Name: Vineshroom
#:: * Email: vine@shroom.com
#:: * Status: ''In Progress''
#:: * Upload Location: ''Not uploaded yet''
#::
#:: ----</pre>
 
=== Downloading your Chunk ===
 
==== Automatic Method ====
 
This method uses the newly discovered [https://github.com/astralfoxy/twitch-archive Twitch Archiver app] by astralfoxy. Unfortunately, the app was just recently put together, and demands stream IDs rather than URLs, so a few workarounds have to be made. If the download is interrupted, it will continue where it left off when started with the same parameters.
 
# Install the Copy Selected Links addon for [https://chrome.google.com/webstore/detail/link2clip/fmieebpnfbcjdackmfajcbbknaikebla/related Chrome] (there's a Firefox equivalent, but it is outdated...)
# After you've chosen your chunk, highlight all the stream links in your chunk, right-click, and select '''Copy All Links Selected'''.
# Paste those links into [http://regexr.com/39a5s this Regex Tool], in the '''Text''' box.
#* We've set up the tool to automatically extract the stream IDs from the links.
# Copy the converted stream IDs from the '''Substitution''' section of the Regex Tool to the clipboard.
# Download, install, and run the [https://github.com/astralfoxy/twitch-archive/releases Twitch Archiver].
# Paste the stream IDs into the right textbox of the Twitch Archiver.
# The Twitch Archiver will automatically download every single stream and constituent parts, and shelve them into nice folders.
 
==== Manual Method ====
 
This method is the old "copy links and download parts manually" method.
 
# After you've chosen your chunk, right-click and copy the URL of one stream, and put it into the [http://www.twitchtools.com/video-download.php Twitch Stream Download Tool]. The parts are split into 30-minute segments, download all of them.
#* You might find it useful to use a concurrent download manager, such as DownThemAll or JDownloader (described below)
# After downloading all the parts from a single stream, group them together into a folder. The folder name should follow this format:
#* <code>YYYY-MM-DD [streamer] title</code>
#* e.g. <code>2013-02-12 [Joel] Daikatana</code>
# Repeat these steps until all the streams in your chunk have been downloaded.
 
'''Please don't rename or merge the video files!'''


* Doing so will take too much extra time and effort.  
* Doing so will take too much extra time and effort.  
Line 31: Line 81:
Just leave the videos as is. The folders are enough.
Just leave the videos as is. The folders are enough.


== How to Download Streams ==
=== Uploading your Chunk ===


Here is the basic procedure:
# Once you're done downloading a chunk, go to The Checklist, find your chunk, and edit your reservation to the following:
#:<pre>
#:: * Status: '''Downloaded!''' </pre>
# Put all the stream folders into one big folder, named after your Chunk:
#* e.g. <code>Chunk 2014-10_C22</code>
# Upload that folder to the [http://vinesauce.com/vinetalk/index.php?topic=4321.msg81998#msg81998 /r/Vinesauce FTP Server], or mega.co.nz.
#* For FTP Upload, use [https://filezilla-project.org/ Filezilla].
#* Ignore the existing file structure on the server! Just put the folder under <code>/Vinesauce/Chunks/</code>
# After uploading, go to The Checklist, find your chunk, and edit your reservation to the following:
#:<pre>
#:: * Upload Location: '''Uploaded!''' (insert link here) </pre>
# Download another chunk!


1. Open up [http://twitch.zazi.bar/ Zazibar's Twitch VOD Download List].
== The Tools ==
2. Search for '''Vinesauce'''.
 
3. Click  '''Download''' on whatever stream you want to download.
=== Twitch Mass Download App ===
4. Download all the parts of that stream.
 
** You might find it useful to use a concurrent download manager, such as DownThemAll or JDownloader (described below)
This Mono/.NET application automates the process of downloading a whole list of streams; or even from an entire user.
5. After all the parts are downloaded, put them into a folder. Follow the [[Twitch.tv/Vinesauce#Standard Folder Name Format|Standard Folder Name Format]] explained above.
 
6. Go to the checklist and
Just give it the name of a streamer, or a bunch of stream IDs, and it will start slurping them down.
 
* [https://github.com/astralfoxy/twitch-archive/releases Download]
* [https://github.com/astralfoxy/twitch-archive Source Code]
 
=== Twitch Download Tool ===
 
Just copy the URL of a stream and put it into the '''Twitch Download Tool''' to obtain the links to download the full stream video. It will be divided into 30 minute parts.
 
Link: http://www.twitchtools.com/video-download.php


=== Zazibar's Twitch VOD Download Lister ===
=== Zazibar's Twitch VOD Download Lister ===
Line 48: Line 118:


Link: http://twitch.zazi.bar/
Link: http://twitch.zazi.bar/
How to use:
# Open up [http://twitch.zazi.bar/ Zazibar's Twitch VOD Download List].
# Search for '''Vinesauce'''.
# Click  '''Download''' on whatever stream you want to download.
# Download all the parts of that stream.
#* You might find it useful to use a concurrent download manager, such as DownThemAll or JDownloader (described below)


=== DownThemAll - Addon for faster downloads with Firefox ===
=== DownThemAll - Addon for faster downloads with Firefox ===
Line 53: Line 131:
For faster concurrent downloading, use the Firefox Addon '''DownThemAll'''
For faster concurrent downloading, use the Firefox Addon '''DownThemAll'''


1. Install [https://addons.mozilla.org/en-US/firefox/addon/downthemall/ DownThemAll] on Firefox.
# Install [https://addons.mozilla.org/en-US/firefox/addon/downthemall/ DownThemAll] on Firefox.
2. Open [http://twitch.zazi.bar/ Zazibar's Twitch VOD Download Tool].
# Open [http://twitch.zazi.bar/ Zazibar's Twitch VOD Download Tool].
3. Search for '''Vinesauce''' .
# Search for '''Vinesauce''' .
4. Click  '''Download''' on whatever stream you want to download.
# Click  '''Download''' on whatever stream you want to download.
5. Now, right click, and in the menu click ''downthemall''.
# Now, right click, and in the menu click ''downthemall''.
6. The DTA window will open, then you will e able to select the FLV files all at once to download.
# The DTA window will open, then you will e able to select the FLV files all at once to download.


A piece of advice: I don't think you can change the file names before downloading, so make sure to make a folder that says what they are before you dump them in some big folder.  
A piece of advice: I don't think you can change the file names before downloading, so make sure to make a folder that says what they are before you dump them in some big folder.  
Line 77: Line 155:


I'll slap together a quick and dirty Python slurping script sooner or later.
I'll slap together a quick and dirty Python slurping script sooner or later.
== The Checklist ==
Vinesauce currently has 1714 videos in its VOD backlog.
Join the effort to archive the videos by reserving a chunk to archive now!
Follow [[Twitch.tv/Vinesauce#The Procedure|The Procedure]] to download streams, as explained above.
This comprehensive checklist is based on [http://vinesauce.com/vinetalk/index.php?topic=4316.msg81601#msg81601 Null's HTML list.] However, his list seems to be missing a few videos here and there, so fix as you see fit.
=== Checklists ===
* 2014
** [[Twitch.tv/Vinesauce/August 2014|August]]
** [[Twitch.tv/Vinesauce/July 2014|July]]
** [[Twitch.tv/Vinesauce/June 2014|June]]
** [[Twitch.tv/Vinesauce/May 2014|May]]
** [[Twitch.tv/Vinesauce/April 2014|April]]
** [[Twitch.tv/Vinesauce/March 2014|March]]
** [[Twitch.tv/Vinesauce/February 2014|February]]
** [[Twitch.tv/Vinesauce/January 2014|January]]
* 2013
** [[Twitch.tv/Vinesauce/December 2013|December]]
** [[Twitch.tv/Vinesauce/November 2013|November]]
** [[Twitch.tv/Vinesauce/September 2013|September]]
** [[Twitch.tv/Vinesauce/August 2013|August]]
** [[Twitch.tv/Vinesauce/July 2013|July]]
** [[Twitch.tv/Vinesauce/January to June 2013|January to June]]
* [[Twitch.tv/Vinesauce/2012|2012]]
----
* [[Twitch.tv/Vinesauce/Checklist|Full Checklist of all Vinesauce Streams]] (DO NOT EDIT THIS)
=== Known Collections ===
Some fans have been collecting streams long before this collection was started.
* [[Twitch.tv/Vinesauce/HungarianGameOver|HungarianGameOver]] - Currently has the largest known collection of Vinesauce streams, at 150GB and growing. He goes for the cream of the crop, however, rather than big chunks.
  * Streams he has archived are marked on the checklist with an (HGO).


== External Links ==
== External Links ==


* [http://vinesauce.com/vinetalk/index.php?topic=4321.msg81672 Vinesauce Forum - Effort to save our Stream Videos]
* [http://vinesauce.com/vinetalk/index.php?topic=4321.msg81672 Vinesauce Forum - Effort to save our Stream Videos]

Revision as of 23:20, 4 December 2017

Twitch.tv - Vinesauce Streams
Twitch.tv/Vinesauce logo
Vinesauce com.jpg
URL http://twitch.tv/vinesauce/profile
Status Deadline: 2014-08-27
Archiving status ??? / 1714 Streams
Archiving type Unknown
Project source Vinesauce Forums - Archiving Streams
IRC channel #none (on hackint)

Vinesauce is a team of streamers that play a variety of games, from retro to new; maybe some you've never even heard of.

The Vinesauce team has used Twitch.tv since 2013, when they moved over from Livestream. Now, Twitch.tv gives a deadline of three weeks, starting from August 6th, 2014, before they will delete recorded broadcasts older than 14 days.

Most Vinesauce livestreams are not archived elsewhere, have no official highlights (or unofficial ones), and will be irrevocably lost upon deletion.

In response, the Vinesauce community has banded together to archive almost 1,714 streams, totaling a minimum of 500GB. You can help us out by marking a stream for archival on The Checklist, and downloading them using the method described below.

All Vinesauce streams collected will be uploaded to the Internet Archive's "Archiving Virtual Worlds" Collection, serving as a major artifact of contemporary Internet Folklife.

The Process

A basic overview of how the project will work:

  1. Download as many streams as you can to your hard drives.
  2. When they're ready to be shared, upload them to Mega.co.nz or the /r/Vinesauce FTP Server.
  3. Put the download link for each stream on The Checklist.
  4. Once the three week deadline strikes, we will download the streams from the links, and upload them to the Internet Archive's "Archiving Virtual Worlds" Collection for permanent archival.
  5. The archived streams will be viewable using the Internet Archive webplayer, like we did for The 4chan Cup. You can also download the videos in their original format, as necessary.

The Procedure

This is how a prospective volunteer (i.e. you) can download streams from Twitch.tv on your own.

Reserving a Chunk

  1. Create an account on the ArchiveTeam Wiki.
  2. Go to The Checklist below and decide on which chunk you want to upload.
  3. Reserve the chunk: Edit the section, and copy/paste this wiki markup into it (edit accordingly).
    * Name: Vineshroom
    * Email: vine@shroom.com
    * Status: ''In Progress''
    * Upload Location: ''Not uploaded yet''
    ----

Downloading your Chunk

Automatic Method

This method uses the newly discovered Twitch Archiver app by astralfoxy. Unfortunately, the app was just recently put together, and demands stream IDs rather than URLs, so a few workarounds have to be made. If the download is interrupted, it will continue where it left off when started with the same parameters.

  1. Install the Copy Selected Links addon for Chrome (there's a Firefox equivalent, but it is outdated...)
  2. After you've chosen your chunk, highlight all the stream links in your chunk, right-click, and select Copy All Links Selected.
  3. Paste those links into this Regex Tool, in the Text box.
    • We've set up the tool to automatically extract the stream IDs from the links.
  4. Copy the converted stream IDs from the Substitution section of the Regex Tool to the clipboard.
  5. Download, install, and run the Twitch Archiver.
  6. Paste the stream IDs into the right textbox of the Twitch Archiver.
  7. The Twitch Archiver will automatically download every single stream and constituent parts, and shelve them into nice folders.

Manual Method

This method is the old "copy links and download parts manually" method.

  1. After you've chosen your chunk, right-click and copy the URL of one stream, and put it into the Twitch Stream Download Tool. The parts are split into 30-minute segments, download all of them.
    • You might find it useful to use a concurrent download manager, such as DownThemAll or JDownloader (described below)
  2. After downloading all the parts from a single stream, group them together into a folder. The folder name should follow this format:
    • YYYY-MM-DD [streamer] title
    • e.g. 2013-02-12 [Joel] Daikatana
  3. Repeat these steps until all the streams in your chunk have been downloaded.

Please don't rename or merge the video files!

  • Doing so will take too much extra time and effort.
  • It also makes it harder for an automated script to figure out which videos we have.
  • The filenames are critical unique metadata IDs!

Just leave the videos as is. The folders are enough.

Uploading your Chunk

  1. Once you're done downloading a chunk, go to The Checklist, find your chunk, and edit your reservation to the following:
    * Status: '''Downloaded!'''
  2. Put all the stream folders into one big folder, named after your Chunk:
    • e.g. Chunk 2014-10_C22
  3. Upload that folder to the /r/Vinesauce FTP Server, or mega.co.nz.
    • For FTP Upload, use Filezilla.
    • Ignore the existing file structure on the server! Just put the folder under /Vinesauce/Chunks/
  4. After uploading, go to The Checklist, find your chunk, and edit your reservation to the following:
    * Upload Location: '''Uploaded!''' (insert link here)
  5. Download another chunk!

The Tools

Twitch Mass Download App

This Mono/.NET application automates the process of downloading a whole list of streams; or even from an entire user.

Just give it the name of a streamer, or a bunch of stream IDs, and it will start slurping them down.

Twitch Download Tool

Just copy the URL of a stream and put it into the Twitch Download Tool to obtain the links to download the full stream video. It will be divided into 30 minute parts.

Link: http://www.twitchtools.com/video-download.php

Zazibar's Twitch VOD Download Lister

Zazibar has made a nice Twitch VOD Download List utility. It will list all videos from a Twitch profile (e.g. Vinesauce) and create easy download links for you.

Link: http://twitch.zazi.bar/

How to use:

  1. Open up Zazibar's Twitch VOD Download List.
  2. Search for Vinesauce.
  3. Click Download on whatever stream you want to download.
  4. Download all the parts of that stream.
    • You might find it useful to use a concurrent download manager, such as DownThemAll or JDownloader (described below)

DownThemAll - Addon for faster downloads with Firefox

For faster concurrent downloading, use the Firefox Addon DownThemAll

  1. Install DownThemAll on Firefox.
  2. Open Zazibar's Twitch VOD Download Tool.
  3. Search for Vinesauce .
  4. Click Download on whatever stream you want to download.
  5. Now, right click, and in the menu click downthemall.
  6. The DTA window will open, then you will e able to select the FLV files all at once to download.

A piece of advice: I don't think you can change the file names before downloading, so make sure to make a folder that says what they are before you dump them in some big folder.

As far as telling them apart and the order: They should be in the order they appear on the page in the DTA window and the folder you download them to. So just be careful. If you are worried you can check the individual links that you get from the zazi.bar downloader.

Source: Vinesauce Forum User - RedBreloom

Jdownloader

Another way to make concurrent downloads.

Link: http://jdownloader.org/

Future Method: Mass Download with a Script

Usually I would use youtube-dl, but it has issues with Twitch due to the sudden change in policy.

I'll slap together a quick and dirty Python slurping script sooner or later.

The Checklist

Vinesauce currently has 1714 videos in its VOD backlog.

Join the effort to archive the videos by reserving a chunk to archive now!

Follow The Procedure to download streams, as explained above.

This comprehensive checklist is based on Null's HTML list. However, his list seems to be missing a few videos here and there, so fix as you see fit.

Checklists


Known Collections

Some fans have been collecting streams long before this collection was started.

  • HungarianGameOver - Currently has the largest known collection of Vinesauce streams, at 150GB and growing. He goes for the cream of the crop, however, rather than big chunks.
 * Streams he has archived are marked on the checklist with an (HGO).

External Links