Yahoo! Video Warroom

From Archiveteam
Jump to navigation Jump to search

Planning room for Yahoo! Video.


Yahoovideo.jpg

Phase One: Username Acquisition

"You people are working so hard!"

Acquiring the usernames and which have videos is done by the script yvscan. Ask Jason for a copy. The process is somewhat slow, as going too fast gets the dreaded 999. As a result, we need a large number of people and machines doing the work, so we have to put them here to keep track of who's doing what. The scraper pulls in userpics, a copy of the user page, and generates a userlist.txt of all the accounts it has found.

There are roughly 9,360,000 user accounts on Yahoo Video. (Let us not fool ourselves as to thinking these are real people.) Some are "Networks", i.e. collections of other videos (but not many, maybe 1 out of every 50,000, or less.) Most have no videos at all, but to make a hitlist of videos to download, we need those.

When You Finish

In these instructions, 0_000_000 is the beginning number of your block.

Zip up the files you downloaded:

tar cjvf 0_000_000-to-0_050_000.tar.bz2 userpics userpages

Name the userlist.txt and networklist.txt (if it exists) to something unique, then compress:

mv userlist.txt userlist_0_000_000.txt
mv networklist.txt networklist_0_000_000.txt
bzip2 userlist_0_000_000.txt
bzip2 networklist_0_000_000.txt

Email the user and network lists to { jason at textfiles.com }. They should take up just a few megabytes in total. If you have a website with enough space, please put the .tar.bz2 there and include the url in the mail. Otherwise, ask for assistance.

Summary

  • 0,000,000 - 4,500,000 Jason Scott, Chronomex, Joey, Underscor, Soultcer
  • 4,750,000 - 5,300,000 Jason Scott, Underscor
  • 5,500,000 - 6,000,000 Ymgve, Underscor
  • 7,500,000 - 9,360,000 Underscor, Jason Scott, Spirit, Sungo

Finished

  • 0,000,000 - 0,750,000 Jason Scott, Chronomex
  • 0,750,000 - 1,000,000 Chronomex
  • 1,000,000 - 1,750,000 Joey
  • 1,750,000 - 2,000,000 Underscor
  • 2,000,000 - 2,250,000 Joey
  • 2,250,000 - 2,500,000 Underscor/Chronomex (Chronomex is finishing)
  • 2,500,000 - 2,750,000 Soultcer
  • 2,750,000 - 3,000,000 Underscor
  • 3,000,000 - 3,250,000 Soultcer
  • 3,250,000 - 3,500,000 Underscor
  • 3,500,000 - 4,000,000 Chronomex
  • 4,000,000 - 4,250,000 Chronomex (gir)
  • 4,250,000 - 4,500,000 Chronomex (dedi)
  • 4,500,000 - 4,625,000 Underscor
  • 4,750,000 - 5,000,000 Underscor
  • 5,000,000 - 5,300,000 Jason Scott
  • 5,500,000 - 5,750,000 Ymgve
  • 5,750,000 - 6,000,000 Underscor
  • 6,050,000 - 6,100,000 robzy
  • 6,100,000 - 6,150,000 robzy
  • 6,150,000 - 6,200,000 phuz
  • 6,250,000 - 6,300,000 phuz
  • 6,350,000 - 6,400,000 NovaKing
  • 6,400,000 - 6,450,000 NovaKing
  • 6,450,000 - 6,500,000 NovaKing
  • 6,500,000 - 6,550,000 phuz
  • 6,550,000 - 6,600,000 phuz
  • 6,650,000 - 6,700,000 NovaKing
  • 6,750,000 - 7,000,000 chronomex (dante01)
  • 7,000,000 - 7,150,000 chronomex (dante02)
  • 7,500,000 - 8,000,000 Jason Scott
  • 8,000,000 - 8,250,000 Spirit
  • 8,250,000 - 8,500,000 Sungo
  • 8,500,000 - 8,750,000 Sungo
  • 8,750,000 - 9,000,000 Spirit
  • 9,000,000 - 9,360,000 Jason Scott/Underscor/Ymgve

Continuing

  • 4,625,000 - 4,750,000 Underscor
  • 5,250,000 - 5,500,000 Jason Scott/Spirit (Spirit is finishing)
  • 6,000,000 - 6,050,000 Underscor (server down) (NovaKing duping)
  • 6,200,000 - 6,250,000 cb43569
  • 6,300,000 - 6,350,000 NovaKing
  • 6,600,000 - 6,650,000 phuz
  • 6,700,000 - 6,750,000 NovaKing
  • 7,150,000 - 7,175,000 NovaKing
  • 7,175,000 - 7,250,000 NovaKing
  • 7,200,000 - 7,225,000 NovaKing
  • 7,225,000 - 7,250,000 NovaKing
  • 7,250,000 - 7,500,000 chronomex (dante03)

Phase Two: Video Acquisition

"Oh my god, some of these videos!"

Once the hotlists of who has videos have been grabbed (and that is going to happen rather shortly, thanks to the army of folks), we'll be running a program called grapefruit (the g and fruit are silent, ræp) that yanks down all the videos from a given user. The number of users with video is a tiny percentage of the total users (we're seeing rates of less than 1 video-keeping user for every 10-20 of the users, and less), it will still be significant diskspace. It's video, after all. After an initial test run of grapefruit, we'll start handing out numbers and hitlists to people, and they'll begin downloading.

We have no idea where we're going to put this. We're going to need a bigger boat.

Verified and On-Drive

Talk to underscor to get your stuff uploaded.

  • 0,700,000 - 0,750,000 (3,213 Users, 20,767 videos, 127gb) Underscor D3

Uploading

Please try to keep this box up to date.

  • 0,500,000 - 0,600,000 (6,543 Users, 13,849 videos, 115gb) NovaKing D1
  • 1,100,000 - 1,200,000 (7,613 Users, 12,892 videos, 177gb ) NovaKing D1
  • 1,500,000 - 1,600,000 (5,252 Users, 7642 videos, 86g) NovaKing D1
  • 1,800,000 - 1,900,000 (4,744 Users, 20,494 videos, 169g) bluknight D1
  • 2,100,000 - 2,200,000 (4,519 Users, 16,597 videos, 144g) Joey D1
  • 3,000,000 - 3,100,000 (6,297 Users, 18,418 videos, 163gb) Guan D1
  • 3,600,000 - 3,700,000 (4,494 Users, 21,863 videos, 251gb) chronomex D1
  • 7,600,000 - 7,700,000 (4,200 Users, 17,885 videos, 189gb) NovaKing D1
  • 8,000,000 - 8,100,000 (3,416 Users, x videos, xgb) bluknight D1
  • 8,800,000 - 8,850,000 (2,170 Users, 8,368 videos, 71gb ) sdboyd D1

Finished

  • 0,100,000 - 0,200,000 (6,247 Users, x videos, 159gb) Underscor
  • 0,200,000 - 0,300,000 (6,461 Users, x videos, 141gb) Underscor
  • 0,400,000 - 0,500,000 (6,231 Users, x videos, 159gb) Underscor
  • 0,750,000 - 0,800,000 (3,735 Users, 1,097 videos, 7gb) atg (Use as test)
  • 0,850,000 - 0,900,000 (3,749 Users, 18,562 videos, 138gb) filer
  • 1,600,000 - 1,700,000 (4,933 Users, 8250 videos, 82g) Masterono
  • 1,800,000 - 1,900,000 (4,744 Users, 20,494 videos, 169g) bluknight
  • 2,000,000 - 2,100,000 (4,408 Users, 20,583 videos, 167g) dashcloud
  • 2,400,000 - 2,500,000 (2,802 Users) NovaKing (Userscrape seems truncated, may need redo, transfering to underscor)
  • 2,500,000 - 2,600,000 (3,949 Users, 7,849 videos, 82gb) Zumthing
  • 3,100,000 - 3,200,000 (5,760 Users, 15,661 videos, 186gb) Masterono
  • 3,300,000 - 3,400,000 (4,252 Users) NovaKing (transfering to underscor)
  • 3,400,000 - 3,500,000 (4,560 Users) NovaKing (transfering to underscor)
  • 3,700,000 - 3,800,000 (4,262 Users) NovaKing (transfering to underscor)
  • 5,150,000 - 5,200,000 (3,313 Users, 18,127 videos, 235gb) Guan
  • 7,500,000 - 7,600,000 (4,347 Users, 20,418 videos, 240gb) Guan
  • 7,700,000 - 7,800,000 (4,028 Users, 15,037 videos, 171gb) K4K
  • 7,900,000 - 8,000,000 (3,517 Users, 10,392 videos, 111gb) matzie
  • 8,100,000 - 8,200,000 (3,380 Users, 13,882 videos. 160gb) Primus104
  • 8,300,000 - 8,400,000 (2,920 Users, 8,973 videos, 93gb) Masterono
  • 8,400,000 - 8,500,000 (3,535 Users, 16,784 videos, 199gb) yoyoguy2
  • 8,700,000 - 8,800,000 (3,730 Users, 16,698 videos, 186gb) t0phux
  • 9,000,000 - 9,050,000 (1,745 Users) atg
  • 9,050,000 - 9,100,000 (1,796 Users, 6,142 videos, 79gb) Masterono
  • 9,150,000 - 9,200,000 (5,173 Users, 22,655 videos, 261gb) Underscor
  • 9,200,000 - 9,300,000 (8,941 Users, 30,179 videos, 341gb) Zumthing (overlaps with drspangle)

In-Process

LEGEND.

  • "???" = unassigned
  • "OPEN" = unassigned, hotlist available

And now the list.

  • 0,000,000 - 0,100,000 (72,176 Users) Underscor 0-10k Done
  • 0,300,000 - 0,400,000 (6,909 Users) Underscor Downloading
  • 0,600,000 - 0,700,000 (7,833 Users) Underscor Downloading
  • 0,800,000 - 0,850,000 (3,140 Users) Crust
  • 0,900,000 - 1,000,000 (6,453 Users) K4K
  • 1,000,000 - 1,100,000 (6,331 Users) Zumthing
  • 1,200,000 - 1,300,000 (7,130 Users) drspangle
  • 1,300,000 - 1,400,000 (7,598 Users) sep332
  • 1,400,000 - 1,500,000 (5,194 Users) beardicus
  • 1,700,000 - 1,800,000 (4,368 Users) atg
  • 1,900,000 - 2,000,000 (4,481 Users) ajclement
  • 2,200,000 - 2,300,000 (4,216 Users) jakemcgra
  • 2,300,000 - 2,400,000 (4,615 Users) DoubleJ
  • 2,600,000 - 2,700,000 (4,208 Users) Frigolit
  • 2,700,000 - 2,800,000 (4,754 Users) ersi
  • 2,900,000 - 3,000,000 (6,221 Users) dj_ryan
  • 3,200,000 - 3,250,000 (2,603 Users) Soultcer
  • 3,250,000 - 3,300,000 (2,439 Users) taaz
  • 3,800,000 - 3,900,000 (3,936 Users) dysproseu
  • 3,900,000 - 4,000,000 (4,055 Users) HA
  • 4,000,000 - 4,100,000 ???
  • 4,100,000 - 4,200,000 ???
  • 4,200,000 - 4,300,000 ???
  • 4,300,000 - 4,400,000 ???
  • 4,400,000 - 4,500,000 (3,434 Users) OPEN
  • 4,600,000 - 4,700,000 ???
  • 4,700,000 - 4,750,000 ???
  • 4,750,000 - 4,800,000 (2,670 Users) crust__
  • 4,800,000 - 4,900,000 (5,764 Users) monst3r_
  • 4,900,000 - 5,000,000 (6,738 Users) jch
  • 5,000,000 - 5,100,000 (6,545 Users) monst3r_
  • 5,100,000 - 5,150,000 (3,254 Users) Spirit
  • 5,200,000 - 5,300,000 (8,456 Users) jake1
  • 5,300,000 - 5,400,000 (6,289 Users) jake1
  • 5,400,000 - 5,500,000 (6,897 Users) dj_ryan
  • 5,600,000 - 5,700,000 (5,903 Users) _kp
  • 5,700,000 - 5,800,000 ???
  • 5,800,000 - 5,900,000 ???
  • 5,900,000 - 6,000,000 ???
  • 6,000,000 - 6,050,000 (3,571 Users) savetz
  • 6,050,000 - 6,100,000 (3,747 Users) t0phux
  • 6,100,000 - 6,200,000 (7,433 Users) incendui
  • 6,200,000 - 6,250,000 (3,347 Users) Pneu
  • 6,250,000 - 6,300,000 (3,109 Users) NovaKing
  • 6,300,000 - 6,400,000 (6,378 Users) Guan
  • 6,400,000 - 6,500,000 (6,097 Users) Masterono
  • 6,500,000 - 6,600,000 (6,588 Users) NovaKing
  • 6,600,000 - 6,650,000 ???
  • 6,650,000 - 6,700,000 (3,023 Users) sdboyd
  • 6,700,000 - 6,750,000 (3,079 Users) yoyoguy2
  • 6,750,000 - 6,800,000 (2,838 Users) Jason Scott
  • 6,800,000 - 6,900,000 (5,234 Users) dashcloud
  • 6,900,000 - 7,000,000 (5,676 Users) bluknight
  • 7,000,000 - 7,100,000 (4,833 Users) bluknight
  • 7,100,000 - 7,200,000 ???
  • 7,200,000 - 7,300,000 ???
  • 7,300,000 - 7,400,000 (4,304 Users) OPEN
  • 7,400,000 - 7,500,000 ???
  • 7,500,000 - 7,600,000 (4,347 Users) sadcarrot
  • 7,700,000 - 7,800,000 (4,028 Users) jakemcgra
  • 7,800,000 - 7,900,000 (3,720 Users) NovaKing
  • 8,000,000 - 8,100,000 (3,416 Users) bluknight (tara)
  • 8,200,000 - 8,300,000 (3,335 Users) NovaKing
  • 8,500,000 - 8,600,000 (3,516 Users) red_cell
  • 8,600,000 - 8,700,000 (7,726 Users) Joey (slowbox)
  • 8,850,000 - 9,000,000 (1,770 Users) Gandalfar
  • 8,900,000 - 9,000,000 (3,793 Users) khaoohs
  • 9,100,000 - 9,150,000 (1,989 Users) Killface
  • 9,250,000 - 9,300,000 (2,835 Users) drspangle (overlaps with Zumthing)
  • 9,300,000 - 9,360,000 (2,179 Users) Robertcu

Phase Three: Final Resting Place

In an ideal world, all the downloaders would hold copies and everyone would just torrent the thing. That's not going to happen in current contemporary machine capacity. Our very rough estimate is that a download of Yahoo! Video is going to be 20 terabytes. While this is under a thousand dollars of newegg-purchased disks, it's still a bit much to ask of anyone. Meanwhile, the more forward-looking of volunteers are asking, "what's the exit strategy?".

Jason and a few others have locations that video downloaders can upload a copy of their acquired data to; these are being placed on a stack of hard drives. At the same time, Archive.org is providing a location for the data to be uploaded to as well. Other entities are being contacted at the present about providing space for copies of the full archive, as well.

The priority, however, is to get all the data in the first place.

Statistics

"Why didn't more people use it! It was awesome!"

People like easy to quote statistics from a project, so here are some, with others coming soon.

Yahoo! Video officially started as a user content system in June of 2006. Previously, it had been a directory of other video sites around the net (harkening back to Yahoo's previous life as a directory instead of its short burst as a content creator). From the start, it was called a "YouTube Clone" and other derisive terms.

However, users did decide to go for this new environment. Archive Team found about 9,360,000 accounts in use on Yahoo Video. "In Use" is a relative term, of course - many, many users consist of no uploaded videos, no user picture, and no other activity - just a name. We'll have more, but for example, of 1.5 million accounts we've scanned so far, only 146,600 have uploaded videos, and if you shear off the first 100,000 (i.e., frontloaded with content partner) accounts, that 10% figure drops down to 5%.

It didn't take very long for users to figure out Yahoo! Video is not the way to go, either as participants or as viewers. The most popular video, "Baby Raccoons Saved from Soda Machine" has 14 million views. Compare with one of the top videos on YouTube - Justin Beiber performing "Baby" with Ludacris, which has 457 million views. By the time the raccoons are viewed, the next most popular video, "Worst Best Man Ever", clocks in at just over 10 million views. After the first 20 videos, the views per video are down to about 4 million. By the time we get to the top 100, we're already skirting 1 million. Again, a single video on Youtube, the most popular, has a number of views basically the same as the top 100 Yahoo! videos combined.