Difference between revisions of "Yahoo! Video Warroom"

From Archiveteam
Jump to navigation Jump to search
(added finished entries, changed all instances of jake1 to jakemcgra, added note about sdboyd)
Line 43: Line 43:
We have no idea where we're going to put this. We're going to need a bigger boat.
We have no idea where we're going to put this. We're going to need a bigger boat.


If you used axel / grapefruit-axel, please tell soultcer or guan on IRC. Axel doesn't store timestamps, thus we try to add that data manually, but we need to know who used axel and who didn't.
If you used axel / grapefruit-axel, please tell soultcer or guan or underscor on IRC. Axel doesn't store timestamps, thus we try to add that data manually, but we need to know who used axel and who didn't.


=== Verified and On-Drive ===
=== Verified and On-Drive ===
Line 51: Line 51:
* '''0,700,000 - 0,750,000''' (3,213 Users, 20,767 videos, 127gb) Underscor D3, D4 GOODTIMES
* '''0,700,000 - 0,750,000''' (3,213 Users, 20,767 videos, 127gb) Underscor D3, D4 GOODTIMES
* '''1,500,000 - 1,600,000''' (5,252 Users, 7642 videos, 86g) NovaKing D3, D4 GOODTIMES
* '''1,500,000 - 1,600,000''' (5,252 Users, 7642 videos, 86g) NovaKing D3, D4 GOODTIMES
* '''2,100,000 - 2,200,000''' (4,519 Users, 16,597 videos, 144g) Joey D3, D4
* '''2,100,000 - 2,200,000''' (4,519 Users, 16,597 videos, 144g) Joey D3, D4 GOODTIMES
* '''3,000,000 - 3,100,000''' (6,297 Users, 18,418 videos, 163gb) Guan D3, D4 GOODTIMES
* '''3,000,000 - 3,100,000''' (6,297 Users, 18,418 videos, 163gb) Guan D3, D4 GOODTIMES
* '''7,600,000 - 7,700,000''' (4,200 Users, 17,885 videos, 189gb) NovaKing D3, D4 GOODTIMES


=== Uploaded, needs factchecking ===
=== Uploaded, needs factchecking ===
* '''0,500,000 - 0,600,000''' (6,543 Users, 13,849 videos, 115gb) NovaKing D1 (0 Missing, verifying timestamps)
* '''0,500,000 - 0,600,000''' (6,543 Users, 13,849 videos, 115gb) NovaKing D1 (0 Missing, verifying timestamps)
* '''1,100,000 - 1,200,000''' (7,613 Users, 12,892 videos, 177gb ) NovaKing D1 (1147 Missing)
* '''1,100,000 - 1,200,000''' (7,613 Users, 12,892 videos, 177gb ) NovaKing D1 (1147 Missing, verifying timestamps)
* '''2,000,000 - 2,100,000''' (4,408 Users, 20,583 videos, 167g) dashcloud D1 (1186 Missing)
* '''2,000,000 - 2,100,000''' (4,408 Users, 20,583 videos, 167g) dashcloud D1 (1186 Missing, verifying timestamps)
* '''7,600,000 - 7,700,000''' (4,200 Users, 17,885 videos, 189gb) NovaKing D1 (0 Missing, verifying timestamps)


=== Uploading ===
=== Uploading ===

Revision as of 05:55, 9 March 2011

Planning room for Yahoo! Video.


Yahoovideo.jpg

Phase One: Username Acquisition (Complete)

"You people are working so hard!"

Summary

In roughly 30 days, 12 Archive Team members scanned 9,360,000 Yahoo! Video user account numbers. Of these, 9,218,844 were actual user accounts (others were deleted or "networks", i.e. playlists). Of these, 555,502 had actually taken the time to upload one or more videos. All told, we showed that 3,585,981 videos were uploaded to Yahoo! Video.

  • 0,000,000 - 1,000,000 Jason Scott, Chronomex
  • 1,000,000 - 2,250,000 Joey/Underscor
  • 2,250,000 - 2,500,000 Underscor/Chronomex
  • 2,500,000 - 3,500,000 Soultcer/Underscor
  • 3,500,000 - 4,500,000 Chronomex
  • 4,500,000 - 5,000,000 Underscor
  • 5,000,000 - 5,500,000 Jason Scott/Spirit
  • 5,500,000 - 5,750,000 Ymgve
  • 5,750,000 - 5,900,000 guan/Jason Scott
  • 5,900,000 - 6,000,000 soultcer
  • 6,000,000 - 6,050,000 Underscor/NovaKing
  • 6,050,000 - 6,150,000 robzy
  • 6,150,000 - 6,200,000 phuz
  • 6,200,000 - 6,250,000 guan
  • 6,250,000 - 6,750,000 phuz/NovaKing
  • 6,750,000 - 7,500,000 chronomex/NovaKing
  • 7,500,000 - 8,000,000 Jason Scott
  • 8,000,000 - 8,250,000 Spirit
  • 8,250,000 - 8,750,000 Sungo
  • 8,750,000 - 9,000,000 Spirit
  • 9,000,000 - 9,360,000 Jason Scott/Underscor/Ymgve

Phase Two: Video Acquisition

"Oh my god, some of these videos!"

Once the hotlists of who has videos have been grabbed (and that is going to happen rather shortly, thanks to the army of folks), we'll be running a program called grapefruit (the g and fruit are silent, ræp) that yanks down all the videos from a given user. The number of users with video is a tiny percentage of the total users (we're seeing rates of less than 1 video-keeping user for every 10-20 of the users, and less), it will still be significant diskspace. It's video, after all. After an initial test run of grapefruit, we'll start handing out numbers and hitlists to people, and they'll begin downloading.

We have no idea where we're going to put this. We're going to need a bigger boat.

If you used axel / grapefruit-axel, please tell soultcer or guan or underscor on IRC. Axel doesn't store timestamps, thus we try to add that data manually, but we need to know who used axel and who didn't.

Verified and On-Drive

Talk to underscor to get your stuff uploaded. (I will perform a final check of the timestamps)

  • 0,300,000 - 0,400,000 (6,909 Users, 41,131 videos, 342gb) Underscor D3, D4 GOODTIMES
  • 0,700,000 - 0,750,000 (3,213 Users, 20,767 videos, 127gb) Underscor D3, D4 GOODTIMES
  • 1,500,000 - 1,600,000 (5,252 Users, 7642 videos, 86g) NovaKing D3, D4 GOODTIMES
  • 2,100,000 - 2,200,000 (4,519 Users, 16,597 videos, 144g) Joey D3, D4 GOODTIMES
  • 3,000,000 - 3,100,000 (6,297 Users, 18,418 videos, 163gb) Guan D3, D4 GOODTIMES
  • 7,600,000 - 7,700,000 (4,200 Users, 17,885 videos, 189gb) NovaKing D3, D4 GOODTIMES

Uploaded, needs factchecking

  • 0,500,000 - 0,600,000 (6,543 Users, 13,849 videos, 115gb) NovaKing D1 (0 Missing, verifying timestamps)
  • 1,100,000 - 1,200,000 (7,613 Users, 12,892 videos, 177gb ) NovaKing D1 (1147 Missing, verifying timestamps)
  • 2,000,000 - 2,100,000 (4,408 Users, 20,583 videos, 167g) dashcloud D1 (1186 Missing, verifying timestamps)

Uploading

Please try to keep this box up to date.

  • 1,800,000 - 1,900,000 (4,744 Users, 20,494 videos, 169g) bluknight D1
  • 2,900,000 - 3,000,000 (6,221 Users, 260G) dj_ryan (to datadump 4421)
  • 3,600,000 - 3,700,000 (4,494 Users, 21,863 videos, 251gb) chronomex D1
  • 5,150,000 - 5,200,000 (3,313 Users, 18,127 videos, 235gb) Guan (to underscor, paused)
  • 5,400,000 - 5,500,000 (6,897 Users, 738gb) dj_ryan (to datadump 4421)
  • 6,300,000 - 6,400,000 (6,378 Users, 44,962 videos, 849gb) Guan (to underscor)
  • 7,500,000 - 7,600,000 (4,347 Users, 20,418 videos, 240gb) Guan (to datadump 4350)
  • 8,000,000 - 8,100,000 (3,416 Users, 16,781 videos, 201gb) bluknight D1
  • 8,600,000 - 8,700,000 (3,863 Users, 20,391 videos, 241gb) Joey rsync.net
  • 8,800,000 - 8,850,000 (2,170 Users, 8,368 videos, 71gb ) sdboyd D1

Multiple uploads by Novaking to 4363.

Finished

  • 0,000,000 - 0,005,000 (3,960 Users, x videos, xgb) Underscor
  • 0,005,000 - 0,010,000 (3,993 Users) Underscor
  • 0,082,200 - 0,083,321 (1,000 Users, 6,536 videos, 60gb) Guan
  • 0,083,322 - 0,100,003 (5,000 Users, 34,236 videos, 324gb) Guan
  • 0,100,000 - 0,200,000 (6,247 Users, x videos, 159gb) Underscor
  • 0,200,000 - 0,300,000 (6,461 Users, x videos, 141gb) Underscor
  • 0,400,000 - 0,500,000 (6,231 Users, x videos, 159gb) Underscor
  • 0,750,000 - 0,800,000 (3,735 Users, 19,032 videos, 136G) atg
  • 0,800,000 - 0,850,000 (3,140 Users, 10,969 videos, 104G) crust
  • 0,850,000 - 0,900,000 (3,749 Users, 18,562 videos, 138gb) filer
  • 1,000,000 - 1,100,000 (6,331 Users, 31,241 videos, 247gb) Zumthing (original download + factcheck cleanup)
  • 1,400,000 - 1,500,000 (5,194 Users, 6,476 videos, 68gb) beardicus
  • 1,600,000 - 1,700,000 (4,933 Users, 8250 videos, 82g) Masterono
  • 1,700,000 - 1,800,000 (4,368 Users, 8013 videos, 97G) atg
  • 1,800,000 - 1,900,000 (4,744 Users, 20,494 videos, 169g) bluknight
  • 2,200,000 - 2,300,000 (4,216 Users, 18,708 videos 168G) jakemcgra
  • 2,300,000 - 2,400,000 (4,615 Users, 20,911 videos, 216g) DoubleJ
  • 2,400,000 - 2,500,000 (2,802 Users) NovaKing (Userscrape seems truncated, may need redo, transfering to underscor)
  • 2,500,000 - 2,600,000 (3,949 Users, 15,994 videos, 146gb) Zumthing (original download + factcheck cleanup)
  • 3,100,000 - 3,200,000 (5,760 Users, 15,661 videos, 186gb) Masterono
  • 3,250,000 - 3,300,000 (2,439 Users) taaz
  • 3,300,000 - 3,400,000 (4,252 Users) NovaKing (transfering to underscor)
  • 3,400,000 - 3,500,000 (4,560 Users) NovaKing (transfering to underscor)
  • 3,700,000 - 3,800,000 (4,262 Users) NovaKing (transfering to underscor)
  • 3,800,000 - 3,900,000 (3,936 Users, 22,414 videos, 312gb) dysproseu
  • 4,100,000 - 4,200,000 (4,511 users, 25,383 videos, 360gb) penemue
  • 6,750,000 - 6,800,000 (2,838 Users, 17,555 videos, 356gb) Jason Scott (Transferring to data dump 1004)
  • 7,000,000 - 7,100,000 (4,833 Users, 24,523 videos, 311gb) bluknight
  • 7,100,000 - 7,200,000 (4,026 Users, 13,244 videos, 167gb) zumthing
  • 7,700,000 - 7,800,000 (4,028 Users, 15,037 videos, 171gb) K4K
  • 7,700,000 - 7,800,000 (4,028 Users, 19,155 videos, 219gb) jakemcgra
  • 7,900,000 - 8,000,000 (3,517 Users, 10,392 videos, 111gb) matzie
  • 8,100,000 - 8,200,000 (3,380 Users, 13,882 videos. 160gb) Primus104 (Transferring to data dump 4362)
  • 8,300,000 - 8,400,000 (2,920 Users, 8,973 videos, 93gb) Masterono
  • 8,400,000 - 8,500,000 (3,535 Users, 16,784 videos, 199gb) yoyoguy2
  • 8,700,000 - 8,800,000 (3,730 Users, 16,698 videos, 186gb) t0phux
  • 8,850,000 - 9,000,000 (1,770 Users, 8,832 videos, 106gb) Gandalfar
  • 9,000,000 - 9,050,000 (1,745 Users, 6,889 videos, 86G) atg
  • 9,050,000 - 9,100,000 (1,796 Users, 6,142 videos, 79gb) Masterono
  • 9,150,000 - 9,200,000 (5,173 Users, 22,655 videos, 261gb) Underscor
  • 9,200,000 - 9,300,000 (8,941 Users, 30,179 videos, 341gb) Zumthing (overlaps with drspangle)
  • 9,300,000 - 9,360,000 (2,179 Users, 4,406 videos, 51gb) Robertcu

In-Process

  • "OPEN" = unassigned, hotlist available

NOTE that the actual ranges FROM 0-083323 are totally WRONG AND MESSED UP. underscore will fix them later. (d'oh, oops)

ksuter is doing a rough dupe of 47-100003, but do not count his effort in assigning ranges.

  • 0,010,000 - 0,015,000 (4,055 Users) soultcer
  • 0,015,000 - 0,020,000 (3,950 Users) jakemcgra
  • 0,020,000 - 0,025,000 (4,052 Users) ???
  • 0,025,000 - 0,030,000 (4,040 Users) ???
  • 0,030,000 - 0,035,000 (4,095 Users) penemue
  • 0,035,000 - 0,040,000 (3,966 Users) penemue
  • 0,040,000 - 0,045,000 (4,024 Users) ???
  • 0,045,000 - 0,050,000 (4,051 Users) ???
  • 0,050,000 - 0,055,000 (3,976 Users) zumthing
  • 0,055,000 - 0,060,000 (4,086 Users) ???
  • 0,060,000 - 0,065,000 (4,017 Users) ???
  • 0,065,000 - 0,070,000 (4,077 Users) ???
  • 0,070,000 - 0,075,000 (4,032 Users) ???
  • 0,075,000 - 0,077,255 (1,801 Users) ???
  • 0,077,256 - 0,079,747 (2,000? Users) Guan
  • 0,079,748 - 0,082,199 (2,000? Users) ksuter
  • 0,600,000 - 0,650,000 (x Users) robertcu
  • 0,650,000 - 0,700,000 (x Users) Hydruh
  • 0,900,000 - 1,000,000 (6,453 Users) K4K
  • 1,200,000 - 1,300,000 (7,130 Users) drspangle
  • 1,300,000 - 1,400,000 (7,598 Users) sep332
  • 1,900,000 - 2,000,000 (4,481 Users) ajclement
  • 2,600,000 - 2,700,000 (4,208 Users) Frigolit
  • 2,700,000 - 2,800,000 (4,754 Users) ersi (timestamps: ok)
  • 2,800,000 - 2,900,000 (5,623 Users) amuck
  • 3,200,000 - 3,250,000 (2,603 Users) Soultcer (timestamps: ok)
  • 3,500,000 - 3,600,000 (4,895 Users) tobbez
  • 3,900,000 - 4,000,000 (4,055 Users) HA
  • 4,000,000 - 4,100,000 (4,251 Users) primus_102 (timestamps: bad)
  • 4,200,000 - 4,300,000 (4,250 Users) atg
  • 4,300,000 - 4,400,000 (4,355 Users) NovaKing
  • 4,400,000 - 4,500,000 (4,715 Users) beardicus
  • 4,500,000 - 4,600,000 (3,434 Users) DoubleJ
  • 4,600,000 - 4,700,000 (5,151 Users) eberon
  • 4,700,000 - 4,750,000 (2,665 Users) Gandalfar
  • 4,750,000 - 4,800,000 (2,670 Users, 18%, 39G) crust
  • 4,800,000 - 4,900,000 (5,764 Users) monst3r_
  • 4,900,000 - 5,000,000 (6,738 Users) jch
  • 5,000,000 - 5,100,000 (6,545 Users) monst3r_
  • 5,100,000 - 5,150,000 (3,254 Users) Spirit
  • 5,200,000 - 5,300,000 (8,456 Users) jakemcgra
  • 5,300,000 - 5,400,000 (6,289 Users) jakemcgra
  • 5,600,000 - 5,700,000 (5,903 Users) _kp
  • 5,700,000 - 5,800,000 (3,194 Users) filer
  • 5,800,000 - 5,900,000 (6,188 Users) jch
  • 5,900,000 - 6,000,000 (7,179 Users) km3k / slow progress (I have downloaded 357 users, who have 2078 videos. Size: 38G QUEUE: I am working on users 5900005-5999982 and have finished 4.900% (357/7179))
  • 6,000,000 - 6,050,000 (3,571 Users) savetz / uploading to 4408
  • 6,050,000 - 6,100,000 (3,747 Users) t0phux
  • 6,100,000 - 6,200,000 (7,433 Users) incendui
  • 6,200,000 - 6,210,039 ( 738 Users) Pneu
  • 6,210,039 - 6,250,000 (2,609 Users) NovaKing
  • 6,250,000 - 6,300,000 (3,109 Users) NovaKing
  • 6,400,000 - 6,500,000 (6,097 Users) Masterono
  • 6,500,000 - 6,600,000 (6,588 Users) NovaKing
  • 6,600,000 - 6,650,000 (3,516 Users) Lambda_Dr
  • 6,650,000 - 6,700,000 (3,023 Users) sdboyd
  • 6,650,000 - 6,700,000 (3,023 Users) jakemcgra (sdboyd having space issues, taking over starting at 6,666,221)
  • 6,700,000 - 6,750,000 (3,079 Users) yoyoguy2
  • 6,800,000 - 6,900,000 (5,234 Users) dashcloud
  • 6,900,000 - 7,000,000 (5,676 Users) bluknight (sandra)
  • 7,200,000 - 7,300,000 (3,939 Users) dysproseu
  • 7,300,000 - 7,400,000 (4,304 Users) amerrykan
  • 7,400,000 - 7,500,000 (4,204 Users) ZPedro
  • 7,500,000 - 7,600,000 (4,347 Users) sadcarrot
  • 7,800,000 - 7,900,000 (3,720 Users) NovaKing
  • 8,200,000 - 8,300,000 (3,335 Users) NovaKing
  • 8,500,000 - 8,600,000 (3,516 Users) red_cell
  • 8,900,000 - 9,000,000 (3,793 Users) khaoohs
  • 9,100,000 - 9,150,000 (1,989 Users) Killface
  • 9,250,000 - 9,300,000 (2,835 Users) drspangle (overlaps with Zumthing)

Phase Three: Final Resting Place

In an ideal world, all the downloaders would hold copies and everyone would just torrent the thing. That's not going to happen in current contemporary machine capacity. Our very rough estimate is that a download of Yahoo! Video is going to be 20 terabytes. While this is under a thousand dollars of newegg-purchased disks, it's still a bit much to ask of anyone. Meanwhile, the more forward-looking of volunteers are asking, "what's the exit strategy?".

Jason and a few others have locations that video downloaders can upload a copy of their acquired data to; these are being placed on a stack of hard drives. At the same time, Archive.org is providing a location for the data to be uploaded to as well. Other entities are being contacted at the present about providing space for copies of the full archive, as well.

The priority, however, is to get all the data in the first place.

Statistics

"Why didn't more people use it! It was awesome!"

People like easy to quote statistics from a project, so here are some, with others coming soon.

Yahoo! Video officially started as a user content system in June of 2006. Previously, it had been a directory of other video sites around the net (harkening back to Yahoo's previous life as a directory instead of its short burst as a content creator). From the start, it was called a "YouTube Clone" and other derisive terms.

However, users did decide to go for this new environment. Archive Team found about 9,360,000 accounts in use on Yahoo Video. "In Use" is a relative term, of course - many, many users consist of no uploaded videos, no user picture, and no other activity - just a name. We'll have more, but for example, of 1.5 million accounts we've scanned so far, only 146,600 have uploaded videos, and if you shear off the first 100,000 (i.e., frontloaded with content partner) accounts, that 10% figure drops down to 5%.

It didn't take very long for users to figure out Yahoo! Video is not the way to go, either as participants or as viewers. The most popular video, "Baby Raccoons Saved from Soda Machine" has 14 million views. Compare with one of the top videos on YouTube - Justin Beiber performing "Baby" with Ludacris, which has 457 million views. By the time the raccoons are viewed, the next most popular video, "Worst Best Man Ever", clocks in at just over 10 million views. After the first 20 videos, the views per video are down to about 4 million. By the time we get to the top 100, we're already skirting 1 million. Again, a single video on Youtube, the most popular, has a number of views basically the same as the top 100 Yahoo! videos combined.