Yahoo! Video Warroom

From Archiveteam
Revision as of 04:57, 15 March 2011 by Underscor (talk | contribs)
Jump to navigation Jump to search

Planning room for Yahoo! Video.


Yahoovideo.jpg

Phase One: Username Acquisition (Complete)

"You people are working so hard!"

Summary

In roughly 30 days, 12 Archive Team members scanned 9,360,000 Yahoo! Video user account numbers. Of these, 9,218,844 were actual user accounts (others were deleted or "networks", i.e. playlists). Of these, 555,502 had actually taken the time to upload one or more videos. All told, we showed that 3,585,981 videos were uploaded to Yahoo! Video.

  • 0,000,000 - 1,000,000 Jason Scott, Chronomex
  • 1,000,000 - 2,250,000 Joey/Underscor
  • 2,250,000 - 2,500,000 Underscor/Chronomex
  • 2,500,000 - 3,500,000 Soultcer/Underscor
  • 3,500,000 - 4,500,000 Chronomex
  • 4,500,000 - 5,000,000 Underscor
  • 5,000,000 - 5,500,000 Jason Scott/Spirit
  • 5,500,000 - 5,750,000 Ymgve
  • 5,750,000 - 5,900,000 guan/Jason Scott
  • 5,900,000 - 6,000,000 soultcer
  • 6,000,000 - 6,050,000 Underscor/NovaKing
  • 6,050,000 - 6,150,000 robzy
  • 6,150,000 - 6,200,000 phuz
  • 6,200,000 - 6,250,000 guan
  • 6,250,000 - 6,750,000 phuz/NovaKing
  • 6,750,000 - 7,500,000 chronomex/NovaKing
  • 7,500,000 - 8,000,000 Jason Scott
  • 8,000,000 - 8,250,000 Spirit
  • 8,250,000 - 8,750,000 Sungo
  • 8,750,000 - 9,000,000 Spirit
  • 9,000,000 - 9,360,000 Jason Scott/Underscor/Ymgve

Phase Two: Video Acquisition

"Oh my god, some of these videos!"

Once the hotlists of who has videos have been grabbed (and that is going to happen rather shortly, thanks to the army of folks), we'll be running a program called grapefruit (the g and fruit are silent, ræp) that yanks down all the videos from a given user. The number of users with video is a tiny percentage of the total users (we're seeing rates of less than 1 video-keeping user for every 10-20 of the users, and less), it will still be significant diskspace. It's video, after all. After an initial test run of grapefruit, we'll start handing out numbers and hitlists to people, and they'll begin downloading.

We have no idea where we're going to put this. We're going to need a bigger boat.

If you used axel / grapefruit-axel, please tell soultcer or guan or underscor on IRC. Axel doesn't store timestamps, thus we try to add that data manually, but we need to know who used axel and who didn't.

Uploaded to Underscor, needs factchecking

Downloading from rsync.net

  • 3,400,000 - 3,500,000 (4,560 Users) NovaKing 4363

Uploaded to rsync.net

  • 2,400,000 - 2,500,000 (2,802 Users) NovaKing 4363
  • 3,300,000 - 3,400,000 (4,252 Users) NovaKing 4363
  • 3,700,000 - 3,800,000 (4,262 Users) NovaKing 4363
  • 6,200,000 - 6,210,039 ( 738 Users) Pneu 4363
  • 6,210,039 - 6,250,000 (2,609 Users) NovaKing 4363
  • 6,250,000 - 6,300,000 (3,109 Users) NovaKing 4363
  • 8,200,000 - 8,300,000 (3,335 Users) NovaKing 4363

Uploading

Please try to keep this box up to date.

  • 0,075,000 - 0,077,255 (1,801 Users, 5,405 videos, 41GB, factchecked) Spirit (to underscor, done, factchecked on underscor's side too, dunno where to move it here so I will leave it)
  • 1,800,000 - 1,900,000 (4,744 Users, 20,494 videos, 169g) bluknight D1
  • 2,900,000 - 3,000,000 (6,221 Users, 260G) dj_ryan 4421
  • 3,600,000 - 3,700,000 (4,494 Users, 21,863 videos, 251gb) chronomex D1
  • 4,100,000 - 4,200,000 (4,511 users, 25,383 videos, ~ 376gb) penemue (via NovaKing) 4363
  • 5,150,000 - 5,200,000 (3,313 Users, 18,127 videos, 235gb) Guan (to underscor, paused)
  • 5,400,000 - 5,500,000 (6,897 Users, 738gb) dj_ryan 4421
  • 6,300,000 - 6,400,000 (6,378 Users, 44,962 videos, 849gb) Guan (to underscor)
  • 6,500,000 - 6,600,000 (6,588 Users) NovaKing 4363
  • 6,650,000 - 6,666,477 (1,000 Users, 5,470 videos, 109gb ) sdboyd D1
  • 7,500,000 - 7,600,000 (4,347 Users, 20,418 videos, 240gb) Guan 4350
  • 7,800,000 - 7,900,000 (3,720 Users) NovaKing 4363
  • 8,000,000 - 8,100,000 (3,416 Users, 16,781 videos, 201gb) bluknight D1
  • 8,600,000 - 8,700,000 (3,863 Users, 20,391 videos, 241gb) Joey rsync.net


Finished

  • 0,000,000 - 0,005,000 (3,960 Users, x videos, xgb) Underscor
  • 0,005,000 - 0,010,000 (3,993 Users) Underscor
  • 0,020,000 - 0,025,000 (4,052 Users) guan
  • 0,025,000 - 0,030,000 (4,040 Users) nwf
  • 0,030,000 - 0,035,000 (4,095 Users, ? videos, ~ 180-210gb) penemue (uploading to Novaking)
  • 0,035,000 - 0,040,000 (3,966 Users, ? videos, 234gb) penemue
  • 0,045,000 - 0,050,000 (4,051 Users, 14133 videos, 131gb) Jason Scott (Ersi running cleanup)
  • 0,055,000 - 0,060,000 (4,086 Users, 13098 videos, 107GB) Lambda_Dr (factcheck complete, timestamps updated)
  • 0,067,800 - 0,070,000 (1,792 Users, 9725 Videos, 100gb) penemue
  • 0,074,000 - 0,075,000 (801 Users, 5376 videos, 28 gb) widget (unavailable for upload until wed 16th)
  • 0,077,256 - 0,079,747 (2,000 Users, 10,603 videos, 102gb) Guan
  • 0,082,200 - 0,083,321 (1,000 Users, 6,536 videos, 60gb) Guan
  • 0,083,322 - 0,100,003 (5,000 Users, 34,236 videos, 324gb) Guan
  • 0,100,000 - 0,200,000 (6,247 Users, x videos, 159gb) Underscor
  • 0,200,000 - 0,300,000 (6,461 Users, x videos, 141gb) Underscor
  • 0,400,000 - 0,500,000 (6,231 Users, x videos, 159gb) Underscor
  • 0,750,000 - 0,800,000 (3,735 Users, 19,032 videos, 136G) atg
  • 0,800,000 - 0,850,000 (3,140 Users, 10,969 videos, 104G) crust
  • 0,850,000 - 0,900,000 (3,749 Users, 18,562 videos, 138gb) filer
  • 1,000,000 - 1,100,000 (6,331 Users, 31,241 videos, 247gb) Zumthing (original download + factcheck cleanup)
  • 1,400,000 - 1,500,000 (5,194 Users, 6,476 videos, 68gb) beardicus
  • 1,600,000 - 1,700,000 (4,933 Users, 8250 videos, 82g) Masterono
  • 1,700,000 - 1,800,000 (4,368 Users, 8013 videos, 97G) atg
  • 1,800,000 - 1,900,000 (4,744 Users, 20,494 videos, 169g) bluknight
  • 2,200,000 - 2,300,000 (4,216 Users, 18,708 videos 168G) jakemcgra
  • 2,300,000 - 2,400,000 (4,615 Users, 20,944 videos, 216gb) DoubleJ (factchecked, ready to upload)
  • 2,500,000 - 2,600,000 (3,949 Users, 15,994 videos, 146gb) Zumthing (original download + factcheck cleanup)
  • 2,600,000 - 2,700,000 (4,208 Users, 17,903 videos, 183G) Frigolit (not factchecked yet)
  • 2,700,000 - 2,800,000 (4,754 Users, 22,600 videos, 192gb) ersi (timestamps: ok)
  • 3,100,000 - 3,200,000 (5,760 Users, 15,661 videos, 186gb) Masterono
  • 3,200,000 - 3,250,000 (2,603 Users) Soultcer (timestamps: ok, utf8: ok, factcheck done)
  • 3,250,000 - 3,300,000 (2,439 Users) taaz
  • 3,800,000 - 3,900,000 (3,936 Users, 22,414 videos, 312gb) dysproseu (factchecked)
  • 4,000,000 - 4,100,000 (4,251 Users, 26,591 videos, 342gb) primus104 (factcheck complete, timestamps updated)
  • 4,500,000 - 4,600,000 (3,434 Users, 22,645 videos, 324gb) DoubleJ (factchecked, ready to upload)
  • 4,785,016 - 4,800,000 ( 710 Users, 4224 videos, 51GB) primus_102 (factcheck complete, timestamps updated)
  • 5,100,000 - 5,150,000 ((3,254 Users, roughly 300GB) Spirit (upload to underscor paused, currently going over missing-hotlist.txt)
  • 5,255,391 - 5,299,993 (3,000 users, 23,847 videos, 330gb ) penemue
  • 5,300,000 - 5,400,000 (6,289 Users, 36,617 videos, 556gb) jakemcgra
  • 5,850,615 - 5,865,958 (1,000 Users, 6,354 videos, 133G) DOubleJ cut from 58xx range, done and factchecked
  • 5,882,964 - 5,894,410 (~725 users, ? videos, 119.5gb) penemue
  • 5,932,108 - 5,944,802 (~750 users, ? videos, 68.7GB) SudoSeth
  • 6,600,000 - 6,650,000 (3,516 Users, 12,404 videos, 277GB) Lambda_Dr (timestamps updated, factcheck complete)
  • 6,650,000 - 6,700,000 (2049 users, 15,021 videos, 293gb) jakemcgra
  • 6,750,000 - 6,800,000 (2,838 Users, 17,555 videos, 356gb) Jason Scott (Transferring to data dump 1004)
  • 7,000,000 - 7,100,000 (4,833 Users, 24,523 videos, 311gb) bluknight
  • 7,100,000 - 7,200,000 (4,026 Users, 13,244 videos, 167gb) zumthing
  • 7,200,000 - 7,300,000 (3,939 Users, 21,731 videos, 292G) dysproseu (factchecked)
  • 7,400,000 - 7,500,000 (4,204 Users, 22,560 videos, 290gb) ZPedro
  • 7,700,000 - 7,800,000 (4,028 Users, 15,037 videos, 171gb) K4K
  • 7,700,000 - 7,800,000 (4,028 Users, 19,155 videos, 219gb) jakemcgra
  • 7,900,000 - 8,000,000 (3,517 Users, 10,392 videos, 111gb) matzie
  • 8,100,000 - 8,200,000 (3,380 Users, 13,882 videos. 160gb) Primus104 (Transferring to data dump 4362)
  • 8,300,000 - 8,400,000 (2,920 Users, 8,973 videos, 93gb) Masterono
  • 8,400,000 - 8,500,000 (3,535 Users, 16,784 videos, 199gb) yoyoguy2
  • 8,700,000 - 8,800,000 (3,730 Users, 16,698 videos, 186gb) t0phux
  • 8,800,000 - 8,850,000 (2,170 Users, 8,368 videos, 71gb ) sdboyd D1 (Finished uploading)
  • 8,850,000 - 9,000,000 (1,770 Users, 8,832 videos, 106gb) Gandalfar (uploading to 4482)
  • 9,000,000 - 9,050,000 (1,745 Users, 6,889 videos, 86G) atg
  • 9,050,000 - 9,100,000 (1,796 Users, 6,142 videos, 79gb) Masterono
  • 9,150,000 - 9,200,000 (5,173 Users, 22,655 videos, 261gb) Underscor
  • 9,200,000 - 9,300,000 (8,941 Users, 30,179 videos, 341gb) Zumthing (overlaps with drspangle)
  • 9,300,000 - 9,360,000 (2,179 Users, 4,406 videos, 51gb) Robertcu

In-Process

  • "OPEN" = unassigned, hotlist available

NOTE that the actual ranges FROM 0-083323 are totally WRONG AND MESSED UP. underscore will fix them later. (d'oh, oops)

ksuter is doing a rough dupe of 47-100003, but do not count his effort in assigning ranges.

  • 0,010,000 - 0,015,000 (4,055 Users) soultcer
  • 0,015,000 - 0,020,000 (3,948 Users) jakemcgra (responsible for 15K - 17K, will allow it to get as far as possible though)
  • 0,040,000 - 0,045,000 (4,024 Users) PMT
  • 0,050,000 - 0,055,000 (3,976 Users) zumthing
  • 0,070,000 - 0,075,000 (3,231 Users) underscor (my part is done, waiting for guan's bit) (guan: 73,002-73,999 and 74,001-74,999)
  • 0,079,748 - 0,082,199 (2,000? Users) ksuter
  • 0,600,000 - 0,650,000 (x Users) robertcu
  • 0,650,000 - 0,700,000 (3,162 Users) Hydruh
  • 0,900,000 - 1,000,000 (6,453 Users) K4K
  • 1,200,000 - 1,300,000 (7,130 Users) drspangle
  • 1,300,000 - 1,400,000 (7,598 Users) sep332
  • 1,900,000 - 2,000,000 (4,481 Users) ajclement
  • 2,800,000 - 2,900,000 (5,623 Users) amuck
  • 3,500,000 - 3,600,000 (4,895 Users) tobbez (3589107-3599985 nwf; finished)
  • 3,900,000 - 4,000,000 (4,055 Users) HA
  • 4,200,000 - 4,300,000 (4,250 Users) atg
  • 4,300,000 - 4,400,000 (4,355 Users) NovaKing
  • 4,400,000 - 4,500,000 (4,715 Users) beardicus
  • 4,600,000 - 4,700,000 (5,151 Users) eberon
  • 4,700,000 - 4,750,000 (2,665 Users) Gandalfar
  • 4,750,000 - 4,784,985 (1,970 Users) 2021 done,13856 videos,174gb done so far by crust, penemue taking over the last 400 users with 250 left for crust)
  • 4,800,000 - 4,900,000 (5,764 Users) monst3r_
  • 4,900,000 - 5,000,000 (6,738 Users) jch
  • 5,000,000 - 5,100,000 (6,545 Users) monst3r_
  • 5,200,029 - 5,255,382 (3,751 Users) jakemcgra (orig hotlist w. duplicates, unsorted, fixed list: http://pastie.org/1668619)
  • 5,600,000 - 5,700,000 (5,903 Users) _kp
  • 5,700,000 - 5,800,000 (3,194 Users) filer
  • 5,800,000 - 5,900,000 (6,188 Users) jch/DoubleJ/Spirit/jakemcgra (DoubleJ, Spirit with the last 3000 users (pretty much done, going over the missing-hotlist.txt), Guan taking 500 users 5,835,908-5,842,902)
  • 5,900,000 - 5,974,072 (5,179 Users) km3k/amerrykan (km3k will do first 700 users in hotlist, amerrykan rest - he's was at 15% before splitting);
  • 5,944,810 - 5,954,407 (750 Users) penemue (split from above)
  • 5,954,430 - 5,974,072 (1,500 Users) DoubleJ (split from above)
  • 5,974,076 - 5,999,982 (2,000 Users) nwf
  • 6,000,000 - 6,050,000 (3,571 Users) savetz / uploading to 4408
  • 6,050,000 - 6,100,000 (3,747 Users) t0phux
  • 6,100,000 - 6,200,000 (7,433 Users) incendui (Guan taking 6183285-6199997 and 6169448-6183223, 2,000 users *done*)(Lambda_Dr taking 6156334-6169434)
  • 6,400,000 - 6,500,000 (6,097 Users) Masterono
  • 6,700,000 - 6,750,000 (3,079 Users) yoyoguy2
  • 6,800,000 - 6,900,000 (5,234 Users) dashcloud
  • 6,900,000 - 7,000,000 (5,676 Users) bluknight (sandra)
  • 7,300,000 - 7,400,000 (4,304 Users) amerrykan
  • 7,500,000 - 7,600,000 (4,347 Users) sadcarrot
  • 8,500,000 - 8,600,000 (3,516 Users) red_cell
  • 8,900,000 - 9,000,000 (3,793 Users) khaoohs
  • 9,100,000 - 9,150,000 (1,989 Users) Killface
  • 9,250,000 - 9,300,000 (2,835 Users) drspangle (overlaps with Zumthing)
  • freaks (1,163 Users): SudoSeth from top, Guan from bottom

Calling the Cleanup Crew

Yahoo!: We can't afford storage, but we can afford Zambonis

In this, the last days of the Yahoo! Video download, we now have people who are downloading but are at less than 50% done, and others who are done but who encountered UTF-8 errors. We need the cleanup crew to take over pieces. If you're available with a machine that can handle UTF-8, please put your name here and also add your name to the list above if you're running cleanup. The easiest way to help a cleanup crew member is to run the factcheck script to determine which users you weren't able to download, and e-mail cleanup crew members with your missing-hotlist.txt file. This is where it gets weird and fast! Let's go!

Available:

Cleanup in progress

  • ersi: downloaded 414 users, who have 5544 videos. Size: 47G | I am working on users 45051-49992 and have finished 29.200% (414/1413)
  • crust: 4,750,000+ (315 users) Guan

Cleanup done

  • Lambda_Driver, 55000+ (114 users, 897 videos downloaded, 731 existing, 943 notitle) Guan
  • crust, 800,000+ (415 users, 2,965 videos downloaded, 4,416 existing, 2,174 notitle) Guan
  • 2,700,000+ (871 users, 29 videos downloaded, 5,960 existing, 3,949 notitle) Guan
  • DoubleJ, 2,300,000+ (1,069 users, 0 videos downloaded, 3,191 existing, 5,024 notitle) Guan
  • DoubleJ, 4,500,000+ (514 users, 18 videos downloaded, 3,312 existing, 1,929 notitle) Guan
  • DoubleJ: 5,852,462 - 5,864,456 (19 users) Guan
  • SudoSeth: 5,932,108 - 5,944,802 (21 users) Guan
  • Lambda_Dr, 6,600,000+ (23 users, 78 videos downloaded, 0 existing, 118 notitle) Guan
  • ZPedro, 7,400,000+ (19 users, 192 videos downloaded, 0 existing, 42 notitle) Guan

UTF-8 Errors What

Some of the filenames in Yahoo! are encoded in Unicode, which flips out on some machines. If you run grapefruit on user 45002 and see errors like "UnicodeEncodeError: 'ascii' codec can't encode characters", then you have this issue. At this point in time, we can't use you, but we love you anyway - come cheer us on.

Phase Three: Final Resting Place

In an ideal world, all the downloaders would hold copies and everyone would just torrent the thing. That's not going to happen in current contemporary machine capacity. Our very rough estimate is that a download of Yahoo! Video is going to be 20 terabytes. While this is under a thousand dollars of newegg-purchased disks, it's still a bit much to ask of anyone. Meanwhile, the more forward-looking of volunteers are asking, "what's the exit strategy?".

Jason and a few others have locations that video downloaders can upload a copy of their acquired data to; these are being placed on a stack of hard drives. At the same time, Archive.org is providing a location for the data to be uploaded to as well. Other entities are being contacted at the present about providing space for copies of the full archive, as well.

The priority, however, is to get all the data in the first place.

Verified and On-Drive

Talk to underscor to get your stuff uploaded. (I will perform a final check of the timestamps)

Set 1 (Disks 1 and 2)

Being used as upload holding pen at the moment

Set 2 (Disks 3 and 4)

  • 0,300,000 - 0,400,000 (6,909 Users, 41,131 videos, 342gb) Underscor
  • 0,500,000 - 0,600,000 (6,543 Users, 13,849 videos, 115gb) NovaKing
  • 0,700,000 - 0,750,000 (3,213 Users, 20,767 videos, 127gb) Underscor
  • 1,500,000 - 1,600,000 (5,252 Users, 7,642 videos, 86g) NovaKing
  • 2,100,000 - 2,200,000 (4,519 Users, 16,597 videos, 144g) Joey
  • 3,000,000 - 3,100,000 (6,297 Users, 18,418 videos, 163gb) Guan
  • 7,600,000 - 7,700,000 (4,200 Users, 17,885 videos, 189gb) NovaKing

Set 3 (Disks 5 and 6)

  • 0,060,000 - 0,065,000 (2,226 Users) underscor
  • 0,065,000 - 0,067,800 (1,791 Users) underscor
  • 1,100,000 - 1,200,000 (7,613 Users, 12,892 videos, 177gb ) NovaKing
  • 2,000,000 - 2,100,000 (4,408 Users, 20,583 videos, 167g) dashcloud

Set 4 (Disks 7 and 8)

Set 5 (Disks 9 and 10)

Disk 10 en shipment

Set 6 (Disks 11 and 12)

Disks still need purchasing

Set 7 (Disks 13 and 14)

Disks still need purchasing

Set 8 (Disks 15 and 16)

Disks still need purchasing

Set 9 (Disks 17 and 18)

Disks still need purchasing

Set 10 (Disks 19 and 20)

Disks still need purchasing


Statistics

"Why didn't more people use it! It was awesome!"

People like easy to quote statistics from a project, so here are some, with others coming soon.

Yahoo! Video officially started as a user content system in June of 2006. Previously, it had been a directory of other video sites around the net (harkening back to Yahoo's previous life as a directory instead of its short burst as a content creator). From the start, it was called a "YouTube Clone" and other derisive terms.

However, users did decide to go for this new environment. Archive Team found about 9,360,000 accounts in use on Yahoo Video. "In Use" is a relative term, of course - many, many users consist of no uploaded videos, no user picture, and no other activity - just a name. We'll have more, but for example, of 1.5 million accounts we've scanned so far, only 146,600 have uploaded videos, and if you shear off the first 100,000 (i.e., frontloaded with content partner) accounts, that 10% figure drops down to 5%.

It didn't take very long for users to figure out Yahoo! Video is not the way to go, either as participants or as viewers. The most popular video, "Baby Raccoons Saved from Soda Machine" has 14 million views. Compare with one of the top videos on YouTube - Justin Beiber performing "Baby" with Ludacris, which has 457 million views. By the time the raccoons are viewed, the next most popular video, "Worst Best Man Ever", clocks in at just over 10 million views. After the first 20 videos, the views per video are down to about 4 million. By the time we get to the top 100, we're already skirting 1 million. Again, a single video on Youtube, the most popular, has a number of views basically the same as the top 100 Yahoo! videos combined.