Yahoo! Video Warroom

From Archiveteam
Jump to navigation Jump to search


Yahoovideo.jpg

Phase One: Username Acquisition (Complete)

"You people are working so hard!"

Summary

In roughly 30 days, 12 Archive Team members scanned 9,360,000 Yahoo! Video user account numbers. Of these, 9,218,844 were actual user accounts (others were deleted or "networks", i.e. playlists). Of these, 555,502 had actually taken the time to upload one or more videos. All told, we showed that 3,585,981 videos were uploaded to Yahoo! Video.

  • 0,000,000 - 1,000,000 Jason Scott, Chronomex
  • 1,000,000 - 2,250,000 Joey/Underscor
  • 2,250,000 - 2,500,000 Underscor/Chronomex
  • 2,500,000 - 3,500,000 Soultcer/Underscor
  • 3,500,000 - 4,500,000 Chronomex
  • 4,500,000 - 5,000,000 Underscor
  • 5,000,000 - 5,500,000 Jason Scott/Spirit
  • 5,500,000 - 5,750,000 Ymgve
  • 5,750,000 - 5,900,000 guan/Jason Scott
  • 5,900,000 - 6,000,000 soultcer
  • 6,000,000 - 6,050,000 Underscor/NovaKing
  • 6,050,000 - 6,150,000 robzy
  • 6,150,000 - 6,200,000 phuz
  • 6,200,000 - 6,250,000 guan
  • 6,250,000 - 6,750,000 phuz/NovaKing
  • 6,750,000 - 7,500,000 chronomex/NovaKing
  • 7,500,000 - 8,000,000 Jason Scott
  • 8,000,000 - 8,250,000 Spirit
  • 8,250,000 - 8,750,000 Sungo
  • 8,750,000 - 9,000,000 Spirit
  • 9,000,000 - 9,360,000 Jason Scott/Underscor/Ymgve

Phase Two: Video Acquisition

"Oh my god, some of these videos!"

Once the hotlists of who has videos have been grabbed (and that is going to happen rather shortly, thanks to the army of folks), we'll be running a program called grapefruit that yanks down all the videos from a given user. The number of users with video is a tiny percentage of the total users (we're seeing rates of less than 1 video-keeping user for every 10-20 of the users, and less), it will still be significant diskspace. It's video, after all. After an initial test run of grapefruit, we'll start handing out numbers and hitlists to people, and they'll begin downloading.

If you used axel / grapefruit-axel, please tell soultcer or guan or underscor on IRC. Axel doesn't store timestamps, thus we try to add that data manually, but we need to know who used axel and who didn't.

Downloading from rsync.net

  • 0,400,000 - 0,500,000 (6,231 Users, 36,205 videos, 336gb) amnesia (YV_10)
  • 6,200,000 - 6,300,000 (YV_10)
    • 6,200,000 - 6,210,039 ( 738 Users) Pneu 4363
    • 6,210,039 - 6,250,000 (2,609 Users) NovaKing 4363
    • 6,250,000 - 6,300,000 (3,109 Users) NovaKing 4363
  • 6,500,000 - 6,600,000 (6,588 Users, 22,413 videos, 492gb) NovaKing 4363 (YV_10)
  • 7,500,000 - 7,600,000 (4,347 Users, 20,418 videos, 240gb) Guan 4350 (YV_10)

Tons of stuff from penemue... will figure out exactly what tomorrow. Underscor 05:49, 22 March 2011 (UTC)

Uploaded to rsync.net

  • 0,100,000 - 0,182,337 (5,131 Users, 28,741 videos, 268gb) Joey 4362
  • 0,200,000 - 0,300,000 (6,461 Users, 33,594 videos, 268GB) DoubleJ 4362
  • 2,700,000 - 2,800,000 (4,754 Users, 22,600 videos, 192gb) ersi/Guan
  • 4,100,000 - 4,200,000 (4,511 users, 29,154 videos, 360gb) penemue (via NovaKing) 4363
  • 4,558,113 - 4,574,989 (771 Users, 4,054 videos, 63gb) DoubleJ 4362
  • 5,869,902 - 5,882,934 Spirit 4406
  • 7,800,000 - 7,900,000 (3,720 Users, 11,322 videos, 128gb) NovaKing 4363
  • 8,200,000 - 8,300,000 (3,335 Users, 9,269 videos, 100gb) NovaKing 4363
  • 8,600,000 - 8,700,000 (3,863 Users, 20,391 videos, 241gb) Joey 1005
  • freaks2 (1,841 Users, 3,731 videos, 38gb) DoubleJ 4362

Uploaded to flophouse/blindtiger

  • 0,010,000 - 0,015,000 (4,055 Users) soultcer gv_7@bt
  • 2,300,000 - 2,400,000 (4,615 Users, 20,944 videos, 216gb) DoubleJ/Guan gv_12@bt
  • 4,000,000 - 4,100,000 (4,251 Users, 26,591 videos, 342gb) primus104/102 gv21@bt
  • 4,785,016 - 4,800,000 ( 710 Users, 4224 videos, 51GB) primus_104/102 gv20@bt

Uploaded to batcave

  • 4,500,000 - 4,600,000 (3,434 Users, 22,645 videos, 324gb) DoubleJ/Guan
  • 4,558,113 - 4,586,425 (770 Users from rescrape)
  • 5,850,615 - 5,865,958 (1,000 Users, 6,354 videos, 133G) DoubleJ/Guan cut from 58xx range

Uploading

Please try to keep this box up to date.

  • 0,650,000 - 0,700,000 (3,162 Users, 17,846 videos, 147gb) Hydruh 4362
  • 1,600,000 - 1,700,000 (4,933 Users, 8250 videos, 82g) Masterono 4362
  • 1,800,000 - 1,900,000 (4,744 Users, 20,494 videos, 169g) bluknight D1
  • 2,900,000 - 3,000,000 (6,221 Users, 260G) dj_ryan 4421
  • 3,100,000 - 3,200,000 (5,760 Users, 15,661 videos, 186gb) Masterono 4362
  • 4,800,000 - 4,900,000 (5,754 Users, 15,759 videos, 483gb) monst3r (Might be just part of 50x-51x) 4363
  • 5,000,000 - 5,100,000 (6,545 Users, 32,041 videos, 428gb) monst3r_ 4363
  • 5,150,000 - 5,200,000 (3,313 Users, 18,127 videos, 235gb) Guan (to underscor, paused)
  • 5,400,000 - 5,500,000 (6,897 Users, 738gb) dj_ryan 4421
  • 5,954,430 - 5,963,115 (700 Users) DoubleJ (split from km3k/amerrykan)
  • 6,300,000 - 6,400,000 (6,378 Users, 44,962 videos, 849gb) Guan (to underscor)
  • 6,400,000 - 6,500,000 (6,097 Users, 32,239 videos, 654GB) Masterono
  • 7,400,000 - 7,500,000 (4,204 Users, 22,560 videos, 290gb) ZPedro (to 4362)
  • 8,000,000 - 8,100,000 (3,416 Users, 16,781 videos, 201gb) bluknight D1
  • 8,300,000 - 8,400,000 (2,920 Users, 8,973 videos, 93gb) Masterono
  • 9,050,000 - 9,100,000 (1,796 Users, 6,142 videos, 79gb) Masterono

Finished

  • 0,015,000 - 0,020,000 (3,948 Users, 24,328 videos, 314gb) jakemcgra
  • 0,025,000 - 0,030,000 (4,040 Users) nwf
  • 0,030,000 - 0,035,000 (4,095 Users, 25,490 videos, 209gb) penemue/NovaKing
  • 0,040,000 - 0,045,000 (4,024 Users) PMT
  • 0,045,000 - 0,050,000 (4,051 Users, 31,110 videos, 288gb) Jason Scott/Ersi
  • 0,050,000 - 0,055,000 (3,976 Users) zumthing
  • 0,055,000 - 0,060,000 (4,086 Users, 13098 videos, 107GB) Lambda_Dr/Guan
  • 0,074,000 - 0,075,000 (801 Users, 5376 videos, 28 gb) widget (unavailable for upload until wed 16th)
  • 0,077,256 - 0,079,747 (2,000 Users, 10,603 videos, 102gb) Guan
  • 0,079,748 - 0,082,199 (2,000? Users) ksuter
  • 0,082,200 - 0,083,321 (1,000 Users, 6,536 videos, 60gb) Guan
  • 0,600,000 - 0,650,000 (3,398 Users, 16,861 videos, 182gb) robertcu (guan re-downloaded)
  • 0,650,000 - 0,700,000 (3,162 Users, 17,846 videos, 147gb) Hydruh
  • 0,750,000 - 0,800,000 (3,735 Users, 19,032 videos, 136G) atg
  • 0,800,000 - 0,850,000 (3,140 Users, 10,969 videos, 104G) crust/Guan
  • 0,850,000 - 0,900,000 (3,749 Users, 18,562 videos, 138gb) filer
  • 0,900,000 - 1,000,000 (6,453 Users, 18,077 videos, 139gb) K4K/NovaKing
  • 1,000,000 - 1,100,000 (6,331 Users, 31,241 videos, 247gb) Zumthing (original download + factcheck cleanup)
  • 1,200,000 - 1,300,000 (7,130 Users) drspangle (finished, guan doing factcheck)
  • 1,300,000 - 1,400,000 (7,598 Users) sep332
  • 1,400,000 - 1,500,000 (5,194 Users, 6,476 videos, 68gb) beardicus
  • 1,700,000 - 1,800,000 (4,368 Users, 8013 videos, 97G) atg
  • 1,800,000 - 1,900,000 (4,744 Users, 20,494 videos, 169g) bluknight
  • 2,200,000 - 2,300,000 (4,216 Users, 18,708 videos 168G) jakemcgra
  • 2,500,000 - 2,600,000 (3,949 Users, 15,994 videos, 146gb) Zumthing (original download + factcheck cleanup)
  • 2,600,000 - 2,700,000 (4,208 Users, 17,903 videos, 183G) Frigolit (utf8: ok, factcheck: ok)
  • 2,700,000 - 2,800,000 (4,754 Users, 22,600 videos, 192gb) ersi/Guan
  • 2,800,000 - 2,900,000 (5,623 Users, 23,879 videos, 249gb) amuck
  • 3,200,000 - 3,250,000 (2,603 Users) Soultcer (timestamps: ok, utf8: ok, factcheck done)
  • 3,250,000 - 3,300,000 (2,439 Users) taaz
  • 3,500,000 - 3,600,000 (4,895 Users, 22,964 videos, 254gb) tobbez/nwf
  • 3,800,000 - 3,900,000 (3,936 Users, 22,414 videos, 312gb) dysproseu (factchecked)
  • 3,900,000 - 4,000,000 (4,055 Users, 11,292 videos, 136gb ) HA/Novaking
  • 4,200,074 - 4,289,987 (4,250 Users) atg
    • 4,224,414 - 4,269,902 (? Users) penemue
    • 4,270,007 - 4,279,957 (438 Users, 2,477 videos, 28gb) guan
    • 4,280,010 - 4,289,987 (406 Users) nwf
  • 4,290,000 - 4,300,000 (456 Users, 2,030 videos, 38gb) savetz/penemue
  • 4,300,000 - 4,400,000 (4,355 Users, 11,947 videos, 157gb) NovaKing
  • 4,400,000 - 4,500,000 (4,715 Users, 21,569 videos, 299G) beardicus
  • 4,557,501 - 4,586,425 (1,713 Users) Lambda_dr (email dsnigier at gmail.com with uploading details...can keep on drive array for as long as needed)
  • 4,700,000 - 4,750,000 (2,665 Users, 8,229 videos, 104gb) Gandalfar/NovaKing
  • 4,750,000 - 4,784,985 (2,127 Users, 15,265 videos, 193gb) crust/Guan
  • 4,790,761 - 4,799,994 (400 Users, 5648 videos & html files, 26.8 gb) penemue (overlap with primus_102)
  • 4,800,000 - 4,900,000 (5,764 Users) jch (Downloaded, factchecking)
  • 4,900,000 - 4,913,495 (1,000 Users) nwf
  • 4,913,497 - 4,937,299 (1,500 Users) guan DONE (1,501 users, 9,030 videos, 113gb)
  • 4,937,299 - 4,945,193 (500 Users, 3411 videos, 41gb) yoyoguy2 DONE
  • 4,966,514 - 4,979,997 (1000 Users, 9680 Videos, 111gb) penemue DONE
  • 4,980,027 - 4,994,982 (780 Users, 11,375 html+flv, 61,9gb) penemue DONE
  • 4,995,045, - 4,999,999 (364 Users, 3293 videos, 33gb) penemue DONE
  • 5,255,391 - 5,299,993 (3,000 users, 23,847 videos, 330gb ) penemue
  • 5,280,381 - 5,291,717 (749 Users, 6,540 videos, 92gb) Guan
  • 5,200,000 - 5,300,000 (6,751 Users, 54,805 videos, 734gb) jakemcgra
  • 5,300,000 - 5,400,000 (6,289 Users, 36,617 videos, 556gb) jakemcgra
  • 5,800,020 - 5,899,977 (6,188 Users) jch/DoubleJ/Spirit/jakemcgra
    • 5,842,906 - 5,850,604 (500 Users, 4,663 videos 91gb) jakemcgra
    • 5,882,964 - 5,894,410 (~725 users, ? videos, 119.5gb) penemue
    • 5,894,410 - 5,899,977 (? Users, ~ 45gb) penemue done
  • 5,900,005 - 5,999,982
    • 5,900,005 - 5,912,492 (700 users, 4133 videos, 87GB) km3k (confirmed done)
    • 5,912,497 - 5,944,029 (? users) amerrykan, done)
    • 5,932,108 - 5,944,802 (~750 users, ? videos, 68.7GB) SudoSeth/Guan
    • 5,944,810 - 5,954,407 (750 Users) penemue (split from km3k/amerrykan)
    • 5,963,126 - 5,967,009 (300 users) guan (split from DoubleJ's split)
    • 5,967,027 - 5,974,072 (500 Users, 81.9gb) penemue (from DoubleJ's split)
    • 5,974,076 - 5,999,982 (2,000 Users) nwf
  • 6,100,000 - 6,200,000 (7,433 Users) incendui (done, except for 897 user missing-hotlist, which was sent to guan, status? yoyoguy2 also working)
    • 6,169,448 - 6,199,997 Guan (2k users, done)
    • 6,156,334 - 6,169,434 Lamda_Dr (done)
  • 6,600,000 - 6,650,000 (3,516 Users, 12,404 videos, 277GB) Lambda_Dr/Guan (timestamps updated, factcheck complete)
  • 6,650,000 - 6,666,477 (1,000 Users, 5,470 videos, 109gb) sdboyd
  • 6,650,000 - 6,700,000 (2049 users, 15,021 videos, 293gb) jakemcgra
  • 6,700,000 - 6,750,000 (3,079 Users, 23,950 videos, 510gb) yoyoguy2
  • 6,750,000 - 6,800,000 (2,838 Users, 17,555 videos, 356gb) Jason Scott (Transferring to data dump 1004)
  • 6,900,000 - 7,000,000 (5,676 Users) bluknight (sandra)
  • 7,000,000 - 7,100,000 (4,833 Users, 24,523 videos, 311gb) bluknight
  • 7,100,000 - 7,200,000 (4,026 Users, 13,244 videos, 167gb) zumthing
  • 7,200,000 - 7,300,000 (3,939 Users, 21,731 videos, 292G) dysproseu (factchecked)
  • 7,300,000 - 7,400,000 (4,304 Users, 23,904 videos, 307gb) amerrykan done, factchecked
  • 7,700,000 - 7,800,000 (4,028 Users, 15,037 videos, 171gb) K4K
  • 7,700,000 - 7,800,000 (4,028 Users, 19,155 videos, 219gb) jakemcgra
  • 7,900,000 - 8,000,000 (3,517 Users, 10,392 videos, 111gb) matzie
  • 8,100,000 - 8,200,000 (3,380 Users, 13,882 videos. 160gb) Primus104 (Transferring to data dump 4362)
  • 8,400,000 - 8,500,000 (3,535 Users, 16,784 videos, 199gb) yoyoguy2
  • 8,500,000 - 8,600,000 (3,516 Users, 19,858 videos, 200GB) red_cell
  • 8,700,000 - 8,800,000 (3,730 Users, 16,698 videos, 186gb) t0phux
  • 8,850,000 - 9,000,000 (1,770 Users, 8,832 videos, 106gb) Gandalfar (uploading to 4482)
  • 8,900,000 - 9,000,000 (3,793 Users, 10,790 videos, 167gb) khaoohs/NovaKing
  • 9,000,000 - 9,050,000 (1,745 Users, 6,889 videos, 86G) atg
  • 9,100,000 - 9,150,000 (1,989 Users, 9,380 videos, 111gb) Killface/yoyoguy2
  • 9,150,000 - 9,200,000 (5,173 Users, 22,655 videos, 261gb) Underscor
  • 9,200,000 - 9,300,000 (8,941 Users, 30,179 videos, 341gb) Zumthing (drspangle has a duplicate subset)
  • 9,300,000 - 9,360,000 (2,179 Users, 4,406 videos, 51gb) Robertcu
  • freaks (1,163 Users): Guan
  • each 50,000st user from the whole range (112 videos, 1G) Spirit (factchecked and done)
  • 0,000,000 - 0,100,000 ksuter has done a safety-run of this historically important range

In-Process

  • 1,900,000 - 2,000,000 (4,481 Users, 10,346 videos, 79G) ajclement/swebb/NovaKing
  • 4,600,000 - 4,700,000 (5,151 Users) eberon NOC (jakemcgra doing just-in-case grab)
  • 5,600,000 - 5,700,000 (5,903 Users) _kp NOC (Hydruh doing just-in-case grab)
  • 5,700,000 - 5,800,000 (3,194 Users) finished 91.300% (2918/3194) 3/21
  • 6,050,000 - 6,100,000 (3,747 Users) t0phux (NovaKing doing backup grab)

Cleanup in progress

  • Spangle combined missing-hotlist (416 users) Guan

UTF-8 Errors What

Some of the filenames in Yahoo! are encoded in Unicode, which flips out on some machines. If you run grapefruit on user 45002 and see errors like "UnicodeEncodeError: 'ascii' codec can't encode characters", then you have this issue. At this point in time, we can't use you, but we love you anyway - come cheer us on.

Phase Three: Final Resting Place

Yahoo!: We can't afford storage, but we can afford Zambonis

In an ideal world, all the downloaders would hold copies and everyone would just torrent the thing. That's not going to happen in current contemporary machine capacity. Our very rough estimate is that a download of Yahoo! Video is going to be 20 terabytes. While this is under a thousand dollars of newegg-purchased disks, it's still a bit much to ask of anyone. Meanwhile, the more forward-looking of volunteers are asking, "what's the exit strategy?".

Jason and a few others have locations that video downloaders can upload a copy of their acquired data to; these are being placed on a stack of hard drives. At the same time, Archive.org is providing a location for the data to be uploaded to as well. Other entities are being contacted at the present about providing space for copies of the full archive, as well.

The priority, however, is to get all the data in the first place.

Verified and On-Drive

Talk to underscor to get your stuff uploaded. (I will perform a final check of the timestamps)

Set 1 (Disks 1 and 2)

Being used as upload holding pen at the moment

Set 2 (Disks 3 and 4)

  • 0,300,000 - 0,400,000 (6,909 Users, 41,131 videos, 342gb) Underscor
  • 0,500,000 - 0,600,000 (6,543 Users, 13,849 videos, 115gb) NovaKing
  • 0,700,000 - 0,750,000 (3,213 Users, 20,767 videos, 127gb) Underscor
  • 1,500,000 - 1,600,000 (5,252 Users, 7,642 videos, 86g) NovaKing
  • 2,100,000 - 2,200,000 (4,519 Users, 16,597 videos, 144g) Joey
  • 3,000,000 - 3,100,000 (6,297 Users, 18,418 videos, 163gb) Guan
  • 7,600,000 - 7,700,000 (4,200 Users, 17,885 videos, 189gb) NovaKing

Set 3 (Disks 5 and 6)

  • 0,060,000 - 0,065,000 (2,226 Users, 23,254 videos, 228gb) underscor
  • 0,065,000 - 0,070,000 (4,077 Users, 23,777 videos, 223gb) underscor
  • 0,070,000 - 0,072,000 (1,613 Users, 10,341 videos, 98gb) underscor
  • 0,075,000 - 0,077,255 (1,801 Users, 5,405 videos, 41GB) Spirit
  • 1,100,000 - 1,200,000 (7,613 Users, 12,892 videos, 177gb ) NovaKing
  • 2,000,000 - 2,100,000 (4,408 Users, 20,583 videos, 167g) dashcloud
  • 3,400,000 - 3,500,000 (4,560 Users, 11,450 videos, 102gb) NovaKing
  • 5,100,000 - 5,150,000 (3,254 Users, 19701 videos, 327gb) Spirit
  • 8,800,000 - 8,850,000 (2,170 Users, 8,368 videos, 71gb ) sdboyd

Set 4 (Disks 7 and 8)

  • 0,000,000 - 0,005,000 (3,960 Users, 27,357 videos, 268gb) Underscor
  • 0,005,000 - 0,010,000 (3,993 Users, 25,858 videos, 248gb) Underscor
  • 0,035,000 - 0,040,000 (3,966 Users, 24,164 videos, 225gb) penemue
  • 2,400,000 - 2,500,000 (2,802 Users, 6,260 videos, 48gb) NovaKing
  • 3,300,000 - 3,400,000 (4,252 Users, 10,672 videos, 93gb) NovaKing
  • 3,600,000 - 3,700,000 (4,494 Users, 21,863 videos, 251gb) chronomex
  • 3,700,000 - 3,800,000 (4,262 Users, 11,790 videos, 142gb) NovaKing
  • 6,000,000 - 6,050,000 (3,571 Users) savetz

Set 5 (Disks 9 and 10)

  • 0,067,800 - 0,070,000 (1,792 Users, 9725 Videos, 100gb) penemue
  • 0,182,338 - 0,199,999 (1118 users, 5681 videos. 45gb) amnesia
  • 7,500,000 - 7,600,000 (4,347 Users, 22,758 videos, 287gb) sadcarrot/amnesia

Set 6 (Disks 11 and 12)

  • 0,018,500 - 0,020,000 (x users, x videos, xgb) penemue
  • 0,020,000 - 0,025,000 (4,053 Users, 27,844 videos, 328gb) guan
  • 0,083,322 - 0,100,003 (5,000 Users, 34,236 videos, 324gb) Guan
  • 5,835,908 - 5,842,902 (518 users, 3,464 videos, 66gb) Guan
  • 6,800,000 - 6,900,000 (5,234 Users, 26,243 videos, 558gb) dashcloud
  • guan's cleanup files (bits of other ranges)


Set 7 (Disks 13 and 14)

Disks still need purchasing

Set 8 (Disks 15 and 16)

Disks still need purchasing

Set 9 (Disks 17 and 18)

Disks still need purchasing

Set 10 (Disks 19 and 20)

Disks still need purchasing


Statistics

"Why didn't more people use it! It was awesome!"

People like easy to quote statistics from a project, so here are some, with others coming soon.

Yahoo! Video officially started as a user content system in June 2006. Previously, it had been a directory of other video sites around the net (harkening back to Yahoo's previous life as a directory instead of its short burst as a content creator). From the start, it was called a "YouTube Clone" and other derisive terms.

However, users did decide to go for this new environment. Archive Team found about 9,360,000 accounts in use on Yahoo Video. "In Use" is a relative term, of course - many, many users consist of no uploaded videos, no user picture, and no other activity - just a name. We'll have more, but for example, of 1.5 million accounts we've scanned so far, only 146,600 have uploaded videos, and if you shear off the first 100,000 (i.e., frontloaded with content partner) accounts, that 10% figure drops down to 5%.

It didn't take very long for users to figure out Yahoo! Video is not the way to go, either as participants or as viewers. The most popular video, "Baby Raccoons Saved from Soda Machine" has 14 million views. Compare with one of the top videos on YouTube - Justin Beiber performing "Baby" with Ludacris, which has 457 million views. By the time the raccoons are viewed, the next most popular video, "Worst Best Man Ever", clocks in at just over 10 million views. After the first 20 videos, the views per video are down to about 4 million. By the time we get to the top 100, we're already skirting 1 million. Again, a single video on YouTube, the most popular, has a number of views basically the same as the top 100 Yahoo! videos combined.