Difference between revisions of "Yahoo! Video Warroom"

From Archiveteam
Jump to navigation Jump to search
(→‎Phase Three: Final Resting Place: Revised my estimate down)
m (→‎Statistics: typos fixed: June of 2006 → June 2006, Youtube → YouTube)
 
(563 intermediate revisions by 35 users not shown)
Line 1: Line 1:
Planning room for Yahoo! Video.
__NOTOC__
__NOTOC__


[[File:Yahoovideo.jpg]]
[[File:Yahoovideo.jpg]]


== Phase One: Username Acquisition ==
== Phase One: Username Acquisition (Complete) ==


[[File:Lf1.png|right|thumb|"You people are working so hard!"]]
[[File:Lf1.png|right|thumb|"You people are working so hard!"]]


Acquiring the usernames and which have videos is done by the script ''yvscan''. Ask Jason for a copy. The process is somewhat slow, as going too fast gets the '''dreaded 999'''. As a result, we need a large number of people and machines doing the work, so we have to put them here to keep track of who's doing what. The scraper pulls in userpics, a copy of the user page, and generates a userlist.txt of all the accounts it has found.
===Summary===


There are roughly 9,360,000 user accounts on Yahoo Video. (Let us not fool ourselves as to thinking these are real people.) Some are "Networks", i.e. collections of other videos (but not many, maybe 1 out of every 50,000, or less.) Most have no videos at all, but to make a hitlist of videos to download, we need those.
In roughly 30 days, 12 Archive Team members scanned 9,360,000 Yahoo! Video user account numbers. Of these, 9,218,844 were actual user accounts (others were deleted or "networks", i.e. playlists). Of these, 555,502 had actually taken the time to upload one or more videos. All told, we showed that 3,585,981 videos were uploaded to Yahoo! Video.


===When You Finish===
* '''0,000,000 - 1,000,000''' Jason Scott, Chronomex
* '''1,000,000 - 2,250,000''' Joey/Underscor
* '''2,250,000 - 2,500,000''' Underscor/Chronomex
* '''2,500,000 - 3,500,000''' Soultcer/Underscor
* '''3,500,000 - 4,500,000''' Chronomex
* '''4,500,000 - 5,000,000''' Underscor
* '''5,000,000 - 5,500,000''' Jason Scott/Spirit
* '''5,500,000 - 5,750,000''' Ymgve
* '''5,750,000 - 5,900,000''' guan/Jason Scott
* '''5,900,000 - 6,000,000''' soultcer
* '''6,000,000 - 6,050,000''' Underscor/NovaKing
* '''6,050,000 - 6,150,000''' robzy
* '''6,150,000 - 6,200,000''' phuz
* '''6,200,000 - 6,250,000''' guan
* '''6,250,000 - 6,750,000''' phuz/NovaKing
* '''6,750,000 - 7,500,000''' chronomex/NovaKing
* '''7,500,000 - 8,000,000''' Jason Scott
* '''8,000,000 - 8,250,000''' Spirit
* '''8,250,000 - 8,750,000''' Sungo
* '''8,750,000 - 9,000,000''' Spirit
* '''9,000,000 - 9,360,000''' Jason Scott/Underscor/Ymgve


In these instructions, 0_000_000 is the beginning number of your block.
== Phase Two: Video Acquisition ==


Zip up the files you downloaded:
[[File:Lf3.png|right|thumb|"Oh my god, some of these videos!"]]
 
tar cjvf 0_000_000-to-0_050_000.tar.bz2 userpics userpages


Name the userlist.txt and networklist.txt (if it exists) to something unique, then compress:
Once the hotlists of who has videos have been grabbed (and that is going to happen rather shortly, thanks to the army of folks), we'll be running a program called ''grapefruit'' that yanks down all the videos from a given user. The number of users with video is a tiny percentage of the total users (we're seeing rates of less than 1 video-keeping user for every 10-20 of the users, and less), it will still be significant diskspace. It's video, after all. After an initial test run of grapefruit, we'll start handing out numbers and hitlists to people, and they'll begin downloading.


mv userlist.txt userlist_0_000_000.txt
If you used axel / grapefruit-axel, please tell soultcer or guan or underscor on IRC. Axel doesn't store timestamps, thus we try to add that data manually, but we need to know who used axel and who didn't.
mv networklist.txt networklist_0_000_000.txt


bzip2 userlist_0_000_000.txt
=== Downloading from rsync.net ===
  bzip2 networklist_0_000_000.txt
* '''0,400,000 - 0,500,000''' (6,231 Users, 36,205 videos, 336gb) amnesia (YV_10)
* '''6,200,000 - 6,300,000''' (YV_10)
** '''6,200,000 - 6,210,039''' ( 738 Users) Pneu 4363
** '''6,210,039 - 6,250,000''' (2,609 Users) NovaKing 4363
** '''6,250,000 - 6,300,000''' (3,109 Users) NovaKing 4363
* '''6,500,000 - 6,600,000''' (6,588 Users, 22,413 videos, 492gb) NovaKing 4363 (YV_10)
* '''7,500,000 - 7,600,000''' (4,347 Users, 20,418 videos, 240gb) Guan 4350 (YV_10)
Tons of stuff from penemue... will figure out exactly what tomorrow. [[User:Underscor|Underscor]] 05:49, 22 March 2011 (UTC)


Email the user and network lists to { jason at textfiles.com }.  They should take up just a few megabytes in total.  If you have a website with enough space, please put the .tar.bz2 there and include the url in the mail.  Otherwise, ask for assistance.
=== Uploaded to rsync.net ===


===Summary===
* '''0,100,000 - 0,182,337''' (5,131 Users, 28,741 videos, 268gb) Joey 4362
* '''0,200,000 - 0,300,000''' (6,461 Users, 33,594 videos, 268GB) DoubleJ 4362
* '''2,700,000 - 2,800,000''' (4,754 Users, 22,600 videos, 192gb) '''ersi'''/Guan
* '''4,100,000 - 4,200,000''' (4,511 users, 29,154 videos, 360gb) penemue (via NovaKing) 4363
* '''4,558,113 - 4,574,989''' (771 Users, 4,054 videos, 63gb) DoubleJ 4362
* '''5,869,902 - 5,882,934''' Spirit 4406
* '''7,800,000 - 7,900,000''' (3,720 Users, 11,322 videos, 128gb) NovaKing 4363
* '''8,200,000 - 8,300,000''' (3,335 Users, 9,269 videos, 100gb) NovaKing 4363
* '''8,600,000 - 8,700,000''' (3,863 Users, 20,391 videos, 241gb) Joey 1005
* '''freaks2''' (1,841 Users, 3,731 videos, 38gb) DoubleJ 4362


* '''0,000,000''' - '''4,500,000''' Jason Scott, Chronomex, Joey, Underscor, Soultcer
=== Uploaded to flophouse/blindtiger ===
* '''4,750,000''' - '''5,300,000''' Jason Scott, Underscor
* '''0,010,000 - 0,015,000''' (4,055 Users) soultcer gv_7@bt
* '''5,500,000''' - '''6,000,000''' Ymgve, Underscor
* '''2,300,000 - 2,400,000''' (4,615 Users, 20,944 videos, 216gb) DoubleJ/Guan gv_12@bt
* '''7,500,000''' - '''9,360,000''' Underscor, Jason Scott, Spirit, Sungo
* '''4,000,000 - 4,100,000''' (4,251 Users, 26,591 videos, 342gb) primus104/102 gv21@bt
* '''4,785,016 - 4,800,000''' (  710 Users, 4224 videos, 51GB) primus_104/102 gv20@bt


===Finished===
=== Uploaded to batcave ===
* '''4,500,000 - 4,600,000''' (3,434 Users, 22,645 videos, 324gb) DoubleJ/Guan
* '''4,558,113 - 4,586,425''' (770 Users from rescrape)
* '''5,850,615 - 5,865,958''' (1,000 Users, 6,354 videos, 133G) DoubleJ/Guan cut from 58xx range


* '''0,000,000''' - '''0,750,000''' Jason Scott, Chronomex
=== Uploading ===
* '''0,750,000''' - '''1,000,000''' Chronomex
Please try to keep this box up to date.
* '''1,000,000''' - '''1,750,000''' Joey
* '''1,750,000''' - '''2,000,000''' Underscor
* '''2,000,000''' - '''2,250,000''' Joey
* '''2,250,000''' - '''2,500,000''' Underscor/Chronomex (Chronomex is finishing)
* '''2,500,000''' - '''2,750,000''' Soultcer
* '''2,750,000''' - '''3,000,000''' Underscor
* '''3,000,000''' - '''3,250,000''' Soultcer
* '''3,250,000''' - '''3,500,000''' Underscor
* '''3,500,000''' - '''4,000,000''' Chronomex
* '''4,000,000''' - '''4,250,000''' Chronomex (gir)
* '''4,250,000''' - '''4,500,000''' Chronomex (dedi)
* '''4,500,000''' - '''4,625,000''' Underscor
* '''4,750,000''' - '''5,000,000''' Underscor
* '''5,000,000''' - '''5,300,000''' Jason Scott
* '''5,500,000''' - '''5,750,000''' Ymgve
* '''5,750,000''' - '''6,000,000''' Underscor
* '''6,050,000 - 6,100,000''' robzy
* '''6,100,000 - 6,150,000''' robzy
* '''6,150,000 - 6,200,000''' phuz
* '''6,250,000 - 6,300,000''' phuz
* '''6,350,000 - 6,400,000''' NovaKing
* '''6,400,000 - 6,450,000''' NovaKing
* '''6,450,000 - 6,500,000''' NovaKing
* '''6,500,000 - 6,550,000''' phuz
* '''6,550,000 - 6,600,000''' phuz
* '''6,650,000 - 6,700,000''' NovaKing
* '''7,500,000''' - '''8,000,000''' Jason Scott
* '''8,000,000''' - '''8,250,000''' Spirit
* '''8,250,000''' - '''8,500,000''' Sungo
* '''8,500,000''' - '''8,750,000''' Sungo
* '''8,750,000''' - '''9,000,000''' Spirit
* '''9,000,000''' - '''9,360,000''' Jason Scott/Underscor/Ymgve


===Continuing===
* '''0,650,000 - 0,700,000''' (3,162 Users, 17,846 videos, 147gb) Hydruh 4362
 
* '''1,600,000 - 1,700,000''' (4,933 Users, 8250 videos, 82g) Masterono 4362
* '''4,625,000''' - '''4,750,000''' Underscor
* '''1,800,000 - 1,900,000''' (4,744 Users, 20,494 videos, 169g) bluknight D1
* '''5,250,000 - 5,500,000''' Jason Scott/Spirit (Spirit is finishing)
* '''2,900,000 - 3,000,000''' (6,221 Users, 260G) dj_ryan 4421
* '''6,000,000 - 6,050,000''' Underscor (server down) (NovaKing duping)
* '''3,100,000 - 3,200,000''' (5,760 Users, 15,661 videos, 186gb) Masterono 4362
* '''6,200,000 - 6,250,000''' cb43569
* '''4,800,000 - 4,900,000''' (5,754 Users, 15,759 videos, 483gb) monst3r (Might be just part of 50x-51x) 4363
* '''6,300,000 - 6,350,000''' NovaKing
* '''5,000,000 - 5,100,000''' (6,545 Users, 32,041 videos, 428gb) monst3r_ 4363
* '''6,600,000 - 6,650,000''' phuz
* '''5,150,000 - 5,200,000''' (3,313 Users, 18,127 videos, 235gb) Guan (to underscor, paused)
* '''6,700,000 - 6,750,000''' NovaKing
* '''5,400,000 - 5,500,000''' (6,897 Users, 738gb) dj_ryan 4421
* '''6,750,000 - 7,000,000''' chronomex (dante01)
* '''5,954,430 - 5,963,115''' (700 Users) DoubleJ (split from km3k/amerrykan)
* '''7,000,000 - 7,250,000''' chronomex (dante02)
* '''6,300,000 - 6,400,000''' (6,378 Users, 44,962 videos, 849gb) Guan (to underscor)
* '''7,250,000 - 7,500,000''' chronomex (dante03)
* '''6,400,000 - 6,500,000''' (6,097 Users, 32,239 videos, 654GB) Masterono
 
* '''7,400,000 - 7,500,000''' (4,204 Users, 22,560 videos, 290gb) ZPedro (to 4362)
== Phase Two: Video Acquisition ==
* '''8,000,000 - 8,100,000''' (3,416 Users, 16,781 videos, 201gb) bluknight D1
 
* '''8,300,000 - 8,400,000''' (2,920 Users, 8,973 videos, 93gb) Masterono
[[File:Lf3.png|right|thumb|"Oh my god, some of these videos!"]]
* '''9,050,000 - 9,100,000''' (1,796 Users, 6,142 videos, 79gb) Masterono
 
Once the hotlists of who has videos have been grabbed (and that is going to happen rather shortly, thanks to the army of folks), we'll be running a program called ''grapefruit'' (the g and fruit are silent, ræp) that yanks down all the videos from a given user. The number of users with video is a tiny percentage of the total users (we're seeing rates of less than 1 video-keeping user for every 10-20 of the users, and less), it will still be significant diskspace. It's video, after all. After an initial test run of grapefruit, we'll start handing out numbers and hitlists to people, and they'll begin downloading.
 
We have no idea where we're going to put this. We're going to need a bigger boat.
=== Verified and On-Drive ===
Talk to underscor to get your stuff uploaded.
 
* '''0,700,000 - 0,750,000''' (3,213 Users, 20,767 videos, 127gb) Underscor


=== Finished ===
=== Finished ===


* '''0,100,000 - 0,200,000''' (6,247 Users, x videos, 159gb) Underscor
* '''0,015,000 - 0,020,000''' (3,948 Users, 24,328 videos, 314gb) jakemcgra
* '''0,200,000 - 0,300,000''' (6,461 Users, x videos, 141gb) Underscor
* '''0,025,000 - 0,030,000''' (4,040 Users) nwf
* '''0,400,000 - 0,500,000''' (6,231 Users, x videos, 159gb) Underscor
* '''0,030,000 - 0,035,000''' (4,095 Users, 25,490 videos, 209gb) penemue/NovaKing
* '''0,500,000 - 0,600,000''' (6,543 Users, 13,849 videos, 115gb) NovaKing
* '''0,040,000 - 0,045,000''' (4,024 Users) PMT
* '''0,750,000 - 0,800,000''' (3,735 Users, 1,097 videos, 7gb) atg (Use as test)
* '''0,045,000 - 0,050,000''' (4,051 Users, 31,110 videos, 288gb) Jason Scott/Ersi
* '''0,050,000 - 0,055,000''' (3,976 Users) zumthing
* '''0,055,000 - 0,060,000''' (4,086 Users, 13098 videos, 107GB) Lambda_Dr/Guan
* '''0,074,000 - 0,075,000''' (801 Users, 5376 videos, 28 gb) widget (unavailable for upload until wed 16th)
* '''0,077,256 - 0,079,747''' (2,000 Users, 10,603 videos, 102gb) Guan
* '''0,079,748 - 0,082,199''' (2,000? Users) ksuter
* '''0,082,200 - 0,083,321''' (1,000 Users, 6,536 videos, 60gb) Guan
* '''0,600,000 - 0,650,000''' (3,398 Users, 16,861 videos, 182gb) robertcu (guan re-downloaded)
* '''0,650,000 - 0,700,000''' (3,162 Users, 17,846 videos, 147gb) Hydruh
* '''0,750,000 - 0,800,000''' (3,735 Users, 19,032 videos, 136G) atg
* '''0,800,000 - 0,850,000''' (3,140 Users, 10,969 videos, 104G) crust/Guan
* '''0,850,000 - 0,900,000''' (3,749 Users, 18,562 videos, 138gb) filer
* '''0,850,000 - 0,900,000''' (3,749 Users, 18,562 videos, 138gb) filer
* '''1,100,000 - 1,200,000''' (7,613 Users, 12,892 videos, 177gb ) NovaKing
* '''0,900,000 - 1,000,000''' (6,453 Users, 18,077 videos, 139gb) K4K/NovaKing
* '''1,500,000 - 1,600,000''' (5,252 Users, 7642 videos, 86g) NovaKing
* '''1,000,000 - 1,100,000''' (6,331 Users, 31,241 videos, 247gb) Zumthing (original download + factcheck cleanup)
* '''1,600,000 - 1,700,000''' (4,933 Users, 8250 videos, 82g) Masterono
* '''1,200,000 - 1,300,000''' (7,130 Users) drspangle (finished, guan doing factcheck)
* '''2,000,000 - 2,100,000''' (4,408 Users, 20,583 videos, 167g) dashcloud
* '''1,300,000 - 1,400,000''' (7,598 Users) sep332
* '''2,100,000 - 2,200,000''' (4,519 Users, 16,597 videos, 144g) Joey (rsyncing)
* '''1,400,000 - 1,500,000''' (5,194 Users, 6,476 videos, 68gb) beardicus
* '''2,400,000 - 2,500,000''' (2,802 Users) NovaKing (Userscrape seems truncated, may need redo, transfering to underscor)
* '''1,700,000 - 1,800,000''' (4,368 Users, 8013 videos, 97G) atg
* '''2,500,000 - 2,600,000''' (3,949 Users, 7,849 videos, 82gb) Zumthing
* '''1,800,000 - 1,900,000''' (4,744 Users, 20,494 videos, 169g) bluknight
* '''3,000,000 - 3,100,000''' (6,297 Users, 18,418 videos, 163gb) Guan (transferring to underscor)
* '''2,200,000 - 2,300,000''' (4,216 Users, 18,708 videos 168G) jakemcgra
* '''3,100,000 - 3,200,000''' (5,760 Users, 15,661 videos, 186gb) Masterono
* '''2,500,000 - 2,600,000''' (3,949 Users, 15,994 videos, 146gb) Zumthing (original download + factcheck cleanup)
* '''3,300,000 - 3,400,000''' (4,252 Users) NovaKing (transfering to underscor)
* '''2,600,000 - 2,700,000''' (4,208 Users, 17,903 videos, 183G) Frigolit (utf8: ok, factcheck: ok)
* '''3,400,000 - 3,500,000''' (4,560 Users) NovaKing (transfering to underscor)
* '''2,700,000 - 2,800,000''' (4,754 Users, 22,600 videos, 192gb) ersi/Guan
* '''3,600,000 - 3,700,000''' (4,494 Users, 21,863 videos, 251gb) chronomex: gir
* '''2,800,000 - 2,900,000''' (5,623 Users, 23,879 videos, 249gb) amuck
* '''3,700,000 - 3,800,000''' (4,262 Users) NovaKing (transfering to underscor)
* '''3,200,000 - 3,250,000''' (2,603 Users) Soultcer (timestamps: ok, utf8: ok, factcheck done)
* '''5,150,000 - 5,200,000''' (3,313 Users, 18,127 videos, 235gb) Guan
* '''3,250,000 - 3,300,000''' (2,439 Users) taaz
* '''7,500,000 - 7,600,000''' (4,347 Users, 20,418 videos, 240gb) Guan
* '''3,500,000 - 3,600,000''' (4,895 Users, 22,964 videos, 254gb) tobbez/nwf
* '''7,600,000 - 7,700,000''' (4,200 Users, 17,885 videos, 189gb) NovaKing
* '''3,800,000 - 3,900,000''' (3,936 Users, 22,414 videos, 312gb) dysproseu (factchecked)
* '''3,900,000 - 4,000,000''' (4,055 Users, 11,292 videos, 136gb ) HA/Novaking
* '''4,200,074 - 4,289,987''' (4,250 Users) atg
** '''4,224,414 - 4,269,902''' (? Users) penemue
** '''4,270,007 - 4,279,957''' (438 Users, 2,477 videos, 28gb) guan
** '''4,280,010 - 4,289,987''' (406 Users) nwf
* '''4,290,000 - 4,300,000''' (456 Users, 2,030 videos, 38gb) savetz/penemue
* '''4,300,000 - 4,400,000''' (4,355 Users, 11,947 videos, 157gb) NovaKing
* '''4,400,000 - 4,500,000''' (4,715 Users, 21,569 videos, 299G) beardicus
* '''4,557,501 - 4,586,425''' (1,713 Users) Lambda_dr (email dsnigier at gmail.com with uploading details...can keep on drive array for as long as needed)
* '''4,700,000 - 4,750,000''' (2,665 Users, 8,229 videos, 104gb) Gandalfar/NovaKing  
* '''4,750,000 - 4,784,985''' (2,127 Users, 15,265 videos, 193gb) crust/Guan
* '''4,790,761 - 4,799,994''' (400 Users, 5648 videos & html files, 26.8 gb) penemue (overlap with primus_102)
* '''4,800,000 - 4,900,000''' (5,764 Users) jch (Downloaded, factchecking)
* '''4,900,000 - 4,913,495''' (1,000 Users) nwf
* '''4,913,497 - 4,937,299''' (1,500 Users) guan DONE (1,501 users, 9,030 videos, 113gb)
* '''4,937,299 - 4,945,193''' (500 Users, 3411 videos, 41gb) yoyoguy2 DONE
* '''4,966,514 - 4,979,997''' (1000 Users, 9680 Videos, 111gb) penemue DONE
* '''4,980,027 - 4,994,982''' (780 Users, 11,375 html+flv, 61,9gb) penemue DONE
* '''4,995,045, - 4,999,999''' (364 Users, 3293 videos, 33gb) penemue DONE
* '''5,255,391 - 5,299,993''' (3,000 users, 23,847 videos, 330gb ) penemue
* '''5,280,381 - 5,291,717''' (749 Users, 6,540 videos, 92gb) Guan
* '''5,200,000 - 5,300,000''' (6,751 Users, 54,805 videos, 734gb) jakemcgra
* '''5,300,000 - 5,400,000''' (6,289 Users, 36,617 videos, 556gb) jakemcgra
* '''5,800,020 - 5,899,977''' (6,188 Users) jch/DoubleJ/Spirit/jakemcgra
** '''5,842,906 - 5,850,604''' (500 Users, 4,663 videos 91gb) jakemcgra
** '''5,882,964 - 5,894,410''' (~725 users, ? videos, 119.5gb) penemue
** '''5,894,410 - 5,899,977''' (? Users, ~ 45gb) penemue done
* '''5,900,005 - 5,999,982'''
** '''5,900,005 - 5,912,492''' (700 users, 4133 videos, 87GB) km3k (confirmed done)
** '''5,912,497 - 5,944,029''' (? users) amerrykan, done)
** '''5,932,108 - 5,944,802''' (~750 users, ? videos, 68.7GB) SudoSeth/Guan
** '''5,944,810 - 5,954,407''' (750 Users) penemue (split from km3k/amerrykan)
** '''5,963,126 - 5,967,009''' (300 users) guan (split from DoubleJ's split)
** '''5,967,027 - 5,974,072''' (500 Users, 81.9gb) penemue (from DoubleJ's split)
** '''5,974,076 - 5,999,982''' (2,000 Users) nwf
* '''6,100,000 - 6,200,000''' (7,433 Users) incendui (done, except for 897 user missing-hotlist, which was sent to guan, status? yoyoguy2 also working)
** '''6,169,448 - 6,199,997''' Guan (2k users, done)
** '''6,156,334 - 6,169,434''' Lamda_Dr (done)
* '''6,600,000 - 6,650,000''' (3,516 Users, 12,404 videos, 277GB) Lambda_Dr/Guan (timestamps updated, factcheck complete)
* '''6,650,000 - 6,666,477''' (1,000 Users, 5,470 videos, 109gb) sdboyd
* '''6,650,000 - 6,700,000''' (2049 users, 15,021 videos, 293gb) jakemcgra
* '''6,700,000 - 6,750,000''' (3,079 Users, 23,950 videos, 510gb) yoyoguy2
* '''6,750,000 - 6,800,000''' (2,838 Users, 17,555 videos, 356gb) Jason Scott (Transferring to data dump 1004)
* '''6,900,000 - 7,000,000''' (5,676 Users) bluknight (sandra)
* '''7,000,000 - 7,100,000''' (4,833 Users, 24,523 videos, 311gb) bluknight
* '''7,100,000 - 7,200,000''' (4,026 Users, 13,244 videos, 167gb) zumthing
* '''7,200,000 - 7,300,000''' (3,939 Users, 21,731 videos, 292G) dysproseu (factchecked)
* '''7,300,000 - 7,400,000''' (4,304 Users, 23,904 videos, 307gb) amerrykan done, factchecked
* '''7,700,000 - 7,800,000''' (4,028 Users, 15,037 videos, 171gb) K4K
* '''7,700,000 - 7,800,000''' (4,028 Users, 15,037 videos, 171gb) K4K
* '''7,700,000 - 7,800,000''' (4,028 Users, 19,155 videos, 219gb) jakemcgra
* '''7,900,000 - 8,000,000''' (3,517 Users, 10,392 videos, 111gb) matzie
* '''7,900,000 - 8,000,000''' (3,517 Users, 10,392 videos, 111gb) matzie
* '''8,300,000 - 8,400,000''' (2,920 Users, 8,973 videos, 93gb) Masterono
* '''8,100,000 - 8,200,000''' (3,380 Users, 13,882 videos. 160gb) Primus104 (Transferring to data dump 4362)
* '''9,000,000 - 9,050,000''' (1,745 Users) atg
* '''8,400,000 - 8,500,000''' (3,535 Users, 16,784 videos, 199gb) yoyoguy2
* '''9,050,000 - 9,100,000''' (1,796 Users, 6,142 videos, 79gb) Masterono
* '''8,500,000 - 8,600,000''' (3,516 Users, 19,858 videos, 200GB) red_cell
* '''8,700,000 - 8,800,000''' (3,730 Users, 16,698 videos, 186gb) t0phux
* '''8,850,000 - 9,000,000''' (1,770 Users, 8,832 videos, 106gb) Gandalfar (uploading to 4482)
* '''8,900,000 - 9,000,000''' (3,793 Users, 10,790 videos, 167gb) khaoohs/NovaKing
* '''9,000,000 - 9,050,000''' (1,745 Users, 6,889 videos, 86G) atg
* '''9,100,000 - 9,150,000''' (1,989 Users, 9,380 videos, 111gb) Killface/yoyoguy2
* '''9,150,000 - 9,200,000''' (5,173 Users, 22,655 videos, 261gb) Underscor
* '''9,150,000 - 9,200,000''' (5,173 Users, 22,655 videos, 261gb) Underscor
* '''9,200,000 - 9,300,000''' (8,941 Users, 30,179 videos, 341gb) Zumthing (overlaps with drspangle)
* '''9,200,000 - 9,300,000''' (8,941 Users, 30,179 videos, 341gb) Zumthing (drspangle has a duplicate subset)
* '''9,300,000 - 9,360,000''' (2,179 Users, 4,406 videos, 51gb) Robertcu
* '''freaks''' (1,163 Users): Guan
* '''each 50,000st user from the whole range''' (112 videos, 1G) Spirit (factchecked and done)
* '''0,000,000 - 0,100,000''' ksuter has done a safety-run of this historically important range


=== In-Process ===
=== In-Process ===


LEGEND.
* '''1,900,000 - 2,000,000''' (4,481 Users, 10,346 videos, 79G) ajclement/swebb/NovaKing
* "???" = unassigned
* '''4,600,000 - 4,700,000''' (5,151 Users) eberon NOC (jakemcgra doing just-in-case grab)
* "OPEN" = unassigned, hotlist available
* '''5,600,000 - 5,700,000''' (5,903 Users) _kp NOC (Hydruh doing just-in-case grab)
* '''5,700,000 - 5,800,000''' (3,194 Users) finished 91.300% (2918/3194) 3/21
* '''6,050,000 - 6,100,000''' (3,747 Users) t0phux (NovaKing doing backup grab)
 
==== Cleanup in progress ====
 
* Spangle combined missing-hotlist (416 users) Guan


And now the list.
=== UTF-8 Errors What ===
* '''0,000,000 - 0,100,000''' (72,176 Users) Underscor 0-10k Done
 
* '''0,300,000 - 0,400,000''' (6,909 Users) Underscor Downloading
Some of the filenames in Yahoo! are encoded in Unicode, which flips out on some machines. If you run ''grapefruit'' on user 45002 and see errors like "UnicodeEncodeError: 'ascii' codec can't encode characters", then you have this issue. At this point in time, we can't use you, but we love you anyway - come cheer us on.
* '''0,600,000 - 0,700,000''' (7,833 Users) Underscor Downloading
* '''0,800,000 - 0,850,000''' (3,140 Users) Crust
* '''0,900,000 - 1,000,000''' (6,453 Users) K4K
* '''1,000,000 - 1,100,000''' (6,331 Users) Zumthing
* '''1,200,000 - 1,300,000''' (7,130 Users) drspangle
* '''1,300,000 - 1,400,000''' (7,598 Users) sep332
* '''1,400,000 - 1,500,000''' (5,194 Users) beardicus
* '''1,700,000 - 1,800,000''' (4,368 Users) atg
* '''1,800,000 - 1,900,000''' (4,744 Users) bluknight (sandra)
* '''1,900,000 - 2,000,000''' (4,481 Users) ajclement
* '''2,200,000 - 2,300,000''' (4,216 Users) jakemcgra
* '''2,300,000 - 2,400,000''' (4,615 Users) DoubleJ (hotlist was troubled, check him)
* '''2,600,000 - 2,700,000''' (4,208 Users) Frigolit
* '''2,700,000 - 2,800,000''' (4,754 Users) ersi
* '''2,900,000 - 3,000,000''' (6,221 Users) dj_ryan
* '''3,200,000 - 3,250,000''' (2,603 Users) Soultcer
* '''3,250,000 - 3,300,000''' (2,439 Users) taaz
* '''3,800,000 - 3,900,000''' (3,936 Users) dysproseu
* '''3,900,000 - 4,000,000''' (4,055 Users) HA
* '''4,000,000 - 4,100,000''' ???
* '''4,100,000 - 4,200,000''' ???
* '''4,200,000 - 4,300,000''' ???
* '''4,300,000 - 4,400,000''' ???
* '''4,400,000 - 4,500,000''' ???
* '''4,600,000 - 4,700,000''' ???
* '''4,700,000 - 4,800,000''' ???
* '''4,800,000 - 4,900,000''' (5,764 Users) monst3r_
* '''4,900,000 - 5,000,000''' (6,738 Users) jch
* '''5,000,000 - 5,100,000''' (6,545 Users) monst3r_
* '''5,100,000 - 5,150,000''' (3,254 Users) Spirit
* '''5,200,000 - 5,300,000''' ???
* '''5,300,000 - 5,400,000''' ???
* '''5,400,000 - 5,500,000''' ???
* '''5,600,000 - 5,700,000''' (5,903 Users) _kp
* '''5,700,000 - 5,800,000''' ???
* '''5,800,000 - 5,900,000''' ???
* '''5,900,000 - 6,000,000''' ???
* '''6,000,000 - 6,100,000''' ???
* '''6,100,000 - 6,200,000''' (7,433 Users) incendui
* '''6,200,000 - 6,250,000''' ???
* '''6,250,000 - 6,300,000''' (3,109 Users) NovaKing
* '''6,300,000 - 6,400,000''' (6,378 Users) Guan
* '''6,400,000 - 6,500,000''' (6,097 Users) Masterono
* '''6,500,000 - 6,600,000''' (6,588 Users) NovaKing
* '''6,600,000 - 6,700,000''' ???
* '''6,700,000 - 6,800,000''' ???
* '''6,800,000 - 6,900,000''' ???
* '''6,900,000 - 7,000,000''' ???
* '''7,000,000 - 7,100,000''' ???
* '''7,100,000 - 7,200,000''' ???
* '''7,200,000 - 7,300,000''' ???
* '''7,300,000 - 7,400,000''' ???
* '''7,400,000 - 7,500,000''' ???
* '''7,500,000 - 7,600,000''' (4,347 Users) sadcarrot
* '''7,700,000 - 7,800,000''' (4,028 Users) jakemcgra
* '''7,800,000 - 7,900,000''' (3,720 Users) NovaKing
* '''8,000,000 - 8,100,000''' (3,416 Users) bluknight (tara)
* '''8,100,000 - 8,200,000''' (3,380 Users) Primus104
* '''8,200,000 - 8,300,000''' (3,335 Users) NovaKing
* '''8,400,000 - 8,500,000''' (3,535 Users) yoyoguy2
* '''8,500,000 - 8,600,000''' (3,516 Users) red_cell
* '''8,600,000 - 8,700,000''' (7,726 Users) Joey (slowbox)
* '''8,700,000 - 8,800,000''' (3,730 Users) t0phux
* '''8,800,000 - 8,850,000''' (2,170 Users) sdboyd
* '''8,850,000 - 9,000,000''' (1,770 Users) Gandalfar
* '''8,900,000 - 9,000,000''' (3,793 Users) khaoohs
* '''9,100,000 - 9,150,000''' (1,989 Users) Killface
* '''9,250,000 - 9,300,000''' (2,835 Users) drspangle (overlaps with Zumthing)
* '''9,300,000 - 9,360,000''' (2,179 Users) Robertcu


== Phase Three: Final Resting Place ==
== Phase Three: Final Resting Place ==
[[Image:Yahooboni.jpg|thumb|right|300px|Yahoo!: We can't afford storage, but we can afford Zambonis]]


In an ideal world, all the downloaders would hold copies and everyone would just torrent the thing. That's not going to happen in current contemporary machine capacity. Our ''very rough'' estimate is that a download of Yahoo! Video is going to be [https://spreadsheets.google.com/ccc?key=0Akj1Dp3CIWDadHRJLUxnWDlSUi1kMk1YUENOVjY1S0E&hl=en&authkey=CP-iwvoF 20 terabytes]. While this is under a thousand dollars of newegg-purchased disks, it's still a bit much to ask of anyone. Meanwhile, the more forward-looking of volunteers are asking, "what's the exit strategy?".
In an ideal world, all the downloaders would hold copies and everyone would just torrent the thing. That's not going to happen in current contemporary machine capacity. Our ''very rough'' estimate is that a download of Yahoo! Video is going to be [https://spreadsheets.google.com/ccc?key=0Akj1Dp3CIWDadHRJLUxnWDlSUi1kMk1YUENOVjY1S0E&hl=en&authkey=CP-iwvoF 20 terabytes]. While this is under a thousand dollars of newegg-purchased disks, it's still a bit much to ask of anyone. Meanwhile, the more forward-looking of volunteers are asking, "what's the exit strategy?".
Line 218: Line 220:


The priority, however, is to get all the data in the first place.
The priority, however, is to get all the data in the first place.
=== Verified and On-Drive ===
Talk to underscor to get your stuff uploaded. (I will perform a final check of the timestamps)
==== Set 1 (Disks 1 and 2) ====
Being used as upload holding pen at the moment
==== Set 2 (Disks 3 and 4) ====
* '''0,300,000 - 0,400,000''' (6,909 Users, 41,131 videos, 342gb) Underscor
* '''0,500,000 - 0,600,000''' (6,543 Users, 13,849 videos, 115gb) NovaKing
* '''0,700,000 - 0,750,000''' (3,213 Users, 20,767 videos, 127gb) Underscor
* '''1,500,000 - 1,600,000''' (5,252 Users, 7,642 videos, 86g) NovaKing
* '''2,100,000 - 2,200,000''' (4,519 Users, 16,597 videos, 144g) Joey
* '''3,000,000 - 3,100,000''' (6,297 Users, 18,418 videos, 163gb) Guan
* '''7,600,000 - 7,700,000''' (4,200 Users, 17,885 videos, 189gb) NovaKing
==== Set 3 (Disks 5 and 6) ====
* '''0,060,000 - 0,065,000''' (2,226 Users, 23,254 videos, 228gb) underscor
* '''0,065,000 - 0,070,000''' (4,077 Users, 23,777 videos, 223gb) underscor
* '''0,070,000 - 0,072,000''' (1,613 Users, 10,341 videos, 98gb) underscor
* '''0,075,000 - 0,077,255''' (1,801 Users, 5,405 videos, 41GB) Spirit
* '''1,100,000 - 1,200,000''' (7,613 Users, 12,892 videos, 177gb ) NovaKing
* '''2,000,000 - 2,100,000''' (4,408 Users, 20,583 videos, 167g) dashcloud
* '''3,400,000 - 3,500,000''' (4,560 Users, 11,450 videos, 102gb) NovaKing
* '''5,100,000 - 5,150,000''' (3,254 Users, 19701 videos, 327gb) Spirit
* '''8,800,000 - 8,850,000''' (2,170 Users, 8,368 videos, 71gb ) sdboyd
==== Set 4 (Disks 7 and 8) ====
* '''0,000,000 - 0,005,000''' (3,960 Users, 27,357 videos, 268gb) Underscor
* '''0,005,000 - 0,010,000''' (3,993 Users, 25,858 videos, 248gb) Underscor
* '''0,035,000 - 0,040,000''' (3,966 Users, 24,164 videos, 225gb) penemue
* '''2,400,000 - 2,500,000''' (2,802 Users, 6,260 videos, 48gb) NovaKing
* '''3,300,000 - 3,400,000''' (4,252 Users, 10,672 videos, 93gb) NovaKing
* '''3,600,000 - 3,700,000''' (4,494 Users, 21,863 videos, 251gb) chronomex
* '''3,700,000 - 3,800,000''' (4,262 Users, 11,790 videos, 142gb) NovaKing
* '''6,000,000 - 6,050,000''' (3,571 Users) savetz
==== Set 5 (Disks 9 and 10) ====
* '''0,067,800 - 0,070,000''' (1,792 Users, 9725 Videos, 100gb) penemue
* '''0,182,338 - 0,199,999''' (1118 users, 5681 videos. 45gb) amnesia
* '''7,500,000 - 7,600,000''' (4,347 Users, 22,758 videos, 287gb) sadcarrot/amnesia
==== Set 6 (Disks 11 and 12) ====
* '''0,018,500 - 0,020,000''' (x users, x videos, xgb) penemue
* '''0,020,000 - 0,025,000''' (4,053 Users, 27,844 videos, 328gb) guan
* '''0,083,322 - 0,100,003''' (5,000 Users, 34,236 videos, 324gb) Guan
* '''5,835,908 - 5,842,902''' (518 users, 3,464 videos, 66gb) Guan
* '''6,800,000 - 6,900,000''' (5,234 Users, 26,243 videos, 558gb) dashcloud
* guan's cleanup files (bits of other ranges)
==== Set 7 (Disks 13 and 14) ====
Disks still need purchasing
==== Set 8 (Disks 15 and 16) ====
Disks still need purchasing
==== Set 9 (Disks 17 and 18) ====
Disks still need purchasing
==== Set 10 (Disks 19 and 20) ====
Disks still need purchasing


== Statistics ==
== Statistics ==
Line 225: Line 285:
People like easy to quote statistics from a project, so here are some, with others coming soon.
People like easy to quote statistics from a project, so here are some, with others coming soon.


Yahoo! Video officially started as a user content system in June of 2006. Previously, it had been a directory of other video sites around the net (harkening back to Yahoo's previous life as a directory instead of its short burst as a content creator). From the start, it was called a "YouTube Clone" and other derisive terms.  
Yahoo! Video officially started as a user content system in June 2006. Previously, it had been a directory of other video sites around the net (harkening back to Yahoo's previous life as a directory instead of its short burst as a content creator). From the start, it was called a "YouTube Clone" and other derisive terms.  


However, users did decide to go for this new environment. Archive Team found about 9,360,000 accounts in use on Yahoo Video. "In Use" is a relative term, of course - many, many users consist of no uploaded videos, no user picture, and no other activity - just a name. We'll have more, but for example, of 1.5 million accounts we've scanned so far, only 146,600 have uploaded videos, and if you shear off the first 100,000 (i.e., frontloaded with content partner) accounts, that 10% figure drops down to 5%.
However, users did decide to go for this new environment. Archive Team found about 9,360,000 accounts in use on Yahoo Video. "In Use" is a relative term, of course - many, many users consist of no uploaded videos, no user picture, and no other activity - just a name. We'll have more, but for example, of 1.5 million accounts we've scanned so far, only 146,600 have uploaded videos, and if you shear off the first 100,000 (i.e., frontloaded with content partner) accounts, that 10% figure drops down to 5%.


It didn't take very long for users to figure out Yahoo! Video is not the way to go, either as participants or as viewers. The most popular video, "Baby Raccoons Saved from Soda Machine" has 14 million views. Compare with one of the top videos on YouTube - Justin Beiber performing "Baby" with Ludacris, which has 457 million views. By the time the raccoons are viewed, the next most popular video, "Worst Best Man Ever", clocks in at just over 10 million views. After the first 20 videos, the views per video are down to about 4 million. By the time we get to the top 100, we're already skirting 1 million. Again, a single video on Youtube, the most popular, has a number of views basically the same as the top 100 Yahoo! videos combined.
It didn't take very long for users to figure out Yahoo! Video is not the way to go, either as participants or as viewers. The most popular video, "Baby Raccoons Saved from Soda Machine" has 14 million views. Compare with one of the top videos on YouTube - Justin Beiber performing "Baby" with Ludacris, which has 457 million views. By the time the raccoons are viewed, the next most popular video, "Worst Best Man Ever", clocks in at just over 10 million views. After the first 20 videos, the views per video are down to about 4 million. By the time we get to the top 100, we're already skirting 1 million. Again, a single video on YouTube, the most popular, has a number of views basically the same as the top 100 Yahoo! videos combined.
 
{{Navigation box}}

Latest revision as of 02:45, 5 December 2017


Yahoovideo.jpg

Phase One: Username Acquisition (Complete)

"You people are working so hard!"

Summary

In roughly 30 days, 12 Archive Team members scanned 9,360,000 Yahoo! Video user account numbers. Of these, 9,218,844 were actual user accounts (others were deleted or "networks", i.e. playlists). Of these, 555,502 had actually taken the time to upload one or more videos. All told, we showed that 3,585,981 videos were uploaded to Yahoo! Video.

  • 0,000,000 - 1,000,000 Jason Scott, Chronomex
  • 1,000,000 - 2,250,000 Joey/Underscor
  • 2,250,000 - 2,500,000 Underscor/Chronomex
  • 2,500,000 - 3,500,000 Soultcer/Underscor
  • 3,500,000 - 4,500,000 Chronomex
  • 4,500,000 - 5,000,000 Underscor
  • 5,000,000 - 5,500,000 Jason Scott/Spirit
  • 5,500,000 - 5,750,000 Ymgve
  • 5,750,000 - 5,900,000 guan/Jason Scott
  • 5,900,000 - 6,000,000 soultcer
  • 6,000,000 - 6,050,000 Underscor/NovaKing
  • 6,050,000 - 6,150,000 robzy
  • 6,150,000 - 6,200,000 phuz
  • 6,200,000 - 6,250,000 guan
  • 6,250,000 - 6,750,000 phuz/NovaKing
  • 6,750,000 - 7,500,000 chronomex/NovaKing
  • 7,500,000 - 8,000,000 Jason Scott
  • 8,000,000 - 8,250,000 Spirit
  • 8,250,000 - 8,750,000 Sungo
  • 8,750,000 - 9,000,000 Spirit
  • 9,000,000 - 9,360,000 Jason Scott/Underscor/Ymgve

Phase Two: Video Acquisition

"Oh my god, some of these videos!"

Once the hotlists of who has videos have been grabbed (and that is going to happen rather shortly, thanks to the army of folks), we'll be running a program called grapefruit that yanks down all the videos from a given user. The number of users with video is a tiny percentage of the total users (we're seeing rates of less than 1 video-keeping user for every 10-20 of the users, and less), it will still be significant diskspace. It's video, after all. After an initial test run of grapefruit, we'll start handing out numbers and hitlists to people, and they'll begin downloading.

If you used axel / grapefruit-axel, please tell soultcer or guan or underscor on IRC. Axel doesn't store timestamps, thus we try to add that data manually, but we need to know who used axel and who didn't.

Downloading from rsync.net

  • 0,400,000 - 0,500,000 (6,231 Users, 36,205 videos, 336gb) amnesia (YV_10)
  • 6,200,000 - 6,300,000 (YV_10)
    • 6,200,000 - 6,210,039 ( 738 Users) Pneu 4363
    • 6,210,039 - 6,250,000 (2,609 Users) NovaKing 4363
    • 6,250,000 - 6,300,000 (3,109 Users) NovaKing 4363
  • 6,500,000 - 6,600,000 (6,588 Users, 22,413 videos, 492gb) NovaKing 4363 (YV_10)
  • 7,500,000 - 7,600,000 (4,347 Users, 20,418 videos, 240gb) Guan 4350 (YV_10)

Tons of stuff from penemue... will figure out exactly what tomorrow. Underscor 05:49, 22 March 2011 (UTC)

Uploaded to rsync.net

  • 0,100,000 - 0,182,337 (5,131 Users, 28,741 videos, 268gb) Joey 4362
  • 0,200,000 - 0,300,000 (6,461 Users, 33,594 videos, 268GB) DoubleJ 4362
  • 2,700,000 - 2,800,000 (4,754 Users, 22,600 videos, 192gb) ersi/Guan
  • 4,100,000 - 4,200,000 (4,511 users, 29,154 videos, 360gb) penemue (via NovaKing) 4363
  • 4,558,113 - 4,574,989 (771 Users, 4,054 videos, 63gb) DoubleJ 4362
  • 5,869,902 - 5,882,934 Spirit 4406
  • 7,800,000 - 7,900,000 (3,720 Users, 11,322 videos, 128gb) NovaKing 4363
  • 8,200,000 - 8,300,000 (3,335 Users, 9,269 videos, 100gb) NovaKing 4363
  • 8,600,000 - 8,700,000 (3,863 Users, 20,391 videos, 241gb) Joey 1005
  • freaks2 (1,841 Users, 3,731 videos, 38gb) DoubleJ 4362

Uploaded to flophouse/blindtiger

  • 0,010,000 - 0,015,000 (4,055 Users) soultcer gv_7@bt
  • 2,300,000 - 2,400,000 (4,615 Users, 20,944 videos, 216gb) DoubleJ/Guan gv_12@bt
  • 4,000,000 - 4,100,000 (4,251 Users, 26,591 videos, 342gb) primus104/102 gv21@bt
  • 4,785,016 - 4,800,000 ( 710 Users, 4224 videos, 51GB) primus_104/102 gv20@bt

Uploaded to batcave

  • 4,500,000 - 4,600,000 (3,434 Users, 22,645 videos, 324gb) DoubleJ/Guan
  • 4,558,113 - 4,586,425 (770 Users from rescrape)
  • 5,850,615 - 5,865,958 (1,000 Users, 6,354 videos, 133G) DoubleJ/Guan cut from 58xx range

Uploading

Please try to keep this box up to date.

  • 0,650,000 - 0,700,000 (3,162 Users, 17,846 videos, 147gb) Hydruh 4362
  • 1,600,000 - 1,700,000 (4,933 Users, 8250 videos, 82g) Masterono 4362
  • 1,800,000 - 1,900,000 (4,744 Users, 20,494 videos, 169g) bluknight D1
  • 2,900,000 - 3,000,000 (6,221 Users, 260G) dj_ryan 4421
  • 3,100,000 - 3,200,000 (5,760 Users, 15,661 videos, 186gb) Masterono 4362
  • 4,800,000 - 4,900,000 (5,754 Users, 15,759 videos, 483gb) monst3r (Might be just part of 50x-51x) 4363
  • 5,000,000 - 5,100,000 (6,545 Users, 32,041 videos, 428gb) monst3r_ 4363
  • 5,150,000 - 5,200,000 (3,313 Users, 18,127 videos, 235gb) Guan (to underscor, paused)
  • 5,400,000 - 5,500,000 (6,897 Users, 738gb) dj_ryan 4421
  • 5,954,430 - 5,963,115 (700 Users) DoubleJ (split from km3k/amerrykan)
  • 6,300,000 - 6,400,000 (6,378 Users, 44,962 videos, 849gb) Guan (to underscor)
  • 6,400,000 - 6,500,000 (6,097 Users, 32,239 videos, 654GB) Masterono
  • 7,400,000 - 7,500,000 (4,204 Users, 22,560 videos, 290gb) ZPedro (to 4362)
  • 8,000,000 - 8,100,000 (3,416 Users, 16,781 videos, 201gb) bluknight D1
  • 8,300,000 - 8,400,000 (2,920 Users, 8,973 videos, 93gb) Masterono
  • 9,050,000 - 9,100,000 (1,796 Users, 6,142 videos, 79gb) Masterono

Finished

  • 0,015,000 - 0,020,000 (3,948 Users, 24,328 videos, 314gb) jakemcgra
  • 0,025,000 - 0,030,000 (4,040 Users) nwf
  • 0,030,000 - 0,035,000 (4,095 Users, 25,490 videos, 209gb) penemue/NovaKing
  • 0,040,000 - 0,045,000 (4,024 Users) PMT
  • 0,045,000 - 0,050,000 (4,051 Users, 31,110 videos, 288gb) Jason Scott/Ersi
  • 0,050,000 - 0,055,000 (3,976 Users) zumthing
  • 0,055,000 - 0,060,000 (4,086 Users, 13098 videos, 107GB) Lambda_Dr/Guan
  • 0,074,000 - 0,075,000 (801 Users, 5376 videos, 28 gb) widget (unavailable for upload until wed 16th)
  • 0,077,256 - 0,079,747 (2,000 Users, 10,603 videos, 102gb) Guan
  • 0,079,748 - 0,082,199 (2,000? Users) ksuter
  • 0,082,200 - 0,083,321 (1,000 Users, 6,536 videos, 60gb) Guan
  • 0,600,000 - 0,650,000 (3,398 Users, 16,861 videos, 182gb) robertcu (guan re-downloaded)
  • 0,650,000 - 0,700,000 (3,162 Users, 17,846 videos, 147gb) Hydruh
  • 0,750,000 - 0,800,000 (3,735 Users, 19,032 videos, 136G) atg
  • 0,800,000 - 0,850,000 (3,140 Users, 10,969 videos, 104G) crust/Guan
  • 0,850,000 - 0,900,000 (3,749 Users, 18,562 videos, 138gb) filer
  • 0,900,000 - 1,000,000 (6,453 Users, 18,077 videos, 139gb) K4K/NovaKing
  • 1,000,000 - 1,100,000 (6,331 Users, 31,241 videos, 247gb) Zumthing (original download + factcheck cleanup)
  • 1,200,000 - 1,300,000 (7,130 Users) drspangle (finished, guan doing factcheck)
  • 1,300,000 - 1,400,000 (7,598 Users) sep332
  • 1,400,000 - 1,500,000 (5,194 Users, 6,476 videos, 68gb) beardicus
  • 1,700,000 - 1,800,000 (4,368 Users, 8013 videos, 97G) atg
  • 1,800,000 - 1,900,000 (4,744 Users, 20,494 videos, 169g) bluknight
  • 2,200,000 - 2,300,000 (4,216 Users, 18,708 videos 168G) jakemcgra
  • 2,500,000 - 2,600,000 (3,949 Users, 15,994 videos, 146gb) Zumthing (original download + factcheck cleanup)
  • 2,600,000 - 2,700,000 (4,208 Users, 17,903 videos, 183G) Frigolit (utf8: ok, factcheck: ok)
  • 2,700,000 - 2,800,000 (4,754 Users, 22,600 videos, 192gb) ersi/Guan
  • 2,800,000 - 2,900,000 (5,623 Users, 23,879 videos, 249gb) amuck
  • 3,200,000 - 3,250,000 (2,603 Users) Soultcer (timestamps: ok, utf8: ok, factcheck done)
  • 3,250,000 - 3,300,000 (2,439 Users) taaz
  • 3,500,000 - 3,600,000 (4,895 Users, 22,964 videos, 254gb) tobbez/nwf
  • 3,800,000 - 3,900,000 (3,936 Users, 22,414 videos, 312gb) dysproseu (factchecked)
  • 3,900,000 - 4,000,000 (4,055 Users, 11,292 videos, 136gb ) HA/Novaking
  • 4,200,074 - 4,289,987 (4,250 Users) atg
    • 4,224,414 - 4,269,902 (? Users) penemue
    • 4,270,007 - 4,279,957 (438 Users, 2,477 videos, 28gb) guan
    • 4,280,010 - 4,289,987 (406 Users) nwf
  • 4,290,000 - 4,300,000 (456 Users, 2,030 videos, 38gb) savetz/penemue
  • 4,300,000 - 4,400,000 (4,355 Users, 11,947 videos, 157gb) NovaKing
  • 4,400,000 - 4,500,000 (4,715 Users, 21,569 videos, 299G) beardicus
  • 4,557,501 - 4,586,425 (1,713 Users) Lambda_dr (email dsnigier at gmail.com with uploading details...can keep on drive array for as long as needed)
  • 4,700,000 - 4,750,000 (2,665 Users, 8,229 videos, 104gb) Gandalfar/NovaKing
  • 4,750,000 - 4,784,985 (2,127 Users, 15,265 videos, 193gb) crust/Guan
  • 4,790,761 - 4,799,994 (400 Users, 5648 videos & html files, 26.8 gb) penemue (overlap with primus_102)
  • 4,800,000 - 4,900,000 (5,764 Users) jch (Downloaded, factchecking)
  • 4,900,000 - 4,913,495 (1,000 Users) nwf
  • 4,913,497 - 4,937,299 (1,500 Users) guan DONE (1,501 users, 9,030 videos, 113gb)
  • 4,937,299 - 4,945,193 (500 Users, 3411 videos, 41gb) yoyoguy2 DONE
  • 4,966,514 - 4,979,997 (1000 Users, 9680 Videos, 111gb) penemue DONE
  • 4,980,027 - 4,994,982 (780 Users, 11,375 html+flv, 61,9gb) penemue DONE
  • 4,995,045, - 4,999,999 (364 Users, 3293 videos, 33gb) penemue DONE
  • 5,255,391 - 5,299,993 (3,000 users, 23,847 videos, 330gb ) penemue
  • 5,280,381 - 5,291,717 (749 Users, 6,540 videos, 92gb) Guan
  • 5,200,000 - 5,300,000 (6,751 Users, 54,805 videos, 734gb) jakemcgra
  • 5,300,000 - 5,400,000 (6,289 Users, 36,617 videos, 556gb) jakemcgra
  • 5,800,020 - 5,899,977 (6,188 Users) jch/DoubleJ/Spirit/jakemcgra
    • 5,842,906 - 5,850,604 (500 Users, 4,663 videos 91gb) jakemcgra
    • 5,882,964 - 5,894,410 (~725 users, ? videos, 119.5gb) penemue
    • 5,894,410 - 5,899,977 (? Users, ~ 45gb) penemue done
  • 5,900,005 - 5,999,982
    • 5,900,005 - 5,912,492 (700 users, 4133 videos, 87GB) km3k (confirmed done)
    • 5,912,497 - 5,944,029 (? users) amerrykan, done)
    • 5,932,108 - 5,944,802 (~750 users, ? videos, 68.7GB) SudoSeth/Guan
    • 5,944,810 - 5,954,407 (750 Users) penemue (split from km3k/amerrykan)
    • 5,963,126 - 5,967,009 (300 users) guan (split from DoubleJ's split)
    • 5,967,027 - 5,974,072 (500 Users, 81.9gb) penemue (from DoubleJ's split)
    • 5,974,076 - 5,999,982 (2,000 Users) nwf
  • 6,100,000 - 6,200,000 (7,433 Users) incendui (done, except for 897 user missing-hotlist, which was sent to guan, status? yoyoguy2 also working)
    • 6,169,448 - 6,199,997 Guan (2k users, done)
    • 6,156,334 - 6,169,434 Lamda_Dr (done)
  • 6,600,000 - 6,650,000 (3,516 Users, 12,404 videos, 277GB) Lambda_Dr/Guan (timestamps updated, factcheck complete)
  • 6,650,000 - 6,666,477 (1,000 Users, 5,470 videos, 109gb) sdboyd
  • 6,650,000 - 6,700,000 (2049 users, 15,021 videos, 293gb) jakemcgra
  • 6,700,000 - 6,750,000 (3,079 Users, 23,950 videos, 510gb) yoyoguy2
  • 6,750,000 - 6,800,000 (2,838 Users, 17,555 videos, 356gb) Jason Scott (Transferring to data dump 1004)
  • 6,900,000 - 7,000,000 (5,676 Users) bluknight (sandra)
  • 7,000,000 - 7,100,000 (4,833 Users, 24,523 videos, 311gb) bluknight
  • 7,100,000 - 7,200,000 (4,026 Users, 13,244 videos, 167gb) zumthing
  • 7,200,000 - 7,300,000 (3,939 Users, 21,731 videos, 292G) dysproseu (factchecked)
  • 7,300,000 - 7,400,000 (4,304 Users, 23,904 videos, 307gb) amerrykan done, factchecked
  • 7,700,000 - 7,800,000 (4,028 Users, 15,037 videos, 171gb) K4K
  • 7,700,000 - 7,800,000 (4,028 Users, 19,155 videos, 219gb) jakemcgra
  • 7,900,000 - 8,000,000 (3,517 Users, 10,392 videos, 111gb) matzie
  • 8,100,000 - 8,200,000 (3,380 Users, 13,882 videos. 160gb) Primus104 (Transferring to data dump 4362)
  • 8,400,000 - 8,500,000 (3,535 Users, 16,784 videos, 199gb) yoyoguy2
  • 8,500,000 - 8,600,000 (3,516 Users, 19,858 videos, 200GB) red_cell
  • 8,700,000 - 8,800,000 (3,730 Users, 16,698 videos, 186gb) t0phux
  • 8,850,000 - 9,000,000 (1,770 Users, 8,832 videos, 106gb) Gandalfar (uploading to 4482)
  • 8,900,000 - 9,000,000 (3,793 Users, 10,790 videos, 167gb) khaoohs/NovaKing
  • 9,000,000 - 9,050,000 (1,745 Users, 6,889 videos, 86G) atg
  • 9,100,000 - 9,150,000 (1,989 Users, 9,380 videos, 111gb) Killface/yoyoguy2
  • 9,150,000 - 9,200,000 (5,173 Users, 22,655 videos, 261gb) Underscor
  • 9,200,000 - 9,300,000 (8,941 Users, 30,179 videos, 341gb) Zumthing (drspangle has a duplicate subset)
  • 9,300,000 - 9,360,000 (2,179 Users, 4,406 videos, 51gb) Robertcu
  • freaks (1,163 Users): Guan
  • each 50,000st user from the whole range (112 videos, 1G) Spirit (factchecked and done)
  • 0,000,000 - 0,100,000 ksuter has done a safety-run of this historically important range

In-Process

  • 1,900,000 - 2,000,000 (4,481 Users, 10,346 videos, 79G) ajclement/swebb/NovaKing
  • 4,600,000 - 4,700,000 (5,151 Users) eberon NOC (jakemcgra doing just-in-case grab)
  • 5,600,000 - 5,700,000 (5,903 Users) _kp NOC (Hydruh doing just-in-case grab)
  • 5,700,000 - 5,800,000 (3,194 Users) finished 91.300% (2918/3194) 3/21
  • 6,050,000 - 6,100,000 (3,747 Users) t0phux (NovaKing doing backup grab)

Cleanup in progress

  • Spangle combined missing-hotlist (416 users) Guan

UTF-8 Errors What

Some of the filenames in Yahoo! are encoded in Unicode, which flips out on some machines. If you run grapefruit on user 45002 and see errors like "UnicodeEncodeError: 'ascii' codec can't encode characters", then you have this issue. At this point in time, we can't use you, but we love you anyway - come cheer us on.

Phase Three: Final Resting Place

Yahoo!: We can't afford storage, but we can afford Zambonis

In an ideal world, all the downloaders would hold copies and everyone would just torrent the thing. That's not going to happen in current contemporary machine capacity. Our very rough estimate is that a download of Yahoo! Video is going to be 20 terabytes. While this is under a thousand dollars of newegg-purchased disks, it's still a bit much to ask of anyone. Meanwhile, the more forward-looking of volunteers are asking, "what's the exit strategy?".

Jason and a few others have locations that video downloaders can upload a copy of their acquired data to; these are being placed on a stack of hard drives. At the same time, Archive.org is providing a location for the data to be uploaded to as well. Other entities are being contacted at the present about providing space for copies of the full archive, as well.

The priority, however, is to get all the data in the first place.

Verified and On-Drive

Talk to underscor to get your stuff uploaded. (I will perform a final check of the timestamps)

Set 1 (Disks 1 and 2)

Being used as upload holding pen at the moment

Set 2 (Disks 3 and 4)

  • 0,300,000 - 0,400,000 (6,909 Users, 41,131 videos, 342gb) Underscor
  • 0,500,000 - 0,600,000 (6,543 Users, 13,849 videos, 115gb) NovaKing
  • 0,700,000 - 0,750,000 (3,213 Users, 20,767 videos, 127gb) Underscor
  • 1,500,000 - 1,600,000 (5,252 Users, 7,642 videos, 86g) NovaKing
  • 2,100,000 - 2,200,000 (4,519 Users, 16,597 videos, 144g) Joey
  • 3,000,000 - 3,100,000 (6,297 Users, 18,418 videos, 163gb) Guan
  • 7,600,000 - 7,700,000 (4,200 Users, 17,885 videos, 189gb) NovaKing

Set 3 (Disks 5 and 6)

  • 0,060,000 - 0,065,000 (2,226 Users, 23,254 videos, 228gb) underscor
  • 0,065,000 - 0,070,000 (4,077 Users, 23,777 videos, 223gb) underscor
  • 0,070,000 - 0,072,000 (1,613 Users, 10,341 videos, 98gb) underscor
  • 0,075,000 - 0,077,255 (1,801 Users, 5,405 videos, 41GB) Spirit
  • 1,100,000 - 1,200,000 (7,613 Users, 12,892 videos, 177gb ) NovaKing
  • 2,000,000 - 2,100,000 (4,408 Users, 20,583 videos, 167g) dashcloud
  • 3,400,000 - 3,500,000 (4,560 Users, 11,450 videos, 102gb) NovaKing
  • 5,100,000 - 5,150,000 (3,254 Users, 19701 videos, 327gb) Spirit
  • 8,800,000 - 8,850,000 (2,170 Users, 8,368 videos, 71gb ) sdboyd

Set 4 (Disks 7 and 8)

  • 0,000,000 - 0,005,000 (3,960 Users, 27,357 videos, 268gb) Underscor
  • 0,005,000 - 0,010,000 (3,993 Users, 25,858 videos, 248gb) Underscor
  • 0,035,000 - 0,040,000 (3,966 Users, 24,164 videos, 225gb) penemue
  • 2,400,000 - 2,500,000 (2,802 Users, 6,260 videos, 48gb) NovaKing
  • 3,300,000 - 3,400,000 (4,252 Users, 10,672 videos, 93gb) NovaKing
  • 3,600,000 - 3,700,000 (4,494 Users, 21,863 videos, 251gb) chronomex
  • 3,700,000 - 3,800,000 (4,262 Users, 11,790 videos, 142gb) NovaKing
  • 6,000,000 - 6,050,000 (3,571 Users) savetz

Set 5 (Disks 9 and 10)

  • 0,067,800 - 0,070,000 (1,792 Users, 9725 Videos, 100gb) penemue
  • 0,182,338 - 0,199,999 (1118 users, 5681 videos. 45gb) amnesia
  • 7,500,000 - 7,600,000 (4,347 Users, 22,758 videos, 287gb) sadcarrot/amnesia

Set 6 (Disks 11 and 12)

  • 0,018,500 - 0,020,000 (x users, x videos, xgb) penemue
  • 0,020,000 - 0,025,000 (4,053 Users, 27,844 videos, 328gb) guan
  • 0,083,322 - 0,100,003 (5,000 Users, 34,236 videos, 324gb) Guan
  • 5,835,908 - 5,842,902 (518 users, 3,464 videos, 66gb) Guan
  • 6,800,000 - 6,900,000 (5,234 Users, 26,243 videos, 558gb) dashcloud
  • guan's cleanup files (bits of other ranges)


Set 7 (Disks 13 and 14)

Disks still need purchasing

Set 8 (Disks 15 and 16)

Disks still need purchasing

Set 9 (Disks 17 and 18)

Disks still need purchasing

Set 10 (Disks 19 and 20)

Disks still need purchasing


Statistics

"Why didn't more people use it! It was awesome!"

People like easy to quote statistics from a project, so here are some, with others coming soon.

Yahoo! Video officially started as a user content system in June 2006. Previously, it had been a directory of other video sites around the net (harkening back to Yahoo's previous life as a directory instead of its short burst as a content creator). From the start, it was called a "YouTube Clone" and other derisive terms.

However, users did decide to go for this new environment. Archive Team found about 9,360,000 accounts in use on Yahoo Video. "In Use" is a relative term, of course - many, many users consist of no uploaded videos, no user picture, and no other activity - just a name. We'll have more, but for example, of 1.5 million accounts we've scanned so far, only 146,600 have uploaded videos, and if you shear off the first 100,000 (i.e., frontloaded with content partner) accounts, that 10% figure drops down to 5%.

It didn't take very long for users to figure out Yahoo! Video is not the way to go, either as participants or as viewers. The most popular video, "Baby Raccoons Saved from Soda Machine" has 14 million views. Compare with one of the top videos on YouTube - Justin Beiber performing "Baby" with Ludacris, which has 457 million views. By the time the raccoons are viewed, the next most popular video, "Worst Best Man Ever", clocks in at just over 10 million views. After the first 20 videos, the views per video are down to about 4 million. By the time we get to the top 100, we're already skirting 1 million. Again, a single video on YouTube, the most popular, has a number of views basically the same as the top 100 Yahoo! videos combined.