Difference between revisions of "Yahoo! Groups"

From Archiveteam
Jump to navigation Jump to search
(the big it's-dead edit)
(40 intermediate revisions by 7 users not shown)
Line 4: Line 4:
| image = groups-yahoo-com.png
| image = groups-yahoo-com.png
| logo = yahoo-groups-logo.png
| logo = yahoo-groups-logo.png
| project_status = {{closing}}
| project_status = {{offline}}
| archiving_status = {{inprogress}}
| archiving_status = {{partiallysaved}}
| tracker = [https://tracker.archiveteam.org/yahoogroups/ yahoogroups], [http://tracker-test.ddns.net/yahoo-groups-api/ yahoo-groups-api]
| source = [https://github.com/ArchiveTeam/yahoogroups-grab yahoogroups-grab], [https://github.com/ArchiveTeam/yahoo-group-archiver, yahoo-group-archiver]
| irc = yahoosucks
| irc = yahoosucks
| irc_network = hackint
}}
}}


'''Yahoo! Groups''' is Yahoo's combination mailing list service/web forum; it's the result of the acquisition of eGroups and some other Yahoo! stuff. In addition to archives of and a web interface for mailing lists, it offers file uploads, photo uploads, links, polls, and an events calendar.
'''Yahoo! Groups''' was [[Yahoo!]]'s combination mailing list service/web forum, the result of the acquisition of eGroups and some other Yahoo! stuff. In addition to archives of and a web interface for mailing lists, it offered file uploads, photo uploads, links, polls, and an events calendar. It had been stable since the late 90s, long enough for some specialised software to be developed to do backups of it. (Not many other websites can say ''that''.) It was shuttered in stages over the course of 2019–2020.


Uploading of new content will be disabled 28 October 2019, and all content, including message history, will be deleted 14 December 2019.<ref>https://help.yahoo.com/kb/groups/SLN31010.html</ref> (The mailing lists themselves will continue to function.)
Uploading of new content was disabled 2019-10-28, and all content, including message history, was made unavailable on 2019-12-14.<ref>https://web.archive.org/web/20201126125219/https://help.yahoo.com/kb/groups/SLN31010.html</ref> Group content was hidden from the web interface by 2019-12-21. After negative media attention, Yahoo announced that they were extending the deadline for users to use their official "GetMyData" export tool (which missed a plethora of attachments, databases, polls, photos, and metadata) to 2020-01-31.<ref>{{URL|https://www.theverge.com/2019/12/10/21004883/yahoo-groups-extend-deadline-download-data-date-time}}</ref><ref>{{URL|https://twitter.com/YahooCare/status/1204312076379926528}}</ref> They stopped accepting GMD requests on 2020-02-04.


It's been stable for a long time (since the late 90s), long enough for some specialised software to be developed to do backups of it. (Not many other websites can say ''that''.)
Groups continued to function solely as mailing lists for a short period. However, the creation of new groups was disabled on 2020-10-12, and the web interface and mailing lists were shut down on 2020-12-16.<ref name="websiteshutdown">{{URL|https://help.yahoo.com/kb/groups/SLN35505.html}}</ref>


== Nominating Notable Non-Private Groups for Archival ==
Group admins and members, please see [[Yahoo! Groups/Archiving Project FAQ for group members]] or join IRC if you have questions.


Groups can be nominated for archival using [https://tinyurl.com/savegroups this form]. Please note that this form should not be used for groups that require administrator approval to join.
== Submitting group data to the public archive ==


== Adding Private Groups to the Public Archive ==
Were you a member of a public group (one that did not require administrator approval to join)? Were you an admin of a private group whose members consent to be part of the public archive? Did you save the group yourself, using GetMyData or any other method?


Administrators / Moderators can request that their private group (we consider a private group to be one that requires approval for new members) be included in the public archive. Before you do this, please ensure that the members of the group are happy about being part of the public archive.
If so, we'd love to have your archives. Upload to a fileshare such as WeTransfer, Dropbox, Google Drive, or Mega.nz and [mailto:archiveteamprivateyahoogroup@gmail.com email us] a link.


To add the group to the list of private groups to be archived, all you need to do is [https://help.yahoo.com/kb/SLN2567.html send a membership invite] to the email ''archiveteamprivateyahoogroup@gmail.com''. (Note that only group admins can do this). We'll be monitoring that email regularly to accept any membership requests we receive. Once that account is a member, the group should be scheduled to be part of the public archive.
Feel free to remove data which should remain private (such as private groups in mixed public/private GetMyData results, or message history from private groups whose members wish to make only files and photos public) before sending us a copy.
 
However, try to make sure the data is otherwise unmodified! In particular, there may be old malware in GMD ZIP files. Modern email software and operating systems are expected to be resistant to this old malware, but some antivirus software may see it and attempt to modify or delete the ZIP file. Please be careful of this!
 
== Project history ==
 
Data collection for this project is over. Yahoo! Groups content is now inaccessible; although we continue to accept individual archives made by group members and admins, we can no longer archive additional groups.
 
While the project was active, volunteers could help in the following ways:
 
=== Nominating non-private groups for archival ===
 
Groups could be nominated for archival using [https://tinyurl.com/savegroups this form]. This was not used for groups that required administrator approval to join.
 
=== Submitting private groups for public archival ===
 
Administrators could request that their private group (we considered a private group to be one that required administrator approval of new members) be included in the public archive. We requested admins to ensure that the members of the group were happy about being part of the public archive.
 
To submit a group for archival, admins could [https://web.archive.org/web/20201020005238/https://help.yahoo.com/kb/SLN2567.html send a membership invite] to the email ''archiveteamprivateyahoogroup@gmail.com'' (''without'' selecting the "Add only to mailing list" option). We monitored that email regularly to accept any membership requests we received, and scheduled the group for archival once our Yahoo account was a member.
 
=== Joining groups and submitting data ===
 
We used [https://github.com/davidferguson/yahoogroups-joiner an extension for Chromium-based browsers] to partially automate the process of joining groups (at first since some groups only made message history visible to members, and later because after the closure of the web interface GetMyData was the only way to access group content). There was at one time a [https://df58.host.cs.st-andrews.ac.uk/yahoogroups/leaderboard leaderboard].
 
Volunteers who joined groups also made GetMyData requests from the accounts they used to join groups (in some cases, multiple requests, if they received results in time to continue joining groups or not all groups were included in the initial result). GMD requests could take up to 10 days to be processed; results were split into 2 GB ZIP files.
 
GMD results were emailed or rsynced to ArchiveTeam.
 
== Private groups of interest ==
 
[https://groups.yahoo.com/neo/groups/numberactivation/info numberactivation] (see all [https://reclaimthenet.org/ofcom-oftel-uk-phone-numbers-yahoo-groups/ the] [https://www.axios.com/yahoo-groups-ofcom-cell-phone-number-porting-51949f81-446e-4b4b-82eb-26790146e9a0.html press] [https://techupdatess.com/some-of-the-uks-phone-number-infrastructure-relies-on-yahoo-groups-the-verge/ coverage]; [https://www.whatdotheyknow.com/request/all_data_held_in_yahoo_groups_us FOI request]). Some external lists: [https://fanlore.org/wiki/Category:Yahoo!_Groups List of groups with Fanlore pages] (contains both private and public groups), [https://archivetransyahoo.noblogs.org/list-of-known-trans-groups/ Archive Trans Yahoo's list] (all private at last check), [https://yahoogroups.southasianamerican.org/ Archive South Asian American Yahoo Groups] (all public), and [https://queerdigital.com/ygpresproject Queer Digital History Project] (no groups listed, presumably all private).


Please make sure that when you invite the Archive Team account, you do '''not''' select the ''Add only to mailing list'' option, as this will prevent Archive Team from archiving the group.
== Statistics ==
== Statistics ==


Line 35: Line 67:
[[File:Yahoo_groups_messages_per_group.png‎]]
[[File:Yahoo_groups_messages_per_group.png‎]]
[[File:Yahoo_groups_post_date.png‎]]
[[File:Yahoo_groups_post_date.png‎]]
== Private groups of interest ==
{| class="wikitable"
! Group
! Notes
! Admin consent?
|-
| [https://groups.yahoo.com/neo/groups/numberactivation/info numberactivation]
| see all [https://reclaimthenet.org/ofcom-oftel-uk-phone-numbers-yahoo-groups/ the] [https://www.axios.com/yahoo-groups-ofcom-cell-phone-number-porting-51949f81-446e-4b4b-82eb-26790146e9a0.html press] [https://techupdatess.com/some-of-the-uks-phone-number-infrastructure-relies-on-yahoo-groups-the-verge/ coverage]
| Not yet contacted; [https://www.whatdotheyknow.com/request/all_data_held_in_yahoo_groups_us FOI request] made
|-
| [https://groups.yahoo.com/neo/groups/hpslash/info hpslash]
| see [https://fanlore.org/wiki/Hpslash_%28mailing_list%29 Fanlore page]
| Not yet contacted
|}
Potentially relevant: [https://fanlore.org/wiki/Category:Yahoo!_Groups List of groups with Fanlore pages] (contains both private and public groups), [https://archivetransyahoo.noblogs.org/list-of-known-trans-groups/ Archive Trans Yahoo's list] (all private at last check)


== Site structure ==
== Site structure ==


There’s a convenient JSON API. Some endpoints require logged-in group membership or other permissions (depending on group settings).
There was a convenient JSON API, most endpoints of which are now down. Some endpoints require logged-in group membership or other permissions (depending on group settings).


===Groups===
===Groups===
Line 66: Line 80:
:- Known params: start, intlCode (au, in, sg, uk, us; ar, e1, es, mx; br; cf, fr; de; hk; it...)
:- Known params: start, intlCode (au, in, sg, uk, us; ar, e1, es, mx; br; cf, fr; de; hk; it...)
:- Pagination: Page size is 10. Does ''not'' have a count param. start is the result index, not the group id. start values 500 and up all return the same set of results.
:- Pagination: Page size is 10. Does ''not'' have a count param. start is the result index, not the group id. start values 500 and up all return the same set of results.
: Groups are listed in fixed but arbitrary order. /0/ is a special value that shows the root node; subcategories can be accessed by using the subcategory id instead (the full "idList" value is not required).  
: Groups are listed in fixed but arbitrary order. /0/ is a special value that shows the root node; subcategories can be accessed by using the subcategory id instead (the full "idList" value is not required).
: Defaults to the US view of the English directory tree. Different languages have different directory trees. Supplying a different intlCode parameter (list not exhaustive, must be lower case) accesses the corresponding view of the appropriate language's tree. Subcategory ids are language-specific and must be used with an appropriate intlCode. The intlCode -> language mapping may be checked at the /0/ endpoint; the root "name" is always "ROOT", but "id" is language-specific.<ref>This id can also be accessed with an appropriate intlCode, but contains the same twelve groups for all languages: the groups in the categories for musical artists "Roots, The" and "Rusted Root", three groups which appear to be Yahoo tests, and one group which appears to be a spam test.</ref> Different intlCode views of the same language list groups in a different order, may have slightly different category names, and appear to have slightly different numbers of categories in the full tree; their group overlap is about 99%.
: Defaults to the US view of the English directory tree. Different languages have different directory trees. Supplying a different intlCode parameter (list not exhaustive, must be lower case) accesses the corresponding view of the appropriate language's tree. Subcategory ids are language-specific and must be used with an appropriate intlCode. The intlCode -> language mapping may be checked at the /0/ endpoint; the root "name" is always "ROOT", but "id" is language-specific.<ref>This id can also be accessed with an appropriate intlCode, but contains the same twelve groups for all languages: the groups in the categories for musical artists "Roots, The" and "Rusted Root", three groups which appear to be Yahoo tests, and one group which appears to be a spam test.</ref> Different intlCode views of the same language list groups in a different order, may have slightly different category names, and appear to have slightly different numbers of categories in the full tree; their group overlap is about 99%.
: The "count" field appears totally inaccurate.
: The "count" field appears totally inaccurate.


* https://groups.yahoo.com/api/v1/groups/concatenative/ (specific group information)
* https://groups.yahoo.com/api/v1/groups/concatenative/ (specific group information)
* https://groups.yahoo.com/api/v1/groups/concatenative/statistics (more group information, with partial overlap)


===Messages===
===Messages===
Line 80: Line 95:
* https://groups.yahoo.com/api/v1/groups/concatenative/messages/1/ (specific message)
* https://groups.yahoo.com/api/v1/groups/concatenative/messages/1/ (specific message)
* https://groups.yahoo.com/api/v1/groups/concatenative/messages/1/raw (specific message, raw content including headers)
* https://groups.yahoo.com/api/v1/groups/concatenative/messages/1/raw (specific message, raw content including headers)
: Some messages may have encoding issues.<ref>https://yahoo.uservoice.com/forums/209451-us-groups/suggestions/9644478-displaying-raw-messages-is-not-8-bit-clean</ref> Sometimes (as in the linked case) the non-raw endpoint has the correct characters, sometimes it does not; this is likely related to the originating email client.
: Original email is largely recoverable from ''rawEmail'' field.
: Message headers and textual body parts have email addresses redacted, with the hosts replaced with "...". For example, "From: ceo@ford.com" and "From: ceo@toyota.com" both get turned into "From: ceo@..." Some addresses may not have been redacted correctly.
: Some messages may have encoding issues.<ref>https://yahoo.uservoice.com/forums/209451-us-groups/suggestions/9644478-displaying-raw-messages-is-not-8-bit-clean</ref> Sometimes (as in the linked case) the non-raw endpoint has the correct characters, sometimes it does not; this is likely related to the originating email client. Removing non-ASCII characters and ^M characters from the 7-bit text should result in valid RFC822 emails.
: Some emails longer than 64kb (minus attachments) may be truncated. This truncation affects not just plain text, but also HTML and encoded Base64 content. Deleting the string "\n(Message over 64 KB, truncated)" from the end of the message part may help prevent parser breakage.
: All attachments are separated, with attachment bodies replaced with the string "[ Attachment content not displayed ]". Recovering the emails involves finding those MIME parts, looking at the filenames, comparing with the list of filenames listed in the "attachmentInfo" section, matching on similarity, and replacing the contents with the downloaded attachments. In very rare cases where a matching MIME section isn't found, it may be necessary to append those attachments as new MIME attachments to the email while reconstructing.


* https://groups.yahoo.com/api/v1/groups/concatenative/history (calendar summary)
* https://groups.yahoo.com/api/v1/groups/concatenative/history (calendar summary)
Line 127: Line 146:


* https://groups.yahoo.com/api/v3/groups/a_furrys_world/albums/1841906391 (specific album)
* https://groups.yahoo.com/api/v3/groups/a_furrys_world/albums/1841906391 (specific album)
:- Observed parameters similar to photos and albums endpoints, with additional ordinal sortOrder option
:- Known params: similar to /photos and /albums endpoints, with additional ordinal sortOrder option
: Photomatic albums ''must'' be loaded with the albumType parameter set to PHOTOMATIC.
: Photomatic albums ''must'' be loaded with the albumType parameter set to PHOTOMATIC.


Line 159: Line 178:


* https://groups.yahoo.com/api/v1/groups/iswipe/members/confirmed (list of confirmed members)
* https://groups.yahoo.com/api/v1/groups/iswipe/members/confirmed (list of confirmed members)
:- Known params: count, start, sortBy, sortOrder, ts, tz, chrome.
:- Known params: count, start, sortBy, sortOrder, ts, tz, chrome
:- Pagination: Page size defaults to 10, with a limit of 100. No known limit on total results.
:- Pagination: Page size defaults to 10, with a limit of 100. No known limit on total results.
: May be blocked for normal members (as may all the other members endpoints). Includes moderators and bouncing members, with identifying metadata.
: May be blocked for normal members (as may all the other members endpoints). Includes moderators and bouncing members, with identifying metadata.
Line 173: Line 192:
Overlaps with Yahoo Calendar API, check yahoo-group-archiver code.
Overlaps with Yahoo Calendar API, check yahoo-group-archiver code.


== Python Yahoo! Group archivers ==  
== Software for archiving groups ==  


* [https://github.com/IgnoredAmbience/yahoo-group-archiver/network/members yahoo-group-archiver] scrapes a group using the JSON API and (for private endpoints) the two cookies Yahoo uses to verify a logged-in user. <s>Relevant forks include [https://github.com/Frankkkkk/yahoo-group-archiver Frankkkkk] and [https://github.com/nsapa/yahoo-group-archiver nsapa]. Needs merging. Various branches have support (largely untested) for file attachments, photos, links, folders, and events.</s> Most stuff has been merged back into IgnoredAmbience's master. (Exceptions: full WARC support?, mtime work from Frankkkkk.) Needs consistent/WARC-appropriate handling for random 500 errors (require retries), attachment 404s (appear permanent), and 502 permissions errors (definitely permanent, currently halt script).
=== Python ===


* [https://github.com/andrewferguson/YahooGroups-Archiver YahooGroups-Archiver] is similar, but scrapes only messages (not files or any other data). It is not currently under active development.
* '''[https://github.com/ArchiveTeam/yahoo-group-archiver yahoo-group-archiver]''' scraped a group using the JSON API and (for private endpoints) the two cookies Yahoo uses to verify a logged-in user. Optionally, it could produce WARCs. ArchiveTeam's preferred tool and fully featured at the time of closure.
** [https://github.com/anirvan/yahoo-group-archive-tools Yahoo Group Archive Tools] (a Perl script) converts yahoo-group-archiver output into clean rfc822 and mbox files, with separated attachments correctly reattached, and many Yahoo truncation/redaction bugs corrected. It also turns list archives into PDF, using [https://github.com/andrewferrier/email2pdf email2pdf], which many non-technical list owners prefer.


* [https://github.com/csaftoiu/yahoo-groups-backup yahoo-groups-backup] scrapes a group using Selenium, storing message info and metadata (both rendered message body and raw email) into a Mongo database. It also provides a script to dump its data to static HTML pages that can be viewed in the browser.
* [https://github.com/andrewferguson/YahooGroups-Archiver YahooGroups-Archiver] is similar, but scraped only messages (not files or any other data). It has been deprecated in favor of the above.


== Other archivers ==
* [https://github.com/csaftoiu/yahoo-groups-backup yahoo-groups-backup] scraped a group's messages and files (but not any other data) using Selenium, storing message info and metadata (both rendered message body and raw email) into a Mongo database. It also provides a script to dump its data to static HTML pages that can be viewed in the browser.


=== Other ===
* [http://www.personalgroupware.com/ PGOffline]: Windows, proprietary. 14-day free trial, after which download and export is disabled (but view still works). Included attachments. Stores data in a SQLite database internally.
** [https://github.com/nsapa/pgo2mbox/ pgo2mbox] converts PGOffline pg4 files to mbox.
* [http://yahoogroupedia.pbworks.com/w/page/93006447/Chrome%20Application%20To%20Download%20Messages Yahoo Messages Export]: Chrome extension. Messages only. Saves as mbox.
* [https://sourceforge.net/projects/grabyahoogroup/ Yahoo Group Archiver]: Perl, defunct.
* [https://sourceforge.net/projects/grabyahoogroup/ Yahoo Group Archiver]: Perl, defunct.
* [http://www.personalgroupware.com/ PGOffline]: Windows, proprietary. 14-day free trial, after which download and export is disabled (but view still works). Includes attachments. Stores data in a SQLite database internally.
 
* [http://yahoogroupedia.pbworks.com/w/page/93006447/Chrome%20Application%20To%20Download%20Messages Yahoo Messages Export]: Chrome extension. Messages only. Saves as mbox.
== Software for viewing archives (in mbox format) ==
 
* [https://www.thunderbird.net/ Mozilla Thunderbird]
** Method using addon: https://addons.thunderbird.net/en-US/thunderbird/addon/importexporttools-ng/
** Method without addon: https://commons.lbl.gov/display/~jwelcher@lbl.gov/Reading+an+mbox+file+with+Thunderbird
* [https://sylpheed.sraoss.jp/en/ Sylpheed]
** Instructions: https://sylpheed.sraoss.jp/doc/manual/en/sylpheed-15.html
** Detailed instructions from Doranwen: https://docs.google.com/document/d/1dXeXfY5Huri_8NTUn4hl-iUZq9MMRL1Qbo7bp5YZpmE/edit
* [https://kde.org/applications/internet/org.kde.kmail2 KMail]
* [http://www.mutt.org/ Mutt]
* [https://neomutt.org/ NeoMutt]
 
== Other archiving efforts ==
 
* Yahoo Groups Fandom Rescue Project
** https://discord.gg/DyCNddf
**archiver1.fandom@gmail.com
* Mods and Members
** https://modsandmembersblog.wordpress.com/
** https://mmsanctuary.groups.io/g/main
** https://twitter.com/featheredleader


== External Links ==
== External Links ==


* https://archive.org/details/archiveteam_yahoogroups
* {{IA item|archiveteam_yahoogroups}}


== Coverage ==
== Coverage ==

Revision as of 07:35, 26 January 2021

Yahoo! Groups
Yahoo! Groups logo
Groups-yahoo-com.png
URL https://groups.yahoo.com/
Status Offline
Archiving status Partially saved
Archiving type Unknown
Project source yahoogroups-grab, yahoo-group-archiver
Project tracker yahoogroups, yahoo-groups-api
IRC channel #yahoosucks (on hackint)

Yahoo! Groups was Yahoo!'s combination mailing list service/web forum, the result of the acquisition of eGroups and some other Yahoo! stuff. In addition to archives of and a web interface for mailing lists, it offered file uploads, photo uploads, links, polls, and an events calendar. It had been stable since the late 90s, long enough for some specialised software to be developed to do backups of it. (Not many other websites can say that.) It was shuttered in stages over the course of 2019–2020.

Uploading of new content was disabled 2019-10-28, and all content, including message history, was made unavailable on 2019-12-14.[1] Group content was hidden from the web interface by 2019-12-21. After negative media attention, Yahoo announced that they were extending the deadline for users to use their official "GetMyData" export tool (which missed a plethora of attachments, databases, polls, photos, and metadata) to 2020-01-31.[2][3] They stopped accepting GMD requests on 2020-02-04.

Groups continued to function solely as mailing lists for a short period. However, the creation of new groups was disabled on 2020-10-12, and the web interface and mailing lists were shut down on 2020-12-16.[4]

Group admins and members, please see Yahoo! Groups/Archiving Project FAQ for group members or join IRC if you have questions.

Submitting group data to the public archive

Were you a member of a public group (one that did not require administrator approval to join)? Were you an admin of a private group whose members consent to be part of the public archive? Did you save the group yourself, using GetMyData or any other method?

If so, we'd love to have your archives. Upload to a fileshare such as WeTransfer, Dropbox, Google Drive, or Mega.nz and email us a link.

Feel free to remove data which should remain private (such as private groups in mixed public/private GetMyData results, or message history from private groups whose members wish to make only files and photos public) before sending us a copy.

However, try to make sure the data is otherwise unmodified! In particular, there may be old malware in GMD ZIP files. Modern email software and operating systems are expected to be resistant to this old malware, but some antivirus software may see it and attempt to modify or delete the ZIP file. Please be careful of this!

Project history

Data collection for this project is over. Yahoo! Groups content is now inaccessible; although we continue to accept individual archives made by group members and admins, we can no longer archive additional groups.

While the project was active, volunteers could help in the following ways:

Nominating non-private groups for archival

Groups could be nominated for archival using this form. This was not used for groups that required administrator approval to join.

Submitting private groups for public archival

Administrators could request that their private group (we considered a private group to be one that required administrator approval of new members) be included in the public archive. We requested admins to ensure that the members of the group were happy about being part of the public archive.

To submit a group for archival, admins could send a membership invite to the email archiveteamprivateyahoogroup@gmail.com (without selecting the "Add only to mailing list" option). We monitored that email regularly to accept any membership requests we received, and scheduled the group for archival once our Yahoo account was a member.

Joining groups and submitting data

We used an extension for Chromium-based browsers to partially automate the process of joining groups (at first since some groups only made message history visible to members, and later because after the closure of the web interface GetMyData was the only way to access group content). There was at one time a leaderboard.

Volunteers who joined groups also made GetMyData requests from the accounts they used to join groups (in some cases, multiple requests, if they received results in time to continue joining groups or not all groups were included in the initial result). GMD requests could take up to 10 days to be processed; results were split into 2 GB ZIP files.

GMD results were emailed or rsynced to ArchiveTeam.

Private groups of interest

numberactivation (see all the press coverage; FOI request). Some external lists: List of groups with Fanlore pages (contains both private and public groups), Archive Trans Yahoo's list (all private at last check), Archive South Asian American Yahoo Groups (all public), and Queer Digital History Project (no groups listed, presumably all private).

Statistics

As of 2019-10-16 the directory lists 5619351 groups. 2752112 of them have been discovered. 1483853 (54%) have public message archives with an estimated number of 2.1 billion messages (1389 messages per group on average so far). 1.8 billion messages (86%) have been archived as of 2018-10-28.

The following graphs are slightly outdated:

Yahoo groups date created.png Yahoo groups messages per group.png Yahoo groups post date.png

Site structure

There was a convenient JSON API, most endpoints of which are now down. Some endpoints require logged-in group membership or other permissions (depending on group settings).

Groups

- Known params: maxHits, offset, query, sortBy (values: OLDEST, RELEVANCE, MEMBERS, LATEST_ACTIVITY, NEWEST)
- Known params: start, intlCode (au, in, sg, uk, us; ar, e1, es, mx; br; cf, fr; de; hk; it...)
- Pagination: Page size is 10. Does not have a count param. start is the result index, not the group id. start values 500 and up all return the same set of results.
Groups are listed in fixed but arbitrary order. /0/ is a special value that shows the root node; subcategories can be accessed by using the subcategory id instead (the full "idList" value is not required).
Defaults to the US view of the English directory tree. Different languages have different directory trees. Supplying a different intlCode parameter (list not exhaustive, must be lower case) accesses the corresponding view of the appropriate language's tree. Subcategory ids are language-specific and must be used with an appropriate intlCode. The intlCode -> language mapping may be checked at the /0/ endpoint; the root "name" is always "ROOT", but "id" is language-specific.[5] Different intlCode views of the same language list groups in a different order, may have slightly different category names, and appear to have slightly different numbers of categories in the full tree; their group overlap is about 99%.
The "count" field appears totally inaccurate.

Messages

- Known params: count, start, sortOrder (ASC, DESC), direction (1, -1)
- Pagination: Page size defaults to 10, with no known limit. start is the message id, not the result index. sortOrder adjusts the order of results in the json response's array, whereas direction determines which way to iterate through ids from start (default: DESC, -1).
Original email is largely recoverable from rawEmail field.
Message headers and textual body parts have email addresses redacted, with the hosts replaced with "...". For example, "From: ceo@ford.com" and "From: ceo@toyota.com" both get turned into "From: ceo@..." Some addresses may not have been redacted correctly.
Some messages may have encoding issues.[6] Sometimes (as in the linked case) the non-raw endpoint has the correct characters, sometimes it does not; this is likely related to the originating email client. Removing non-ASCII characters and ^M characters from the 7-bit text should result in valid RFC822 emails.
Some emails longer than 64kb (minus attachments) may be truncated. This truncation affects not just plain text, but also HTML and encoded Base64 content. Deleting the string "\n(Message over 64 KB, truncated)" from the end of the message part may help prevent parser breakage.
All attachments are separated, with attachment bodies replaced with the string "[ Attachment content not displayed ]". Recovering the emails involves finding those MIME parts, looking at the filenames, comparing with the list of filenames listed in the "attachmentInfo" section, matching on similarity, and replacing the contents with the downloaded attachments. In very rare cases where a matching MIME section isn't found, it may be necessary to append those attachments as new MIME attachments to the email while reconstructing.
- Known params: ts, tz, chrome
- Redundancy: Generatable from /messages data.

Topics

- Known params: count, startTopicId, sortOrder (ASC, DESC), direction (1, -1)
- Pagination: Page size defaults to 25, with a limit of 100. sortOrder and direction as for messages.
- Known params: maxResults.
- Pagination: Page size defaults to 30 (messages in topic), with no known limit (maximum tested: 57). No known start param.
- Redundancy: Generatable from /messages data.
"messages" field is an array, each element of which seems to have the same contents as the corresponding /message/<id>/ (non-raw) endpoint; metadata ("totalMsgInTopic", "prevTopicId", "nextTopicId") could be reconstructed. Not known whether a message can fail to be associated with any topic.

Attachments

- Known params: count, start, sort (TITLE, TIME), order (ASC, DESC)
- Pagination: Page size defaults to 20, with no known limit (maximum tested: 93).

Attachment may be of several types: photo, file, ...?

Files

- Known params: sfpath (pass in a pathURI to retrieve the file listings of this subdirectory)
- Pagination: None.
Entries with "type" 0 are files; 1, directories.

Photos

- Known params: count, start, orderBy (MTIME), sortOrder (ASC, DESC), ownedByMe (TRUE, FALSE), lastFetchTime, photoFilter (ALL, PHOTOS_WITH_EXIF "Originals", PHOTOS_WITHOUT_EXIF "Shared")
- Pagination: Page size defaults to 20, with no known limit.
"totalPhotos" field in response gives total in group.
- Known params: count, start, albumType (PHOTOMATIC, NORMAL), orderBy (MTIME, TITLE), sortOrder (ASC, DESC)
- Pagination: Page size defaults to 12, with no known limit.
albumType defaults to NORMAL. PHOTOMATIC albumType requires the "READ" permission for "ATTACHMENTS". "total" field in response gives total number of albums of the selected type in group; however, this seems to have an off-by-one error for the NORMAL type of albums.
- Known params: similar to /photos and /albums endpoints, with additional ordinal sortOrder option
Photomatic albums must be loaded with the albumType parameter set to PHOTOMATIC.

Links

- Known params: linkdir
- Pagination: None.
linkdir takes the folder parameter from a dir. Nested folders should be joined with '/'. You need to keep track of the path to a given folder yourself (eg, linkdir + '/' + folder).

Polls

- Known params: count, start
- Pagination: Page size defaults to 10, with no known limit. There is no "total" field in the response.
Polls return all votes cast, non-anonymised, including identifying metadata for all viewers.

Databases

- Pagination: None.
- Known params: format (CSV, TSV)

Members

- Known params: count, start, sortBy, sortOrder, ts, tz, chrome
- Pagination: Page size defaults to 10, with a limit of 100. No known limit on total results.
May be blocked for normal members (as may all the other members endpoints). Includes moderators and bouncing members, with identifying metadata.
Very often (always?) blocked for normal members.
Very often (always?) blocked for normal members.

Events

Overlaps with Yahoo Calendar API, check yahoo-group-archiver code.

Software for archiving groups

Python

  • yahoo-group-archiver scraped a group using the JSON API and (for private endpoints) the two cookies Yahoo uses to verify a logged-in user. Optionally, it could produce WARCs. ArchiveTeam's preferred tool and fully featured at the time of closure.
    • Yahoo Group Archive Tools (a Perl script) converts yahoo-group-archiver output into clean rfc822 and mbox files, with separated attachments correctly reattached, and many Yahoo truncation/redaction bugs corrected. It also turns list archives into PDF, using email2pdf, which many non-technical list owners prefer.
  • YahooGroups-Archiver is similar, but scraped only messages (not files or any other data). It has been deprecated in favor of the above.
  • yahoo-groups-backup scraped a group's messages and files (but not any other data) using Selenium, storing message info and metadata (both rendered message body and raw email) into a Mongo database. It also provides a script to dump its data to static HTML pages that can be viewed in the browser.

Other

  • PGOffline: Windows, proprietary. 14-day free trial, after which download and export is disabled (but view still works). Included attachments. Stores data in a SQLite database internally.
    • pgo2mbox converts PGOffline pg4 files to mbox.
  • Yahoo Messages Export: Chrome extension. Messages only. Saves as mbox.
  • Yahoo Group Archiver: Perl, defunct.

Software for viewing archives (in mbox format)

Other archiving efforts

External Links

Coverage

References

  1. https://web.archive.org/web/20201126125219/https://help.yahoo.com/kb/groups/SLN31010.html
  2. https://www.theverge.com/2019/12/10/21004883/yahoo-groups-extend-deadline-download-data-date-time[IAWcite.todayMemWeb]
  3. https://twitter.com/YahooCare/status/1204312076379926528[IAWcite.todayMemWeb]
  4. https://help.yahoo.com/kb/groups/SLN35505.html[IAWcite.todayMemWeb]
  5. This id can also be accessed with an appropriate intlCode, but contains the same twelve groups for all languages: the groups in the categories for musical artists "Roots, The" and "Rusted Root", three groups which appear to be Yahoo tests, and one group which appears to be a spam test.
  6. https://yahoo.uservoice.com/forums/209451-us-groups/suggestions/9644478-displaying-raw-messages-is-not-8-bit-clean