Mailing Lists

Mailing lists
Status	Unknown
Archiving status	Not saved yet
Archiving type	Unknown
Project source	mailing-lists-grab
IRC channel	#flaming-lists (on hackint)

Mailing lists are a type of discussion group that is a precursor to forums (like Discourse) and is declining in popularity as email declines in use, being replaced by forums, and then social media. Existing mailing lists have a lot of history that would be useful to save.

Requirements

• Some sites only support TLS 1.0, or maybe even SSL 3 or earlier
• Some software (notably Sympa) or sites need POST plus a cookie to get otherwise public archives

Sources

There are some things that could be useful to find mailing list services/software:

• Wikidata
  • instance of (P31) electronic mailing list (Q304863)
  • might be a few more with other links to that item via refs etc
• Search engine enumeration
• https://en.wikipedia.org/wiki/List_of_mailing_list_software
• https://en.wikipedia.org/wiki/Category:Mailing_list_software
• https://alternativeto.net/tag/mailing-list/
• https://builtwith.com/mailman
• https://github.com/juzeon/fast-mail-bomber/blob/master/data/providers.json

Services

There are a number of notable services that include mailing lists, some have public archives:

• Usenet
• Gmane (now NNTP-only)
• The Mail Archive
  • raw emails not publicly accessible
  • some lists archived elsewhere too, but sole archive source for some lists (eg Hyphanet devl)
• Mailing list ARChives (MARC)
  • raw or partially raw mails public
  • some lists archived elsewhere too, but sole archive source for some lists (eg some KDE lists)
• thr3ads.net
  • archives lists on Google Groups, Mailman 2 instances and other services
• Nabble
  • supports custom domains, examples: http://forum.world.st/
• Narkive
  • main site is Usenet
  • lists on subdomains, for eg https://debian-knoppix.linuxtag.narkive.com/
• FreeLists
• Google Groups
  • raw messages are not available on most lists, anti-eg [1] [2] [3]
  • some lists have Message-Id based URLs in the mail's footer, eg bitcoindev
• groups.io
  • Has support for custom domains, for e.g. https://lists.lfenergy.org/g/main and https://lists.hubmapconsortium.org/
• Simplelists
  • Has support for custom domains
    • https://lists.security.metacpan.org/
    • https://srfi-email.schemers.org/ - protected account - index of lists here instead https://srfi.schemers.org/srfi-list-subscribe.html
    • https://list.valvesoftware.com/ - requires subscription to access list archives
• SourceForge
  • Derived from Mailman
  • example https://lists.sourceforge.net/lists/listinfo/valgrind-announce
• Framasoft
• LWN
• notmuch
• PostgreSQL
• LibreOffice
• ActiveState
• SecLists

Message-Id lookup

There are some services that offer Message-Id lookup. It is important that we archive both the Message-Id based URLs and the URLs they redirect to. We should also extract the Message-Ids from every message to develop a mapping between Message-Ids and URLs, and later create a service for searching that.

http://mid.gmane.org/alpine.BSF.2.00.1004291655110.61392@x.fncre.vasb
nntp://news.gmane.io/65491cda.c80a0220.5fa79.7688SMTPIN_ADDED_BROKEN@mx.google.com
http://news.gmane.org/find-root.php?message_id=alpine.BSF.2.00.1004291655110.61392@x.fncre.vasb
https://marc.info/?i=Prayer.1.0.18.0811191202370.4107@sabinus.cs.umn.edu
http://mid.mail-archive.com/9dc4201d0807272359r1f0ad9d6ta44f31f439de58b7@mail.gmail.com
https://www.mail-archive.com/search?l=mid&q=9dc4201d0807272359r1f0ad9d6ta44f31f439de58b7@mail.gmail.com
https://www.freebsd.org/cgi/mid.cgi?db=mid&id=199802242058.MAA24843@monk.via.net
https://docs.FreeBSD.org/cgi/mid.cgi?db=mid&id=199802242058.MAA24843@monk.via.net
https://mail-archive.freebsd.org/cgi/mid.cgi?db=mid&id=199802242058.MAA24843@monk.via.net
https://lists.debian.org/msgid-search/20050606213954.GC884@finlandia.infodrom.north.de
https://lists.debian.org/msgid-search/?m=YvogMm/B0cM/AUMu@einval.com
https://lists.debconf.org/cgi-lurker/keyword.cgi?doc-url=/lurker&format=en.html&query=id:87vey4ogcj.fsf@vorlon.ganneff.de
https://www.w3.org/mid/alpine.BSF.2.00.1004291655110.61392@x.fncre.vasb
https://www.postgresql.org/message-id/20060419051850.96025.qmail@web31802.mail.mud.yahoo.com
https://public-inbox.org/git/Pine.LNX.4.58.0504150753440.7211@ppc970.osdl.org
https://article.olduse.net/40478@ucbvax.BERKELEY.EDU
http://al.howardknight.net/msgid.cgi?STYPE=msgid&A=0&MSGI=<usenet-hier-faq-1516003264%246620%40hope.eyrie.org>
https://lwn.net/ml/linux-kernel/20181020134908.GA32218@kroah.com/
https://lwn.net/ml/all/20181020134908.GA32218@kroah.com/
https://lore.kernel.org/all/20181020134908.GA32218@kroah.com/
https://yhbt.net/lore/all/20181020134908.GA32218@kroah.com/
https://public-inbox.org/git/1486427537.16949.42.camel@bonedaddy.net/
https://www.43-1.org/app/nntp/mid/20050606213954.GC884@finlandia.infodrom.north.de
https://forum.dlang.org/post/d19o8m$1jlj$1@digitaldaemon.com
https://www.nntp.perl.org/group/perl.beginners/;msgid=01041316021102.03716%40localhost.localdomain
https://nmbug.notmuchmail.org/nmweb/show/156832191038.4937.14365134928759130092%40fireG0
https://www.ctan.org/ctan-ann/id/mailman.3878.1440779487.4386.ctan-ann@dante.de
lists.sr.ht graphql API
https://archives.spi-inc.org/message-id/14d2a7e9-3269-d881-7f15-506bbf78299f%40spi-inc.org
https://groups.google.com/d/msgid/bitcoindev/aDWfDI03I-Rakopb%40petertodd.org
https://patch.msgid.link/9ef87524-d15c-4b2c-9f86-00417dad9c48@kernel.dk

Software

There are lots of software for mailing lists that have instances with public archives:

• Mailman 2 (pipermail archives)
  • Requires a very special strategy for Mailman 2 instances
  • pabs is monitoring ArchiveBot for Mailman 2 URL patterns
  • Message-Id in three places:

    • <HEAD> <LINK REL="made" HREF="mailto:...?...In-Reply-To=...">

    • <BODY> <A HREF="mailto:...?...In-Reply-To=...">

      (may get corrupted by Cloudflare/etc email address protection
    • in the compressed mbox archives
• Mailman 3 (usually HyperKitty archives)
  • Message-Id in mailto:...?...In-Reply-To links (may be broken by email protection), might be an API somewhere to avoid that
  • Uses these archivebot ignores to skip POST-based URLs and login pages:

    • ^https?://{primary_netloc}/accounts/login/

    • ^https?://{primary_netloc}/accounts/signup/

    • ^https?://{primary_netloc}/hyperkitty/list/[^/]+/message/[^/]+/vote$

    • ^https?://{primary_netloc}/hyperkitty/list/[^/]+/thread/[^/]+/favorite$

  • https://lists.darmstadt.ccc.de/hyperkitty/ https://lists.fedoraproject.org/ https://lists.hacknology.de/ https://mailman.subsurface-divelog.org https://lists.easaonline.org/ https://lists.ilias.de/ https://lists.osmocom.org/ https://lists.das-labor.org/
• Sympa
  • May need POST plus a cookie to get otherwise public archives
  • http://lists.ee.ethz.ch/wws/ http://lists.sameswifi.fr/wws/ https://framagroupes.org/abc/en/ https://framalistes.org/sympa/ https://groupes.renater.fr/sympa/ https://liste.normandie-univ.fr/sympa/ https://listes.april.org/wws/ https://listes.aquilenet.fr/ https://listes.asso.univ-rennes.fr/wws/ https://listes.centrale-marseille.fr/ https://listes.creachlabs.fr/wws https://listes.doctorat-bretagne.fr/wws/ https://listes.ens-lyon.fr/sympa/ https://listes.ens-paris-saclay.fr/wws/ https://listes.ensc-rennes.fr/wws https://listes.entrouvert.com/ https://listes.etudiant.univ-rennes.fr/wws/ https://listes.existrans.org/ https://listes.hadoly.fr/ https://listes.iiens.net/sympa/ https://listes.infini.fr/ https://listes.renater.fr/wws/ https://listes.sciencespo-rennes.fr/wws/ https://listes.services.cnrs.fr/wws/ https://listes.u-bretagneloire.fr/wws/ https://listes.univ-rennes.fr/wws/ https://listes.univ-reunion.fr/wws/ https://lists.actux.eu.org/sympa/ https://lists.cam.ac.uk/sympa/ https://lists.digi-newb.eu/wws/ https://lists.ens.psl.eu/wws/ https://lists.fdn.fr/ https://lists.fit.edu/sympa/ https://lists.grnet.gr/wws/ https://lists.laglab.org/wws/ https://lists.linuxfromscratch.org/sympa/ https://lists.nnedv.org/lists/ https://lists.piratenpartei.de/ https://lists.piratenpartei-nrw.de/ https://lists.puscii.nl/ https://lists.riseup.net/ https://lists.squat.net/wws/ https://lists.uniroma2.it/ https://lists.vis.ethz.ch/sympa/ https://lists.vmp.ethz.ch/sympa/ https://lists.vvn-bda.de/ https://lists36.net/wws/ https://mail.linuxtag.org/wws/ https://mailing.sogo.nu/sympa/ https://mlistes.resel.fr/wws/ https://newsletter.ende-gelaende.org/ https://newsletters.univ-rennes.fr/wws/ https://post.bund.net/ https://psilists.ethz.ch/sympa/ https://sympa.inria.fr/sympa/ https://sympa.laas.fr/wws/ https://sympa.lix.polytechnique.fr/ https://sympa.obspm.fr/wws/ https://sympa.uvsq.fr/sympa/ https://www.iut-tlse3.fr/wws/ https://www.legi.grenoble-inp.fr/servlist/wws/ https://www.listes.upmc.fr/wws/ https://www.listserv.dfn.de/sympa/ https://lists.hefn.org/ https://listas.alainet.org/listas https://lists.aspirationtech.org/ https://lists.compasspoint.org/
• Apache Pony Mail
  • https://lists.apache.org/ (huge)
• public-inbox
  • https://inbox.sourceware.org/ (huge) https://inbox.vuxu.org/ https://list.orgmode.org/ https://lore.barebox.org/ https://lore.kernel.org/ (huge) https://lore.pengutronix.de/ https://lore.ptxdist.org/ https://yhbt.net/kgio-public/ https://yhbt.net/raindrops-public/ https://yhbt.net/unicorn-public/ https://yhetil.org/
• LISTSERV
  • https://listserv.ihs.gov/ (saving in AB)
• lyris
• mlmmj
  • seems to use external archivers
  • lists.lighttpd.net
• lurker
  • https://lists.dyne.org/ https://www.guilde.asso.fr/lurker/
• mhonarc
  • Message-Id in encoded HTML comments

    • <!--X-Message-Id: 19190830-af45-4646-8344-4f788db48a0f@disroot.org -->

  • https://identicalsoftware.com/maillist/xwpe/ https://archives.seul.org/ lists.gnu.org (saved by AB)
  • lists.debian.org (not saved yet)
  • lists.launchpad.net (shutting down, contacted Canonical)
• hypermail
  • Message-Id in a HTML comment, the @ might be escaped to _at_ or something else

    • <!-- id="9C19DACF-698F-4AF8-8873-A2DB2C6CB875_at_yahoo.com" -->

    • <!-- id="199911012305.PAA05714@hawk.prod.itd.earthlink.net" -->

  • http://lalists.stanford.edu/ http://lists.boost.org/boost-interest https://list.skarnet.org/lists/ https://lists.boost.org/Archives/boost/ https://lists.boost.org/boost-announce/ https://lists.boost.org/boost-bugs/ https://lists.boost.org/boost-build/ https://lists.boost.org/boost-cmake/ https://lists.boost.org/boost-commit https://lists.boost.org/boost-docs/ https://lists.boost.org/boost-gil/ https://lists.boost.org/boost-maint/ https://lists.suckless.org/ https://lists.w3.org/Archives/Public/ https://lkml.iu.edu/hypermail/linux/kernel/ https://potter-yachters.org/archive/ https://www.circlemud.org/maillist/ https://www.daml.org/listarchive/ https://www.informatik.uni-kiel.de/~curry/listarchive/ https://www.lehigh.edu/~wsm0/vse-l/vse-l_archives.shtml#hypermail https://www.w1.fi/hostap/
• Majordomo2
  • https://mail.kspei.com/cgi-bin/mj_wwwusr?domain=lists.mj2.org&func=lists-full&extra=mj2-dev https://lists.openbsd.org/cgi-bin/mj_wwwusr
• ezmlm/ezmlm-idx/ezmlm-web (associated with qmail/djbdns)
  • https://cr.yp.to/lists.html
• SourceHut
  • mbox downloads behind a POST request
  • https://lists.sr.ht/ https://lists.2mb.codes/ https://todo.argonaut-constellation.org/ https://lists.ditigal.xyz/~ruther https://lists.open-music-kontrollers.ch/~hp https://lists.xenrox.net/~xenrox https://lists.alpinelinux.org/
• CREN ListProc (more info)
• Unknown
  • http://www.techwr-l.com/archives/
  • https://lists.racket-lang.org/

Tools

These are some pages of and individual tools that are useful for archiving mailing lists:

• Mailman
• forum-dl

Projects

These are some former archiving projects by ArchiveTeam that involved mailing lists:

• Freeml
• DiscussionApps
• Yahoo! Groups
• Google Groups]
• Gna!
• BerliOS
• Google Groups Files

Pages

There are a number of wiki pages that mention mailing lists in some way.

• OWASP
• Gopher
• Spanish Revolution
• Hackerspaces
• WikiApiary
• OpenSolaris
• Internet infrastructure
• Chaos Computer Club
• AdelaideUniversityMerger

Articles

There are some articles about hosting mailing lists that may be useful.

• Creating an email archive with public-inbox