.hu domains seed

From Archiveteam
Jump to navigation Jump to search
.hu domains
Hudomains logo.png
Status Special case
Archiving status In progress...
Archiving type other
IRC channel #archiveteam-bs (on hackint)
Project lead bzc6p
Data[how to use] In the Wayback Machine

Hundreds of .hu domains are registered and expiring every single day. Many of them are just parked and never actually used, for example because they are common words or catchy phrases. Without links to them, even Internet Archive crawls are unable to catch them, leaving us with no publicly available proof that such domains have ever been registered.

Justification

The Council of Internet Service Providers (ISZT), which manages the .hu domain registry, maintains up-to-date lists of recently registered and recently expired domains. Even though the Internet Archive crawler occasionally checks these lists, and therefore could have access to these web pages via links on these lists, this frequency isn't sufficient for all domains be archived, due to the fact that new domain names are held in this list only for 2 weeks.

user:bzc6p has been saving these lists since early 2021, and even though many of these domains have been deleted since then, the database built from this data (including known registration and expiration times of all domains) suggests that tens of thousands of domains registered throughout this period are still available (with or without actual websites behind the domains).

Purpose

The purpose of this project is to seed all available domains into the Wayback Machine, by triggering a Save Page Now action (programmatically, of course...), so that the Wayback Machine knows that the given domain exists, and it can then create subsequent captures on its own.

The project started on 2024-12-17.

Methodology

  • A database has been created from all domain names, their registration and (actual) expiration dates since early 2021.
  • Queries have been made to find out which registered domains have not been expired yet (or, have been re-registered after their expiration) – in other words, that are active in the registry.
  • A script is going through each and every domain on the list, first querying the Wayback Machine if there's a snapshot and if it's recent enough (i.e. created after the last registration date of the domain). For this purpose, the following API is used: https://archive.org/wayback/available?url=example.com No, this one doesn't seem to work correctly, or somehow filters out snapshots that should be considered valid for this purpose. So, instead, the JSON fetched in the background by the Wayback Machine web interface is used to find out the date of last snapshot.
  • If needed, a Save Page Now action is triggered, with saving the error pages as well (e.g. 404), with an appropriately composed POST request to the https://web.archive.org/save endpoint.

Progress and statistics

The following table will be updated as results come in.

Year Known active domains of which
snapshot already existed snapshotting failed (DNS error) first snapshot created
2021
50,277[1]
TBA
TBA
TBA
2022
TBA
TBA
TBA
TBA
2023
TBA
TBA
TBA
TBA
2024
TBA
TBA
TBA
TBA
Total
50,277
TBA
TBA
TBA
  1. as of 2024-12-14


     Hungarian websites     
Red entries indicate websites which don't have an article on this wiki yet. Striked-through entries indicate websites that have already been shut down.
Archives & Digital Libraries mek.oszk.hu  · epa.oszk.hu  · dka.oszk.hu  · webarchivum.oszk.hu  · NAVA  · Fortepan  · fentrol.hu
Blogging Blog.hu  · Blogter  · Freeblog  · Blogger.hu  · reblog.hu  · xfree.hu  · cafeblog.hu
Social networks iWiW  · myVIP  · hotdog.hu  · Baratikor.com  · network.hu  · Mommo  · privi.hu
Webhosting Extra  · tar.hu  · ATW  · Ingyenweb  · Freeweb  · Ultraweb  · x3.hu  · ini.hu  · ininet.hu  · G-Portál  · uCoz  · eOldal  · ewk  · 5mp.eu  · mindenkilapja  · Webnode
Forums, message boards* Index  · SG  · Nők Lapja Cafe  · Hoxa
Video hosting Indavideó  · Videa  · videoplayer.hu  · xfree.hu  · videok.hu
Image hosting Kepfeltoltes.hu  · Fotoalbum.hu  · Indafotó  · Kephost.com  · pics.coldline.hu  · kep.tar.hu  · noob.hu  · PSharing (a.k.a. ivPicture)  · Kephost.hu  · kepfeltoltes.eu  · kephost.net  · kepkuldes.com  · xfree.hu  · GTF Képhost  · fotozz.hu  · Kepkezelo.com  · keptarad.hu  · darkweb.hu  · fos.hu
Questions and Answers gyakorikerdesek.hu  · tudjatok.hu
File sharing data.hu  · toldacuccot.hu  · hellshare.hu  · addat.hu  · fileposta.hu
Document sharing doksi.hu  · Docplayer
Fun Demotiváló  · keptelenseg.hu  · csubakka.hu  · funpic.hu  · nemkutya.com  · legalja.hu  · szanalmas.hu  · trollfesz.cc  · gumicsizma.hu
Trash napiszar.com  · napiszar.hu  · netszar.com  · napiszar.org
Other .hu domains seed  · News+C  · moly.hu  · gyertyalang.hu  · Volán websites  · Szuperinfó