Distributed recursive crawls
Jump to navigation
Jump to search
| Distributed recursive crawls | |
| Status | Special case |
| Archiving status | In progress... |
| Archiving type | DPoS |
| Project source | recursive-grab |
| Project tracker | recursive |
| IRC channel | #Y (on hackint) |
| Data[how to use] | archiveteam_recursive🔒 |
| Distributed recursive crawls (2022-2023) | |
| Status | Offline |
| Archiving status | Partially saved |
| Archiving type | DPoS |
| Project source | grab-grab |
| Project tracker | grab |
| IRC channel | #Y (on hackint) |
| Data[how to use] | archiveteam_grab🔒 |
This is a project to recursively crawl large websites that have no clear structure that can easily be split into work items the way we usually do on DPoS projects. It is somewhat comparable to ArchiveBot in that crawls are started manually for specific sites of interest.
Useful due to
- rate limiting
- slow hardware
- geo-restrictions
- very large
- blocks ArchiveBot
- still updated but abandoned
- features missing in ArchiveBot
- automatic open dir scanning
Candidates
Some sites that are candidates for this project:
- MoinMoin instances - rate limiting, slow and AB discovers non-sequential diff URLs
- https://www.cravingtech.com/ - rate limiting (connect timeouts)
- https://brianwilson.websitetoolbox.com/ - embedded in brianwilson.com - owner died - rate limited to ~40-50 req/s - partially done in AB, but the site was undetectably disabled for some of that time, and there have been new posts since then, and the AB pipeline rebooted killing the job
- https://forum.hardware.fr/ - more than 100 million messages - related to a site that is potentially in danger
- https://www.flamesofwar.com/ https://events.battlefront.co.nz/ https://forcesv4.flamesofwar.com/ https://forces.team-yankee.com/ https://forces.flamesofwar.com/ https://battlerankings.com/ https://www.battlefrontgroup.com/ https://www.team-yankee.com/ https://battlefront-community.com/ - fails in AB - shutting down
- https://www.blogalia.com/ - abandoned blogging platform that is still in use by users so needs a continuous project
- http://bmwfans.info/ - notoriously unstable and slow but contains very valuable information
- https://clay.earth/ - acquired - huge and rate limited
- https://lists.gnu.org/ - very large and needs a continuous project, maybe just as part of Mailing Lists
- https://www.premio.nl/
- https://gobiernu.cw/ - rate limiting
- https://ellieirons.com/ - slow
- https://akb.au.int/ - rate limiting
- https://issues.mediagoblin.org/ - rate limiting - almost finished in AB
- https://ukraineforum.de/ - needs to run super slowly in order to not get refused connections
- https://jimfitzpatrick.com/ gets 409s if run too fast (which is still way slower than the default speed)
- https://www.mordorintelligence.com/ - 429s even with incredibly long delays
- https://www.whosampled.com/
- https://blog.csdn.net/ - too large for AB
- https://transfer.archivete.am/VUfaK/sina.com.cn-subdomains.txt - too large/slow for AB
- https://forum.ixbt.com/ - too large for AB
- https://www.buholegal.com/ and https://buholegal.com/ - too slow for AB
- https://revistas-colaboracion.juridicas.unam.mx/ - too slow for AB
- https://www.culture.ru/ - huge and rate limited
- https://www.abgeordnetenwatch.de/ - rate limited
- https://mchs.gov.ru/ and subdomains - not accessible outside Russia (see job:259v3jnwad35xr46e52dbq2pc)
- https://hub.hku.hk/ - slow
- https://www.npr.org/ - crashed on AB
- https://www.pbs.org/ - crashed on AB
- https://langson.dcs.vn/ - slow
- https://www.visitsaudi.com/ - rate limiting
- https://www.pl.md/ / https://pl.md/ (via #vooterbooter) - rate limiting (refuses connections after a while even with -c 1 -d 8000 12000)
- https://www.geschichtewiki.wien.gv.at/ - rate limiting (403s) and size
- https://www.losreporteros.mx/ - rate limiting
- https://kalaburagi.nic.in/ - rate limiting
- https://www.jewworldorder.org/ - rate limiting
- https://ildb.nadir.org/ - size and rate limiting
- https://www.etilaatroz.com/ and https://kabulnow.com/ - connect timeouts/size
- https://community.unix.com/ - rate limiting (refused connections)
- https://www.emol.com/ - rate limiting (connection resets)
- https://www.artinliverpool.com/ - rate limiting (readline timeouts)
- https://makeyev.ru/ - rate limiting (connect timeouts)
- https://www.suicidegirls.com/ - huge and bans after a while (not sure if rate-limiting or not)
- https://parovoz.com/ - size and rate limiting (readline timeouts)
- https://guyanachronicle.com/ - size and rate limiting (readline timeouts)
- https://armando.info/ - rate-limiting (readline timeouts)
- Namuwiki - namu.wiki and related pages like board.namu.wiki and arca.live (very strict rate limiting, huge, apparently blocked SPN)
- https://russiancouncil.ru/ - rate-limiting (readline timeouts) and size
- https://www.defence.go.ug/ - responding slowly (readline timeouts)
- https://mmnt.net/ - huge, ArchiveBot DB was ~70GB when it was aborted. Not updated for over a decade.
- https://aappb.org/ - rate-limiting (connect timeouts even with 30-60s delays)
- https://www.president.bg/ and https://text.president.bg/ - connect timeouts unless run with 60-120s delay
- https://www.tsp.gob.cu/ - rate-limiting (refused connections with ~10s delay)
- https://multinationales.org/ - rate-limiting (connect timeouts) at long delays; takes long to recover
- https://www.elnacional.com/ - rate-limiting (403s)
- https://ukraina.ru/ - rate-limiting (responding slowly)
- https://pnn.ps/ - huge (requested by a webuser)
- https://jstor.org/ - huge (requested by a webuser; RIP Aaron Swartz)
- https://www.sharonlbutler.com/ / https://sharonlbutler.com/ - rate limiting
- https://diatlovonews.by/ - Slow to respond
- https://www.juventudrebelde.cu/ - frequently failing (HTTP 5XX) and recovering
- https://wiki.radxa.com/ - readline timeouts; no longer maintained
- https://www.fidesz.hu/ - blocked in most places (Hungary and Romania have been seen working)
- https://dreamparkfestival.se/ - rate-limiting
- https://www.atk-ks.org/ - rate-limiting
- https://hypertextmag.com/ - bans for (yet) uknown reasons (403)
- https://die-partei.sh/ - rate-limiting (refused connections)
- https://www.thestore-dassault-aviation.com/ - 403s even with long delays; Magento
- https://hrhcrimea.org.ua/ - 429s even with long delays
- https://www.rottentomatoes.com/ - Too big for archivebot
- https://stefanunkovic.com/ - rate-limiting (429s, then connect/readline timeouts)
- https://sanaduri.se/ - rate-limiting (429s, then connect/readline timeouts)
- https://pen.org.ua/ - rate-limiting (429s even with long delays)
- https://www.burkina.campusfrance.org/ - aggressive rate-limiting
- https://ppa.hku.hk/ - rate limiting (quick to respond with connect timeouts"
- https://espaciopublico.ong/ - rate limiting (403s)
- https://moviemeter.nl/ - IMDB limited reviews to the latest 10 without login
- https://www.moviemeter.com/ - alt for moviemeter.nl
- https://www.rottentomatoes.com/ - IMDB limited reviews to the latest 10 without login
- https://www.themoviedb.org/ - another movie database
- https://www.militar.org.ua/ - rate-limiting (403s and readline timeouts)
- https://www.alghadalsoury.com/ (Syrian politics) - rate-limiting (still 429s with a 10s delay)
- https://www.last.fm/ - large and quick to respond with 406s and 600s
- https://tek.sapo.pt/ - large and rate-limited (429s)
- https://www.workers.org/ - rate-limiting (429s); deploys Anubis
- https://joannenova.com.au/ - rate limiting (503s still at long delays)
- https://transfer.archivete.am/inline/15VaYX/er.ru_regional-subdomains.txt - aggressive bans
- https://en-dangcongsan.nhandan.vn/ - slow to respond, many readline timeouts
- https://vientosur.info/ - eventual bans (refused connections)
- https://core.ac.uk/ - huge
- https://www.insanelymac.com/ - eventual ban (403s; did not recover after a month in ArchiveBot)