Distributed recursive crawls
Jump to navigation
Jump to search
| Distributed recursive crawls | |
| Status | Special case |
| Archiving status | On hiatus |
| Archiving type | DPoS |
| Project source | grab-grab |
| Project tracker | grab |
| IRC channel | #Y (on hackint) |
| Data[how to use] | archiveteam_grab |
This is a project to recursively crawl large websites that have no clear structure that can easily be split into work items the way we usually do on DPoS projects. It is somewhat comparable to ArchiveBot in that crawls are started manually for specific sites of interest.
Useful due to
- rate limiting
- slow hardware
- geo-restrictions
- very large
- blocks ArchiveBot
- still updated but abandoned
- features missing in ArchiveBot
- automatic open dir scanning
Candidates
Some sites that are candidates for this project:
- MoinMoin instances - rate limiting, slow and AB invents non-sequential diff URLs
- https://www.cravingtech.com/ - rate limiting (connect timeouts)
- https://brianwilson.websitetoolbox.com/ - embedded in brianwilson.com - owner died - rate limited to ~40-50 req/s - partially done in AB, but the site was undetectably disabled for some of that time, and there have been new posts since then, and the AB pipeline rebooted killing the job
- https://forum.hardware.fr/ - more than 100 million messages - related to a site that is potentially in danger
- https://www.flamesofwar.com/ https://events.battlefront.co.nz/ https://forcesv4.flamesofwar.com/ https://forces.team-yankee.com/ https://forces.flamesofwar.com/ https://battlerankings.com/ https://www.battlefrontgroup.com/ https://www.team-yankee.com/ https://battlefront-community.com/ - fails in AB - shutting down
- https://www.blogalia.com/ - abandoned blogging platform that is still in use by users so needs a continuous project
- http://bmwfans.info/ - notoriously unstable and slow but contains very valuable information
- https://clay.earth/ - acquired - huge and rate limited
- https://lists.gnu.org/ - very large and needs a continuous project, maybe just as part of Mailing Lists
- https://www.premio.nl/
- https://gobiernu.cw/ - rate limiting
- https://ellieirons.com/ - slow
- https://akb.au.int/ - rate limiting
- https://issues.mediagoblin.org/ - rate limiting - almost finished in AB
- https://ukraineforum.de/ - needs to run super slowly in order to not get refused connections
- https://jimfitzpatrick.com/ gets 409s if run too fast (which is still way slower than the default speed)
- https://www.mordorintelligence.com/ - 429s even with incredibly long delays
- https://www.whosampled.com/
- https://blog.csdn.net/ - too large for AB
- https://transfer.archivete.am/VUfaK/sina.com.cn-subdomains.txt - too large/slow for AB
- https://forum.ixbt.com/ - too large for AB
- https://www.buholegal.com/ and https://buholegal.com/ - too slow for AB
- https://revistas-colaboracion.juridicas.unam.mx/ - too slow for AB
- https://www.culture.ru/ - huge and rate limited
- https://www.abgeordnetenwatch.de/ - rate limited
- https://mchs.gov.ru/ and subdomains - not accessible outside Russia (see job:259v3jnwad35xr46e52dbq2pc)
- https://hub.hku.hk/ - slow
- https://www.npr.org/ - crashed on AB
- https://www.pbs.org/ - crashed on AB
- https://langson.dcs.vn/ - slow
- https://www.visitsaudi.com/ - rate limiting
- https://www.pl.md/ / https://pl.md/ (via #vooterbooter) - rate limiting (refuses connections after a while even with -c 1 -d 8000 12000)
- https://www.geschichtewiki.wien.gv.at/ - rate limiting (403s) and size
- https://www.losreporteros.mx/ - rate limiting
- https://kalaburagi.nic.in/ - rate limiting
- https://www.jewworldorder.org/ - rate limiting
- https://ildb.nadir.org/ - size and rate limiting
- https://www.etilaatroz.com/ and https://kabulnow.com/ - connect timeouts/size
- https://community.unix.com/ - rate limiting (refused connections)
- https://www.emol.com/ - rate limiting (connection resets)
- https://www.artinliverpool.com/ - rate limiting (readline timeouts)
- https://makeyev.ru/ - rate limiting (connect timeouts)
- https://www.suicidegirls.com/ - huge and bans after a while (not sure if rate-limiting or not)
- https://parovoz.com/ - size and rate limiting (readline timeouts)
- https://guyanachronicle.com/ - size and rate limiting (readline timeouts)
- https://armando.info/ - rate-limiting (readline timeouts)
- Namuwiki - namu.wiki and related pages like board.namu.wiki and arca.live (very strict rate limiting, huge, apparently blocked SPN)
- https://russiancouncil.ru/ - rate-limiting (readline timeouts) and size
- https://www.defence.go.ug/ - responding slowly (readline timeouts)
- https://mmnt.net/ - huge, ArchiveBot DB was ~70GB when it was aborted. Not updated for over a decade.
- https://aappb.org/ - rate-limiting (connect timeouts even with 30-60s delays)
- https://www.president.bg/ and https://text.president.bg/ - connect timeouts unless run with 60-120s delay
- https://www.tsp.gob.cu/ - rate-limiting (refused connections with ~10s delay)
- https://multinationales.org/ - rate-limiting (connect timeouts) at long delays; takes long to recover
- https://www.elnacional.com/ - rate-limiting (403s)