Distributed recursive crawls

From Archiveteam
Jump to navigation Jump to search
Distributed recursive crawls
Status Special case
Archiving status On hiatus
Archiving type DPoS
Project source grab-grab
Project tracker grab
IRC channel #Y (on hackint)
Data[how to use] archiveteam_grab

This is a project to recursively crawl large websites that have no clear structure that can easily be split into work items the way we usually do on DPoS projects. It is somewhat comparable to ArchiveBot in that crawls are started manually for specific sites of interest.