Distributed recursive crawls
|Distributed recursive crawls|
|Archiving status||In progress...|
|IRC channel||(on hackint)|
This is a project to recursively crawl large websites that have no clear structure that can easily be split into work items the way we usually do on DPoS projects. It is somewhat comparable to ArchiveBot in that crawls are started manually for specific sites of interest.