Distributed recursive crawls
Jump to navigation
Jump to search
| Distributed recursive crawls | |
| Status | Special case |
| Archiving status | On hiatus |
| Archiving type | DPoS |
| Project source | grab-grab |
| Project tracker | grab |
| IRC channel | #Y (on hackint) |
| Data[how to use] | archiveteam_grab |
This is a project to recursively crawl large websites that have no clear structure that can easily be split into work items the way we usually do on DPoS projects. It is somewhat comparable to ArchiveBot in that crawls are started manually for specific sites of interest.