I've uploaded a Yioop checkout that introduces a new process for distributed archive crawling, where the archive files are placed on the name server and automatically distributed to the fetchers via HTTP requests. Web crawls and Yioop re-crawls work as they always have, and the process should be working for ARC, MediaWiki, and ODP RDF archives. Additionally, the performance of these last three iterators has been improved, and they have all been modified to be able to stop and resume iteration with a new instance relatively efficiently. Finally, the commits add a new tool, arc_extract.php, which extracts pages from archive files and supports stopping and resuming (this is mainly to facilitate testing the various iterators).

This work was for an independent study project. The project blog is at:

http://www.cs.sjsu.edu/faculty/pollett/masters/Semesters/Spring12/shawn/?blog.html#top

The Yioop checkout that implements the changes is hosted at:

http://dl.dropbox.com/u/4411193/DistributedArchiveCrawls.zip

Good job. f422c0 is in the main repository. Here are the remaining outstanding issues with this ticket which might be because I am running PHP 5.3 (minimum level of PHP I am supporting for everything):

(0) Investigate Notice: Undefined index: url in /Applications/XAMPP/xamppfiles/htdocs/git/yioop/controllers/admin_controller.php on line 775

Notice: Undefined index: url in /Applications/XAMPP/xamppfiles/htdocs/git/yioop/models/crawl_model.php on line 835

Warning: Invalid argument supplied for foreach() in /Applications/XAMPP/xamppfiles/htdocs/git/yioop/models/crawl_model.php on line 835

and Warning: syntax error, unexpected ''' in Unknown on line 6 in /Users/cpollett/test.php on line 12 (seems to be only in PHP5.3)

(2) Recrawl stops after 298 pages.

(3) Re-Recrawl (recrawl of a recrawl) indexes but index seems not to be searchable.

(4) Add Class comments on arc_extract.php

The issues below have been resolved so closing this ticket. Good work, shawn.

(0) fixed.

(2) Recrawl number is because there are somethings which have been downloaded by fetchers which haven't been sent to queue_servers when a crawl is stopped. These get incorporated into the queue_servers during a recrawl.

(3) Works now

(4) arc_extract has been merged with arc_tool, comments have been added throughout updated code

Fixed with a1813d6