When using a mix archive bundle iterator, either fresh or restored from a save point, many kinds of queries do not respect the requested number of pages passed to the nextPages() method. The iterators don't seem to correctly set the results_per_block member variable, which causes them to return more results than requested. In turn, the logic in phrase_model.php that implements pagination through fixed windows of results breaks down, yielding both the wrong number of results for each call to nextPages(), and skipped results. I'm not sure whether this behavior applies to normal web searches.

I've attached a patch (fix_index_bundle_iterators.patch) that attempts an initial fix, but probably still has issues. The remaining issue I'm aware of (and there are probably others) is with the union_iterator, which sets its own results_per_block as the *sum* of its constituent iterators' results_per_block values. That behavior doesn't make sense to me if we're thinking about results_per_block as a limit—perhaps it's left over from a different use for that variable? It means that if a query is the union of two words, and we're requesting two pages, you can end up getting back either 2 or 3 documents with each nextPages() request (3 because one word iterator is limited to 2, and another to 1, since it's the intersection of the word and the index). I think the union iterator's results_per_block should instead be the minimum of its constituents', but I'm not sure.

Steps to reproduce: From the attached files, first index the flat-text archive called farm_ads_neg (this is from another project). Then apply debug_mix_itr.patch. That should create a new bin script that automates creating a crawl mix for a particular index, deleting it, and requesting some pages. Set the CRAWLTIME constant in that script to the crawl timestamp for the index you just created. Now run the test_mix_itr.sh script, which will use the debug_mix_itr.php script to create a crawl mix for the new index (with an empty query), and run through three iterations of calling nextPages(5). Between each iteration the mix iterator will be saved back to disk and reconstituted, simulating separate requests. You should see that the returned number of pages is NOT 5 (before applying the attempted fix), and that there are big gaps in the stream of fetched documents. After applying the attempted fix, both of these issues SHOULD be remedied.

Additional information: You can invoke test_mix_itr.sh with arguments to change the number of nextPages() calls, the number of pages requested by each call, and the query used to construct the mix iterator:

usage: ./test_mix_itr.sh ITERATIONS NUM_PAGES QUERY

The query should be quoted. Check the script for the defaults. In addition to specifying NUM_PAGES as a simple integer, you can specify it as NxM, which will result in M requests of N pages for each iteration; this allows you to see the effect of successive calls to nextPages() without a save and restore. Compare

$ ./test_mix_itr.sh 3 5 "disease"

and

$ ./test_mix_itr.sh 1 5x3 "disease"

The first goes through 3 iterations of saving and restoring the iterator, each time requesting 5 pages. The second goes through only one save/restore iteration, but requests the same number of pages in 3 back-to-back calls to nextPages(5).

Hey Shawn,

I removed the duplicates of this issue you had. The issue seems to maximum column size supported by my mysql instance (I guess mantis stores things in giant blobs). I uploaded everything except the data file to the issue though. I have already applied index_bundle_iterators.patch to my local copy. Thanks for the patch. It should go live in a day or so.

Best, Chris

Attachments:

debug_mix_itr.patch

fix_index_bundle_iterators.patch

arc_description.ini