The cache page validation feature uses a B-Tree data structure (based on Introduction to Algorithms, Cormen) for storage and look-up of ETags and Expires headers. The fetcher is modified to extract ETags and Expires headers from downloaded web pages and send to Queue Server along with robot and index data. The Queue Server is modified to save ETags and Expires headers to a B-Tree in the form of key-value pairs where key is the hash of the URL using Yioop's hash function. The value consists of ETag and Expires headers. When the Queue Server produces a fetch batch, it looks up ETag and Expires headers using the URLs extracted from the priority queue. If the crawl timestamp is greater than the Expires header, the URL is added to schedules. If only an ETag is found, it is latched on to the URL before writing the URL to schedules. On reading a schedule, the fetcher checks for an ETag. If found, it adds "If-None-Match: ETag" to the HTTP header before sending the request for the URL.

Hey Akshat,

I applied the patch and it looked largely good. I have pushed your changes live. Here are some comments I have before closing the issue: (1) Make sure when you have a patch not to end any lines with a space. Otherwise, git will give some warnings. (2) I am worried about the name choice of the class Node that you use in your btree. It is likely to cause a name collision. Could you move this code into the btree.php file and maybe call it BTNode? Probably, the php5.4 way to do this is namespaces, but I am trying to ensure backward compatibility. (3) During your talk yesterday you mentioned hashes were 11 bytes. This got me nervous. They should be 8bytes, it would be 11 bytes if you were storing the base64 variant of the hash. Can you go through your code and make sure you are calling the 8 byte variant of crawlHash. I.e., use $raw=true for the second parameter.

Best, Chris

Implemented by 5c48224d8

Thanks for your work on this akshat. Implemented by 5c48224d8.

Attachments:

cache_page_validation_feature.patch

cache_page_validation.patch

cache_page_validation_revised.patch

cache_page_validation_May23.patch

cache_feature.patch