Encapsulates the data structures needed to have a queue of to crawl urls
(hash of url, weights) are stored in a PriorityQueue, (hash of url, index in PriorityQueue, offset of url in WebArchive) is stored in a HashTable urls are stored in a WebArchive in an uncompressed format
max_url_archive_offset
The largest offset for the url WebArchive before we rebuild it.
Entries are never deleted from the url WebArchive even if they are deleted from the priority queue. So when we pass this value we make a new WebArchive containing only those urls which are still in the queue.
__construct(string $dir_name, integer $filter_size, integer $num_urls_ram, string $min_or_max)
Makes a WebQueueBundle with the provided parameters
| string | $dir_name | folder name used by this WebQueueBundle |
| integer | $filter_size | size of each partition in the page exists BloomFilterBundle |
| integer | $num_urls_ram | number of entries in ram for the priority queue |
| string | $min_or_max | when the priority queue maintain the heap property with respect to the least or the largest weight |
adjustQueueWeight(string $url, float $delta, boolean $flush = true)
Adjusts the weight of the given url in the priority queue by amount delta
In a page importance crawl. a given web page casts its votes on who to crawl next by splitting its crawl money amongst its child links. This entails a mechanism for adusting weights of elements in the priority queue periodically is necessary. This function is used to solve this problem.
| string | $url | url whose weight in queue we want to adjust |
| float | $delta | change in weight (usually positive). |
| boolean | $flush |
setQueueFlag(string $url, integer $flag)
Sets the flag which provides additional information about the kind of url, for a url already stored in the queue. For instance, might say if it is a robots.txt url, or if the url has already passed the robots.txt test, or if it has a crawl-delay
| string | $url | url whose weight in queue we want to adjust |
| integer | $flag | should be one of self::ROBOT, self::NO_FLAGS, self::SCHEDULABLE or self::SCHEDULABLE + crawl_delay |
removeQueue(string $url, boolean $isHash = false)
Removes a url from the priority queue.
This method would typical be called during a crawl after the given url is scheduled to be crawled. It only deletes the item from the bundles priority queue and hash table -- not from the web archive.
| string | $url | the url or hash of url to delete |
| boolean | $isHash | flag to say whether or not is the hash of a url |
differenceSeenUrls(\seekquarry\yioop\library\array& $url_array, array $field_names = null)
Removes all url objects from $url_array which have been seen
| \seekquarry\yioop\library\array& | $url_array | objects to check if have been seen |
| array | $field_names | an array of components of a url_array element which contain a url to check if seen |
addRobotPaths(string $host, array $paths)
Adds all the paths for a host to the Robots Web Archive.
| string | $host | name that the paths are to be added for. |
| array | $paths | an array with two keys CrawlConstants::ALLOWED_SITES and CrawlConstants::DISALLOWED_SITES. For each key one has an array of paths |
constructHashTable(string $name, integer $num_values) : object
Mainly, a Factory style wrapper around the HashTable's constructor.
However, this function also sets up a rebuild frequency. It is used as part of the process of keeping the to crawl table from having too many entries
| string | $name | filename to store the hash table persistently |
| integer | $num_values | size of HashTable's arraya |
the newly built hash table
lookupHashTable(string $key, integer $return_probe_value = \seekquarry\yioop\library\HashTable::RETURN_VALUE) : mixed
Looks up $key in the to-crawl hash table
| string | $key | the things to look up |
| integer | $return_probe_value | one of self::ALWAYS_RETURN_PROBE, self::RETURN_PROBE_ON_KEY_FOUND, self::RETURN_VALUE, or self::BOTH. Here value means the value associated with the key and probe is either the location in the array where the key was found or the first location in the array where it was determined the key could not be found. |
would be string if the value is being returned, otherwise, false if the key is not found
insertHashTable(string $key, string $value, integer $probe = false) : boolean
Inserts the $key, $value pair into this web queue's to crawl table
| string | $key | intended to be a hash of a url |
| string | $value | intended to be offset into a webarchive for urls together with an index into the priority queue |
| integer | $probe | if the location in the hash table is already known to be $probe then this variable can be used to save a lookup |
whether the insert was a success or not
rebuildHashTable()
Makes a new HashTable without deleted rows
The hash table in Yioop is implemented using open addressing. i.e., We store key value pair in the table itself and if there is a collision we look for the next available slot. Two codes are use to indicate space available in the table. One to indicate empty never used, the other used to indicate empty but previously used and deleted. The reason you need two codes is to ensure that if somebody inserted an item B, it hashes to the same value as A and we move to the next empty slot, to store B, then if we delete A we should still be able to lookup B. The problem is as the table gets reused a lot, it tends to fill up with a lot of deleted entries making lookup times get more and more linear in the hash table size. By rebuilding the table we mitigate against this problem. By choosing the rebuild frequency appropriately, the amortized cost of this operation is only O(1).
emptyRobotData() : string
Delete the Bloom filters used to store robots.txt file info.
Then construct empty new ones. This is called roughly once a day so that robots files will be reloaded and so the policies used won't be too old.
$message with what happened during empty process
notify(integer $index, array $data)
Callback which is called when an item in the priority queue changes position. The position is updated in the hash table.
The priority queue stores (hash of url, weight). The hash table stores (hash of url, web_archive offset to url, index priority queue). This method actually buffers changes in $this->notify_buffer, they are applied when notifyFlush is called or when the buffer reaches self::NOTIFY_BUFFER_SIZE.
| integer | $index | new index in priority queue |
| array | $data | (hash url, weight) |