Class: RobotProcessor

void makeCanonicalRobotPath( $path)

For robot paths

foo is treated the same as /foo Path might contain urlencoded characters. These are all decoded except for %2F which corresponds to a / (this is as per http://www.robotstxt.org/norobots-rfc.txt)

Parameters:

array process( string $page, string $url)

Parses the contents of a robots.txt page extracting allowed, disallowed paths, crawl-delay, and sitemaps. We also extract a list of all user agent strings seen.

Tags:

Overrides PageProcessor::process() (Should be implemented to compute a summary based on a)

Parameters: