If the server seems to be giving 503 errors because it is overloaded, Yioop! should detect this and back-off

Fixed with a393ac2. Adds two variables to config.php /**

*  Number of error page 400 or greater seen from a host before crawl-delay
*  host and dump remainder from current schedule
*/

define('DOWNLOAD_ERROR_THRESHOLD', 10);

/** Crawl-delay to set in the event that DOWNLOAD_ERROR_THRESHOLD exceeded*/ define('ERROR_CRAWL_DELAY', 20);