If a page fails to download (not error just fails to download), it should be rescheduled and given one more try to download properly

Fixed with revision e8cb403. Minor code tweaks done in queue_server so as to send fetcher information about who is crawl_delayed. The remainder of the changes were done fetcher.php . The revision allows a second download try for pages that come from non crawl-delayed hosts. This second try occurs after a first attempt has been made for all pages in the current fetch batch.