«
Back
Issue #190
- 2012-02-10 08:19 Reported by administrator
- 2012-02-19 18:02 Marked fixed
Status:
Marked fixed
Priority: Medium
Investigate if Yioop having its own DNS cache independent of cURL speeds up crawling
(c) 2026 Seekquarry - Open Source Search Engine Software
Right now comcast connection is b/w 22-30 Mbps. Presumably, the average page downloaded is less than the 50k threshold Yioop imposes. This will probably be confirmed or refuted after the following is implemented: http://www.seekquarry.com/mantis/view.php?id=79 Also, it looks like queue_servers are being under utilized. This suggests that the fetchers are not crawling as fast as they could and DNS caching might be an issue. Should investigate if this is true, and if so, implement a separate DNS caching scheme than the one that comes with cURL.
Fixed with 097e53b