Yioop Repo - Seekquarry

« Back Issue #361

2012-02-10 08:19 Reported by administrator
2012-02-19 18:02 Marked fixed

Status: Marked fixed

Priority: Medium

Investigate if Yioop having its own DNS cache independent of cURL speeds up crawling

administrator 2012-02-10 08:19

Right now comcast connection is b/w 22-30 Mbps. Presumably, the average page downloaded is less than the 50k threshold Yioop imposes. This will probably be confirmed or refuted after the following is implemented: http://www.seekquarry.com/mantis/view.php?id=79 Also, it looks like queue_servers are being under utilized. This suggests that the fetchers are not crawling as fast as they could and DNS caching might be an issue. Should investigate if this is true, and if so, implement a separate DNS caching scheme than the one that comes with cURL.

administrator 2012-02-19 18:02

Fixed with 097e53b