«
Back
Issue #477
- 2019-10-16 01:54 Reported by xianghong
- 2020-05-16 20:13 Marked fixed
Status:
Marked fixed
Priority: Medium
Chinese word segmentation class
(c) 2026 Seekquarry - Open Source Search Engine Software
Use dictionary and static method to segment Chinese sentence into words. patch and dataset are attached.
Steps to reproduce: Use SegmentTool.php to build dictionary and to segment files of texts: php SegmentTool.php train file_a file_b ... php SegmentTool.php segment file_a file_b ... Use StochasticChineseWordSegmenter class to train or segment texts, sentence..
Patch No3 is up.
Hey Forrest,
Thanks for the update! I pushed your Patch 3. You'll need to remind me tomorrow how to use it with token tool and then we need to hook it into to the default Chinese language processing.
Best, Chris
Hey Forrest,
You're patch 3 doesn't apply. Can you get me a fresh patch that works?
Best, Chris
Sorry for the late patch 4. But it seems you've already pushed the patch 3 on 11/11, which is good. The patch 4 notes: 1. Added more features in Chinese Segmenter 2. Moved part of the code from StochasticTermSegmenter class to zh_CN\Tokenizer class for readability 3. Changed the default segmenter for Chinese from reverseMaximalMatch to StochasticTermSegmenter
Best, Forrest
Patch 5 is up Hmm... I modified a word in .patch file and hope it wont cause any trouble. And, I have no idea why this patch is so huge.. Say, it applied half of my previous patch but not the other half? i'm not sure... Maybe I need get a new copy of Yioop next time... I didn't apply the patch for test but I think it will be fine. Here's the updates for this patch: 1. Fixed RMM spacing problem; 2. Added some updates on Chinese Segmentation dataset training; 3. Implemented a memory saving mode on Chinese Segmenetation;
Patch 6 is up Basically I redo the things shown in patch 5 except the RMM part (because it does not work now, i will take a look tomorrow)... There are a lot of new codes, it took me some time to migrate my code...
Something to notice: 1. remove $max_num_term_weights to see the real result, it should consume no more than 70MB in training. If it crashed... then??? (By the way, if you keep $max_num_term_weights, it will only consume 11MB in training, and a lot of words will be missing) 2. you can adjust the $cache_pct = 0.06 in the __construct, this number should be set between 0 - 1, higher value will consume more memory but the segmentation will be faster. Some example data of memory vs speed are provided in the description of this value. 3. Please re-train the dataset
P.S. I didn't even notice this patch was really big.. And, currently, I am testing it with the Chinese Tree Bank dataset, which I downloaded from internet. I am contacting lingustic department to accept me as a member so I can access the latest dataset. Linguistic Data Consortium is selling this dataset for 300 bucks. (But members are free)
Please note the edit to your last post. The reason for $max_num_term_weights was for crawling not training. Fetchers kept running out of memory when using the segmenter. Have you done crawl tests on the IBM site with the new segmenter? I have applied your patch now and will push it shortly.
Best, Chris
I have attached the log... so would you please take a look? I don't know how the crash looks like, compare to the current one. so I cannot tell.
Patch 6.5 is up It contains everthing in patch 6 and make the segmenter to be static so it won't get destoryed every time it have one job done.
Not sure if I should create a new Issue or keep use this one.
ahh.. I forgot to upload it on tuesday...
https://www.dropbox.com/sh/7p3u2gq7sz9f38g/AAAWqUgXZS6obbLoiaUx8vcXa?dl=0
trained data files pull them to locale\zh_CN\resources to make everything work.
best, Forrest
Hey Forrest,
Thans for your patch! Can you say what each of the files at the dropbox link is for? Can you briefly tell me what your patch does? I.e., if you do a crawl will it be obvious code in your patch is running? How do I see your coding running?
Best, Chris
Hey Forrest,
When I try to just test (via Page Options - Test) the main Chinese Wikipedia page after installing your patch, it doesn't seem to segment things too much. Also, there don't seem to be question answer terms extracted.
Best, Chris
QA first patch is up. Tried some unit test, but not sure if it will work in practise.
Best, Forrest
Hey Forrest,
Thanks for the patch. I have applied it.
Best, Chris
This issue has morphed to much/accumulated too many incremental changes. Several patches have already been applied toward Chinese word segmentation. Let's start a new issue for further changes.
Attachments:
my2.patch
my3.patch
my4.patch
my5.patch
my6.patch
_binary Last url processed was https://1.cms.s81c.com/robots.txt. Current memory usage:50807088 [668 Sat, 01 Feb 2020 13:30:14 -0800] ..Still building inverted index. Have processed 0 of 73 documents. [667 Sat, 01 Feb 2020 13:30:14 -0800] Start building mini inverted index ... Current Memory:50792896 [666 Sat, 01 Feb 2020 13:30:14 -0800] ...30992 bytes of schedule data [665 Sat, 01 Feb 2020 13:30:14 -0800] ...3164 bytes of cache page validation data [664 Sat, 01 Feb 2020 13:30:14 -0800] ...416 bytes of robot data [663 Sat, 01 Feb 2020 13:30:14 -0800] Updating machine: http://localhost:8080/ [662 Sat, 01 Feb 2020 13:30:14 -0800] Start Update Server [661 Sat, 01 Feb 2020 13:30:14 -0800] Done Update Found Sites Array Time 0.047153949737549 [660 Sat, 01 Feb 2020 13:30:14 -0800] 78. https://www.ibm.com/downloads/cas/sitemap/sitemap17.xml [659 Sat, 01 Feb 2020 13:30:14 -0800] 77. https://www.ibm.com/downloads/cas/sitemap/sitemap16.xml [658 Sat, 01 Feb 2020 13:30:14 -0800] 76. https://www.ibm.com/downloads/cas/sitemap/sitemap15.xml [657 Sat, 01 Feb 2020 13:30:14 -0800] 75. https://www.ibm.com/downloads/cas/sitemap/sitemap14.xml [656 Sat, 01 Feb 2020 13:30:14 -0800] 74. https://www.ibm.com/downloads/cas/sitemap/sitemap13.xml [655 Sat, 01 Feb 2020 13:30:14 -0800] 73. https://www.ibm.com/downloads/cas/sitemap/sitemap12.xml [654 Sat, 01 Feb 2020 13:30:14 -0800] 72. https://www.ibm.com/downloads/cas/sitemap/sitemap11.xml [653 Sat, 01 Feb 2020 13:30:14 -0800] 71. https://www.ibm.com/downloads/cas/sitemap/sitemap10.xml [652 Sat, 01 Feb 2020 13:30:14 -0800] 70. https://www.ibm.com/downloads/cas/sitemap/sitemap9.xml [651 Sat, 01 Feb 2020 13:30:14 -0800] 69. https://www.ibm.com/downloads/cas/sitemap/sitemap8.xml [650 Sat, 01 Feb 2020 13:30:14 -0800] 68. https://www.ibm.com/downloads/cas/sitemap/sitemap7.xml [649 Sat, 01 Feb 2020 13:30:14 -0800] 67. https://www.ibm.com/downloads/cas/sitemap/sitemap6.xml [648 Sat, 01 Feb 2020 13:30:14 -0800] 66. https://www.ibm.com/downloads/cas/sitemap/sitemap5.xml [647 Sat, 01 Feb 2020 13:30:14 -0800] 65. https://www.ibm.com/downloads/cas/sitemap/sitemap4.xml [646 Sat, 01 Feb 2020 13:30:14 -0800] 64. https://www.ibm.com/downloads/cas/sitemap/sitemap3.xml [645 Sat, 01 Feb 2020 13:30:14 -0800] 63. https://www.ibm.com/downloads/cas/sitemap/sitemap2.xml [644 Sat, 01 Feb 2020 13:30:14 -0800] 62. https://www.ibm.com/downloads/cas/sitemap/sitemap1.xml [643 Sat, 01 Feb 2020 13:30:14 -0800] 61. https://1.www.s81c.com/robots.txt [642 Sat, 01 Feb 2020 13:30:14 -0800] 60. https://www.ibm.com/accessibility/us/en/ [641 Sat, 01 Feb 2020 13:30:14 -0800] 59. https://www.ibm.com/legal/us/en/ [640 Sat, 01 Feb 2020 13:30:14 -0800] 58. https://www.ibm.com/privacy/us/en/ [639 Sat, 01 Feb 2020 13:30:14 -0800] 57. https://www.ibm.com/contact/us/en/ [638 Sat, 01 Feb 2020 13:30:14 -0800] 56. https://www.ibm.com/marketplace/?lnk=404 [637 Sat, 01 Feb 2020 13:30:14 -0800] 55. https://www.ibm.com/connect/ibm [636 Sat, 01 Feb 2020 13:30:14 -0800] 54. https://www.ibm.com/sitemap/us/en/ [635 Sat, 01 Feb 2020 13:30:14 -0800] 53. https://www.ibm.com/us/en/ [634 Sat, 01 Feb 2020 13:30:14 -0800] 52. https://www.ibm.com/internal/cms/sitemap.xml?page=3 [633 Sat, 01 Feb 2020 13:30:14 -0800] 51. https://www.ibm.com/internal/cms/sitemap.xml?page=2 [632 Sat, 01 Feb 2020 13:30:14 -0800] 50. https://www.ibm.com/internal/cms/sitemap.xml?page=1 [631 Sat, 01 Feb 2020 13:30:14 -0800] 49. https://www.ibm.com/aw-en/ [630 Sat, 01 Feb 2020 13:30:14 -0800] 48. https://www.ibm.com/au-en/ [629 Sat, 01 Feb 2020 13:30:14 -0800] 47. https://www.ibm.com/at-de/ [628 Sat, 01 Feb 2020 13:30:14 -0800] 46. https://www.ibm.com/ar-es/ [627 Sat, 01 Feb 2020 13:30:14 -0800] 45. https://www.ibm.com/ao-pt/ [626 Sat, 01 Feb 2020 13:30:14 -0800] 44. https://www.ibm.com/ai-en/ [625 Sat, 01 Feb 2020 13:30:14 -0800] 43. https://www.ibm.com/ag-en/ [624 Sat, 01 Feb 2020 13:30:14 -0800] 42. https://www.ibm.com/af-en/ [623 Sat, 01 Feb 2020 13:30:14 -0800] 41. https://www.ibm.com/ae-en/ [622 Sat, 01 Feb 2020 13:30:14 -0800] 40. https://www.ibm.com/cn-zh/products/category/business/collabo ration [621 Sat, 01 Feb 2020 13:30:14 -0800] 39. https://www.ibm.com/cn-zh/products/category/technology/cogni tive-computing-and-AI [620 Sat, 01 Feb 2020 13:30:14 -0800] 38. https://www.ibm.com/cn-zh/products/category/business/busines s-operations [619 Sat, 01 Feb 2020 13:30:14 -0800] 37. https://1.dam.s81c.com/robots.txt [618 Sat, 01 Feb 2020 13:30:14 -0800] 36. https://www.ibm.com/cn-zh/products/category/business/custome r-service-and-CRM [617 Sat, 01 Feb 2020 13:30:14 -0800] 35. https://www.ibm.com/cn-zh/products/category/technology/IT-in frastructure [616 Sat, 01 Feb 2020 13:30:14 -0800] 34. https://www.ibm.com/cn-zh/products/category/business/content -management [615 Sat, 01 Feb 2020 13:30:14 -0800] 33. https://www.ibm.com/cn-zh/products [614 Sat, 01 Feb 2020 13:30:14 -0800] 32. https://www.ibm.com/cn-zh/products/category/business/human-r esources [613 Sat, 01 Feb 2020 13:30:14 -0800] 31. https://www.ibm.com/employment/cn-zh/ [612 Sat, 01 Feb 2020 13:30:14 -0800] 30. https://www.ibm.com/cn-zh/products/watson-studio-desktop?lnk =STW_CN_HPT_T4_TL&lnk2=goto_StudioDesktop&psrc=none& amp;pexp=def [611 Sat, 01 Feb 2020 13:30:14 -0800] 29. https://www.ibm.com/account/reg/cn-zh/signup?formid=urx-3591 7&lnk=STW_CN_HPT_T3_BTN&lnk2=trial_WASCloud&psrc =none&pexp=def [610 Sat, 01 Feb 2020 13:30:14 -0800] 28. https://www.ibm.com/cn-zh/products/cognos-analytics/pricing? lnk=STW_CN_HPT_T2_BTN&lnk2=trial_CogAnalytics&psrc=n one&pexp=def [609 Sat, 01 Feb 2020 13:30:14 -0800] 27. https://www.ibm.com/account/reg/cn-zh/signup?formid=urx-3703 1&lnk=STW_CN_HPT_T1_BTN&lnk2=trial_SPSS&psrc=non e&pexp=def [608 Sat, 01 Feb 2020 13:30:14 -0800] 26. https://www.ibm.com/ibm/cn/specialoffer?lnk=STW_CN_HPT_T1_TL &lnk2=learn_SPSSstat&psrc=none&pexp=def [607 Sat, 01 Feb 2020 13:30:14 -0800] 25. https://www-31.ibm.com/robots.txt [606 Sat, 01 Feb 2020 13:30:14 -0800] 24. https://www.ibm.com/cn-zh/analytics/cloud-pak-for-data [605 Sat, 01 Feb 2020 13:30:14 -0800] 23. https://www.ibm.com/cn-zh/products/category/technology/analy tics [604 Sat, 01 Feb 2020 13:30:14 -0800] 22. https://www.ibm.com/cn-zh/products/category/business/finance [603 Sat, 01 Feb 2020 13:30:14 -0800] 21. https://www.ibm.com/cn-zh/products/category/business/commerc e [602 Sat, 01 Feb 2020 13:30:14 -0800] 20. https://www.ibm.com/cn-zh/products/category/business/supply- chain-management [601 Sat, 01 Feb 2020 13:30:14 -0800] 19. https://www.ibm.com/cn-zh/products/category/business/marketi ng [600 Sat, 01 Feb 2020 13:30:14 -0800] 18. https://www.ibm.com/cn-zh/it-infrastructure/linuxone [599 Sat, 01 Feb 2020 13:30:14 -0800] 17. https://www.ibm.com/account/reg/cn-zh/signup?formid=urx-3591 7 [598 Sat, 01 Feb 2020 13:30:14 -0800] 16. https://www.ibm.com/cn-zh/products/category/technology/softw are-development [597 Sat, 01 Feb 2020 13:30:14 -0800] 15. https://www.ibm.com/cn-zh/products/category/technology/block chain [596 Sat, 01 Feb 2020 13:30:14 -0800] 14. https://www.ibm.com/cn-zh/products/category/technology/cloud -computing [595 Sat, 01 Feb 2020 13:30:14 -0800] 13. https://www.ibm.com/cn-zh/products/category/technology/mobil e-technology [594 Sat, 01 Feb 2020 13:30:14 -0800] 12. https://www.ibm.com/ibm/cn/smart/?re=homepage [593 Sat, 01 Feb 2020 13:30:14 -0800] 11. https://www.ibm.com/cn-zh/products/category/technology/IT-ma nagement [592 Sat, 01 Feb 2020 13:30:14 -0800] 10. https://www.ibm.com/cn-zh/products/category/technology/secur ity [591 Sat, 01 Feb 2020 13:30:14 -0800] 9. https://www.ibm.com/sitemap/cn/zh [590 Sat, 01 Feb 2020 13:30:14 -0800] 8. https://www.ibm.com/cn/zh [589 Sat, 01 Feb 2020 13:30:14 -0800] 7. https://www.ibm.com/cn-zh/it-infrastructure/storage/flash?re =homepage [588 Sat, 01 Feb 2020 13:30:14 -0800] 6. https://1.cms.s81c.com/robots.txt [587 Sat, 01 Feb 2020 13:30:14 -0800] Updating Found Sites Array... [586 Sat, 01 Feb 2020 13:30:14 -0800] Number of summarized pages 73 [585 Sat, 01 Feb 2020 13:30:14 -0800] Process pages time: 3.5156898498535 Current Memory: 50979912 [584 Sat, 01 Feb 2020 13:30:14 -0800] Using Processor...XmlProcessor for https://www.ibm.com/downloads/cas/sitemap/sitemap17.xml [583 Sat, 01 Feb 2020 13:30:14 -0800] Converting from encoding ASCII... [582 Sat, 01 Feb 2020 13:30:14 -0800] Using Processor...XmlProcessor for https://www.ibm.com/downloads/cas/sitemap/sitemap16.xml [581 Sat, 01 Feb 2020 13:30:14 -0800] Converting from encoding ASCII... [580 Sat, 01 Feb 2020 13:30:13 -0800] Using Processor...XmlProcessor for https://www.ibm.com/downloads/cas/sitemap/sitemap15.xml [579 Sat, 01 Feb 2020 13:30:13 -0800] Converting from encoding ASCII... [578 Sat, 01 Feb 2020 13:30:13 -0800] Using Processor...XmlProcessor for https://www.ibm.com/downloads/cas/sitemap/sitemap14.xml [577 Sat, 01 Feb 2020 13:30:13 -0800] Converting from encoding ASCII... [576 Sat, 01 Feb 2020 13:30:13 -0800] Using Processor...XmlProcessor for https://www.ibm.com/downloads/cas/sitemap/sitemap13.xml [575 Sat, 01 Feb 2020 13:30:13 -0800] Converting from encoding ASCII... [574 Sat, 01 Feb 2020 13:30:13 -0800] Using Processor...XmlProcessor for https://www.ibm.com/downloads/cas/sitemap/sitemap12.xml [573 Sat, 01 Feb 2020 13:30:13 -0800] Converting from encoding ASCII... [572 Sat, 01 Feb 2020 13:30:13 -0800] Using Processor...XmlProcessor for https://www.ibm.com/downloads/cas/sitemap/sitemap11.xml [571 Sat, 01 Feb 2020 13:30:13 -0800] Converting from encoding ASCII... [570 Sat, 01 Feb 2020 13:30:13 -0800] Using Processor...XmlProcessor for https://www.ibm.com/downloads/cas/sitemap/sitemap10.xml [569 Sat, 01 Feb 2020 13:30:13 -0800] Converting from encoding ASCII... [568 Sat, 01 Feb 2020 13:30:13 -0800] Using Processor...XmlProcessor for https://www.ibm.com/downloads/cas/sitemap/sitemap9.xml [567 Sat, 01 Feb 2020 13:30:13 -0800] Converting from encoding ASCII... [566 Sat, 01 Feb 2020 13:30:13 -0800] Using Processor...XmlProcessor for https://www.ibm.com/downloads/cas/sitemap/sitemap8.xml [565 Sat, 01 Feb 2020 13:30:13 -0800] Converting from encoding ASCII... [564 Sat, 01 Feb 2020 13:30:13 -0800] Using Processor...XmlProcessor for https://www.ibm.com/downloads/cas/sitemap/sitemap7.xml [563 Sat, 01 Feb 2020 13:30:13 -0800] Converting from encoding ASCII... [562 Sat, 01 Feb 2020 13:30:13 -0800] Using Processor...XmlProcessor for https://www.ibm.com/downloads/cas/sitemap/sitemap6.xml [561 Sat, 01 Feb 2020 13:30:13 -0800] Converting from encoding ASCII... [560 Sat, 01 Feb 2020 13:30:13 -0800] Using Processor...XmlProcessor for https://www.ibm.com/downloads/cas/sitemap/sitemap5.xml [559 Sat, 01 Feb 2020 13:30:13 -0800] Converting from encoding ASCII... [558 Sat, 01 Feb 2020 13:30:13 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/downloads/cas/sitemap/sitemap4.xml [557 Sat, 01 Feb 2020 13:30:13 -0800] UTF-8 data detected! [556 Sat, 01 Feb 2020 13:30:13 -0800] Using Processor...XmlProcessor for https://www.ibm.com/downloads/cas/sitemap/sitemap3.xml [555 Sat, 01 Feb 2020 13:30:13 -0800] Converting from encoding ASCII... [554 Sat, 01 Feb 2020 13:30:13 -0800] Using Processor...XmlProcessor for https://www.ibm.com/downloads/cas/sitemap/sitemap2.xml [553 Sat, 01 Feb 2020 13:30:13 -0800] Converting from encoding ASCII... [552 Sat, 01 Feb 2020 13:30:13 -0800] Using Processor...XmlProcessor for https://www.ibm.com/downloads/cas/sitemap/sitemap1.xml [551 Sat, 01 Feb 2020 13:30:13 -0800] Converting from encoding ASCII... [550 Sat, 01 Feb 2020 13:30:13 -0800] Using Processor...RobotProcessor for https://1.www.s81c.com/robots.txt [549 Sat, 01 Feb 2020 13:30:13 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/accessibility/us/en/ [548 Sat, 01 Feb 2020 13:30:13 -0800] UTF-8 data detected! [547 Sat, 01 Feb 2020 13:30:13 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/privacy/us/en/ [546 Sat, 01 Feb 2020 13:30:13 -0800] UTF-8 data detected! [545 Sat, 01 Feb 2020 13:30:13 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/contact/us/en/ [544 Sat, 01 Feb 2020 13:30:13 -0800] UTF-8 data detected! [543 Sat, 01 Feb 2020 13:30:13 -0800] https://www.ibm.com/connect/ibm response code 404 [542 Sat, 01 Feb 2020 13:30:13 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/sitemap/us/en/ [541 Sat, 01 Feb 2020 13:30:13 -0800] UTF-8 data detected! [540 Sat, 01 Feb 2020 13:30:12 -0800] Using Processor...XmlProcessor for https://www.ibm.com/internal/cms/sitemap.xml?page=3 [539 Sat, 01 Feb 2020 13:30:12 -0800] Converting from encoding ASCII... [538 Sat, 01 Feb 2020 13:30:12 -0800] Using Processor...XmlProcessor for https://www.ibm.com/internal/cms/sitemap.xml?page=2 [537 Sat, 01 Feb 2020 13:30:12 -0800] Converting from encoding ASCII... [536 Sat, 01 Feb 2020 13:30:12 -0800] Using Processor...XmlProcessor for https://www.ibm.com/internal/cms/sitemap.xml?page=1 [535 Sat, 01 Feb 2020 13:30:12 -0800] Converting from encoding ASCII... [534 Sat, 01 Feb 2020 13:30:12 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/cn-zh/products/category/business/collabo ration [533 Sat, 01 Feb 2020 13:30:12 -0800] UTF-8 data detected! [532 Sat, 01 Feb 2020 13:30:12 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/cn-zh/products/category/technology/cogni tive-computing-and-AI [531 Sat, 01 Feb 2020 13:30:12 -0800] UTF-8 data detected! [530 Sat, 01 Feb 2020 13:30:12 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/cn-zh/products/category/business/busines s-operations [529 Sat, 01 Feb 2020 13:30:12 -0800] UTF-8 data detected! [528 Sat, 01 Feb 2020 13:30:12 -0800] Using Processor...RobotProcessor for https://1.dam.s81c.com/robots.txt [527 Sat, 01 Feb 2020 13:30:12 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/cn-zh/products/category/business/custome r-service-and-CRM [526 Sat, 01 Feb 2020 13:30:12 -0800] UTF-8 data detected! [525 Sat, 01 Feb 2020 13:30:12 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/cn-zh/products/category/technology/IT-in frastructure [524 Sat, 01 Feb 2020 13:30:12 -0800] UTF-8 data detected! [523 Sat, 01 Feb 2020 13:30:12 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/cn-zh/products/category/business/content -management [522 Sat, 01 Feb 2020 13:30:12 -0800] UTF-8 data detected! [521 Sat, 01 Feb 2020 13:30:12 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/cn-zh/products [520 Sat, 01 Feb 2020 13:30:12 -0800] UTF-8 data detected! [519 Sat, 01 Feb 2020 13:30:12 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/cn-zh/products/category/business/human-r esources [518 Sat, 01 Feb 2020 13:30:12 -0800] UTF-8 data detected! [517 Sat, 01 Feb 2020 13:30:12 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/cn-zh/products/watson-studio-desktop?lnk =STW_CN_HPT_T4_TL&lnk2=goto_StudioDesktop&psrc=none& amp;pexp=def [516 Sat, 01 Feb 2020 13:30:12 -0800] UTF-8 data detected! [515 Sat, 01 Feb 2020 13:30:12 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/account/reg/cn-zh/signup?formid=urx-3591 7&lnk=STW_CN_HPT_T3_BTN&lnk2=trial_WASCloud&psrc =none&pexp=def [514 Sat, 01 Feb 2020 13:30:11 -0800] UTF-8 data detected! [513 Sat, 01 Feb 2020 13:30:11 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/cn-zh/products/cognos-analytics/pricing? lnk=STW_CN_HPT_T2_BTN&lnk2=trial_CogAnalytics&psrc=n one&pexp=def [512 Sat, 01 Feb 2020 13:30:11 -0800] UTF-8 data detected! [511 Sat, 01 Feb 2020 13:30:11 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/account/reg/cn-zh/signup?formid=urx-3703 1&lnk=STW_CN_HPT_T1_BTN&lnk2=trial_SPSS&psrc=non e&pexp=def [510 Sat, 01 Feb 2020 13:30:11 -0800] UTF-8 data detected! [509 Sat, 01 Feb 2020 13:30:11 -0800] Using Processor...RobotProcessor for https://www-31.ibm.com/robots.txt [508 Sat, 01 Feb 2020 13:30:11 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/cn-zh/analytics/cloud-pak-for-data [507 Sat, 01 Feb 2020 13:30:11 -0800] UTF-8 data detected! [506 Sat, 01 Feb 2020 13:30:11 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/cn-zh/products/category/technology/analy tics [505 Sat, 01 Feb 2020 13:30:11 -0800] UTF-8 data detected! [504 Sat, 01 Feb 2020 13:30:11 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/cn-zh/products/category/business/finance [503 Sat, 01 Feb 2020 13:30:11 -0800] UTF-8 data detected! [502 Sat, 01 Feb 2020 13:30:11 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/cn-zh/products/category/business/commerc e [501 Sat, 01 Feb 2020 13:30:11 -0800] UTF-8 data detected! [500 Sat, 01 Feb 2020 13:30:11 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/cn-zh/products/category/business/marketi ng [499 Sat, 01 Feb 2020 13:30:11 -0800] UTF-8 data detected! [498 Sat, 01 Feb 2020 13:30:11 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/cn-zh/it-infrastructure/linuxone [497 Sat, 01 Feb 2020 13:30:11 -0800] UTF-8 data detected! [496 Sat, 01 Feb 2020 13:30:11 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/account/reg/cn-zh/signup?formid=urx-3591 7 [495 Sat, 01 Feb 2020 13:30:11 -0800] UTF-8 data detected! [494 Sat, 01 Feb 2020 13:30:11 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/cn-zh/products/category/technology/softw are-development [493 Sat, 01 Feb 2020 13:30:11 -0800] UTF-8 data detected! [492 Sat, 01 Feb 2020 13:30:11 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/cn-zh/products/category/technology/cloud -computing [491 Sat, 01 Feb 2020 13:30:11 -0800] UTF-8 data detected! [490 Sat, 01 Feb 2020 13:30:11 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/cn-zh/products/category/technology/mobil e-technology [489 Sat, 01 Feb 2020 13:30:11 -0800] UTF-8 data detected! [488 Sat, 01 Feb 2020 13:30:10 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/cn-zh/products/category/technology/IT-ma nagement [487 Sat, 01 Feb 2020 13:30:10 -0800] UTF-8 data detected! [486 Sat, 01 Feb 2020 13:30:10 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/cn-zh/products/category/technology/secur ity [485 Sat, 01 Feb 2020 13:30:10 -0800] UTF-8 data detected! [484 Sat, 01 Feb 2020 13:30:10 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/cn-zh/it-infrastructure/storage/flash?re =homepage [483 Sat, 01 Feb 2020 13:30:10 -0800] UTF-8 data detected! [482 Sat, 01 Feb 2020 13:30:10 -0800] Using Processor...RobotProcessor for https://1.cms.s81c.com/robots.txt [481 Sat, 01 Feb 2020 13:30:10 -0800] Start process pages... Current Memory:18601904 [480 Sat, 01 Feb 2020 13:30:10 -0800] Downloading complete [479 Sat, 01 Feb 2020 13:30:10 -0800] ...Downloads per hours: 10938.461538462 [478 Sat, 01 Feb 2020 13:30:10 -0800] ....done. [477 Sat, 01 Feb 2020 13:30:10 -0800] Scheduling again.. [476 Sat, 01 Feb 2020 13:30:10 -0800] Sort downloaded/not downloaded 3.814697265625E-5 [475 Sat, 01 Feb 2020 13:30:10 -0800] ..getPages call complete.. [474 Sat, 01 Feb 2020 13:30:10 -0800] Get Page Content time 1.7552509307861 [473 Sat, 01 Feb 2020 13:30:08 -0800] Page Request time 9.5883569717407 [472 Sat, 01 Feb 2020 13:29:59 -0800] Init Get Pages 0.032646894454956 [471 Sat, 01 Feb 2020 13:29:59 -0800] Done getting list of 73 to download... [470 Sat, 01 Feb 2020 13:29:59 -0800] Fetch url list to download time 0.0055320262908936 [469 Sat, 01 Feb 2020 13:29:59 -0800] Percent idle time per hour: 97.05% [468 Sat, 01 Feb 2020 13:29:59 -0800] Time to check Scheduler 0.022182941436768 [467 Sat, 01 Feb 2020 13:29:59 -0800] Minimum fetch loop time is: 5 seconds [466 Sat, 01 Feb 2020 13:29:59 -0800] a:20:{s:1:"b";i:1580592565;s:1:"k";i:1580592598;s:1:"Y";s:2: "ad";s:2:"dz";s:2:"-1";s:2:"ea";s:2:"-1";s:1:"D";s:1:"1";s:2 :"az";s:2:"ax";s:2:"ba";s:1:"2";s:2:"cn";i:1;s:2:"ao";a:0:{} s:1:"Z";s:0:"";s:2:"bq";a:37:{i:0;s:7:"unknown";i:1;s:3:"bmp ";i:2;s:2:"gz";i [465 Sat, 01 Feb 2020 13:29:59 -0800] First 256 bytes of first token in response: [464 Sat, 01 Feb 2020 13:29:59 -0800] Making schedule request: http://localhost:8080/?c=fetch&a=schedule&time=15805 92599&session=ba32a699d10f86871424b03ee8b69477&robot _instance=0-0-localhost-1580591967&machine_uri=/&cra wl_time=1580592565&check_crawl_time=1580592599 [463 Sat, 01 Feb 2020 13:29:59 -0800] Done curl exec [462 Sat, 01 Feb 2020 13:29:59 -0800] Set curl options for single page request [461 Sat, 01 Feb 2020 13:29:59 -0800] Init curl request of a single page [460 Sat, 01 Feb 2020 13:29:59 -0800] Checking http://localhost:8080/ for a new schedule. [459 Sat, 01 Feb 2020 13:29:59 -0800] MAIN LOOP CASE 4 -- WEB SCHEDULER [458 Sat, 01 Feb 2020 13:29:59 -0800] Crawl time stayed same! [457 Sat, 01 Feb 2020 13:29:59 -0800] End Name Server Check [456 Sat, 01 Feb 2020 13:29:59 -0800] Start of response was:a:7:{s:1:"a";i:1;s:1:"b";i:1580592565;s:2:"dq";i:5;s:2:" dv";s:4484:"YTo3OntpOjA7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoi TkFNRSI7czo2OiJEUlVQQUwiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5 OiJTSUdOQVRVUkUiO3M6MTI1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhy ZWYsICcvc2l0ZXMvYWxs [455 Sat, 01 Feb 2020 13:29:59 -0800] Done curl exec [454 Sat, 01 Feb 2020 13:29:59 -0800] Set curl options for single page request [453 Sat, 01 Feb 2020 13:29:59 -0800] Init curl request of a single page [452 Sat, 01 Feb 2020 13:29:59 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592599&session=ba32a699d10f86871424b03ee8b69477& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=1580592565 [451 Sat, 01 Feb 2020 13:29:59 -0800] http://localhost:8080/ to see if active crawl time has changed. [450 Sat, 01 Feb 2020 13:29:59 -0800] Checking name server: [449 Sat, 01 Feb 2020 13:29:49 -0800] No data. Sleeping... [448 Sat, 01 Feb 2020 13:29:49 -0800] Percent idle time per hour: 96.92% [447 Sat, 01 Feb 2020 13:29:49 -0800] Time to check Scheduler 0.0053818225860596 [446 Sat, 01 Feb 2020 13:29:49 -0800] Minimum fetch loop time is: 5 seconds [445 Sat, 01 Feb 2020 13:29:49 -0800] a:1:{s:1:"a";i:2;} [444 Sat, 01 Feb 2020 13:29:49 -0800] First 256 bytes of first token in response: [443 Sat, 01 Feb 2020 13:29:49 -0800] Making schedule request: http://localhost:8080/?c=fetch&a=schedule&time=15805 92589&session=04b4246375a5be3a6dcb65fe717764d8&robot _instance=0-0-localhost-1580591967&machine_uri=/&cra wl_time=1580592565&check_crawl_time=1580592589 [442 Sat, 01 Feb 2020 13:29:49 -0800] Done curl exec [441 Sat, 01 Feb 2020 13:29:49 -0800] Set curl options for single page request [440 Sat, 01 Feb 2020 13:29:49 -0800] Init curl request of a single page [439 Sat, 01 Feb 2020 13:29:49 -0800] Checking http://localhost:8080/ for a new schedule. [438 Sat, 01 Feb 2020 13:29:49 -0800] MAIN LOOP CASE 4 -- WEB SCHEDULER [437 Sat, 01 Feb 2020 13:29:49 -0800] Crawl time stayed same! [436 Sat, 01 Feb 2020 13:29:49 -0800] End Name Server Check [435 Sat, 01 Feb 2020 13:29:49 -0800] Start of response was:a:7:{s:1:"a";i:1;s:1:"b";i:1580592565;s:2:"dq";i:5;s:2:" dv";s:4484:"YTo3OntpOjA7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoi TkFNRSI7czo2OiJEUlVQQUwiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5 OiJTSUdOQVRVUkUiO3M6MTI1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhy ZWYsICcvc2l0ZXMvYWxs [434 Sat, 01 Feb 2020 13:29:49 -0800] Done curl exec [433 Sat, 01 Feb 2020 13:29:49 -0800] Set curl options for single page request [432 Sat, 01 Feb 2020 13:29:49 -0800] Init curl request of a single page [431 Sat, 01 Feb 2020 13:29:49 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592589&session=04b4246375a5be3a6dcb65fe717764d8& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=1580592565 [430 Sat, 01 Feb 2020 13:29:49 -0800] http://localhost:8080/ to see if active crawl time has changed. [429 Sat, 01 Feb 2020 13:29:49 -0800] Checking name server: [428 Sat, 01 Feb 2020 13:29:49 -0800] Update Server Time 4.7075169086456 [427 Sat, 01 Feb 2020 13:29:49 -0800] ... Current Memory:13067416 [426 Sat, 01 Feb 2020 13:29:49 -0800] ... Upload bytes per hours: 0.31948726288758 [425 Sat, 01 Feb 2020 13:29:49 -0800] Updated Queue Server, sent approximately 140272 bytes: [424 Sat, 01 Feb 2020 13:29:49 -0800] This fetcher peak memory usage: 465622792 [423 Sat, 01 Feb 2020 13:29:49 -0800] Web Server peak memory usage: 9456560 [422 Sat, 01 Feb 2020 13:29:49 -0800] Queue Server's crawl time is: 0 [421 Sat, 01 Feb 2020 13:29:49 -0800] Queue Server info response code: 1 [420 Sat, 01 Feb 2020 13:29:49 -0800] ... Data upload complete [419 Sat, 01 Feb 2020 13:29:49 -0800] Messages from Fetch Controller: [418 Sat, 01 Feb 2020 13:29:49 -0800] Done curl exec [417 Sat, 01 Feb 2020 13:29:49 -0800] Set curl options for single page request [416 Sat, 01 Feb 2020 13:29:49 -0800] Init curl request of a single page [415 Sat, 01 Feb 2020 13:29:49 -0800] ...sending about 140272 bytes. [414 Sat, 01 Feb 2020 13:29:49 -0800] Sending Queue Server Part 1 of 1... [413 Sat, 01 Feb 2020 13:29:49 -0800] ... [412 Sat, 01 Feb 2020 13:29:49 -0800] ...137136 bytes of index data [411 Sat, 01 Feb 2020 13:29:49 -0800] .....web encode result. [410 Sat, 01 Feb 2020 13:29:49 -0800] .....add inverted index string. [409 Sat, 01 Feb 2020 13:29:49 -0800] ...Finish Compressing seen URLs. [408 Sat, 01 Feb 2020 13:29:48 -0800] Saving index shard .. wrote doc map. Done save [407 Sat, 01 Feb 2020 13:29:48 -0800] Saving index shard .. packed header [406 Sat, 01 Feb 2020 13:29:48 -0800] Saving index shard .. make prefixes [405 Sat, 01 Feb 2020 13:29:48 -0800] Saving index shard .. done merge postings to string [404 Sat, 01 Feb 2020 13:29:48 -0800] ..Done Merge Index Posting Final Copy [403 Sat, 01 Feb 2020 13:29:48 -0800] ..Merge Index Posting Final Copy [402 Sat, 01 Feb 2020 13:29:48 -0800] Merge index shard postings to string to save memory. [401 Sat, 01 Feb 2020 13:29:48 -0800] Saving Mini Inverted Index... [400 Sat, 01 Feb 2020 13:29:48 -0800] Build mini inverted index time 4.5683839321136 [399 Sat, 01 Feb 2020 13:29:44 -0800] Start building mini inverted index ... Current Memory:3511880 [398 Sat, 01 Feb 2020 13:29:44 -0800] ...2668 bytes of schedule data [397 Sat, 01 Feb 2020 13:29:44 -0800] ...468 bytes of cache page validation data [396 Sat, 01 Feb 2020 13:29:44 -0800] Updating machine: http://localhost:8080/ [395 Sat, 01 Feb 2020 13:29:44 -0800] Start Update Server [394 Sat, 01 Feb 2020 13:29:44 -0800] Done Update Found Sites Array Time 0.0041420459747314 [393 Sat, 01 Feb 2020 13:29:44 -0800] 5. https://www.ibm.com/downloads/cas/sitemap/sitemap.xml [392 Sat, 01 Feb 2020 13:29:44 -0800] 4. https://www.ibm.com/marketplace/storefront-sitemap-index.xml [391 Sat, 01 Feb 2020 13:29:44 -0800] 3. https://www.ibm.com/connect/ibm/sitemap.xml [390 Sat, 01 Feb 2020 13:29:44 -0800] 2. https://www.ibm.com/internal/cms/sitemap-cms.xml [389 Sat, 01 Feb 2020 13:29:44 -0800] 1. https://www.ibm.com/homepage_sitemap.xml [388 Sat, 01 Feb 2020 13:29:44 -0800] 0. https://www.ibm.com/cn-zh [387 Sat, 01 Feb 2020 13:29:44 -0800] Updating Found Sites Array... [386 Sat, 01 Feb 2020 13:29:44 -0800] Number of summarized pages 6 [385 Sat, 01 Feb 2020 13:29:44 -0800] Process pages time: 0.26411604881287 Current Memory: 3508592 [384 Sat, 01 Feb 2020 13:29:44 -0800] Using Processor...XmlProcessor for https://www.ibm.com/downloads/cas/sitemap/sitemap.xml [383 Sat, 01 Feb 2020 13:29:44 -0800] Converting from encoding ASCII... [382 Sat, 01 Feb 2020 13:29:44 -0800] Using Processor...XmlProcessor for https://www.ibm.com/marketplace/storefront-sitemap-index.xml [381 Sat, 01 Feb 2020 13:29:44 -0800] Converting from encoding ASCII... [380 Sat, 01 Feb 2020 13:29:44 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/connect/ibm/sitemap.xml [379 Sat, 01 Feb 2020 13:29:44 -0800] UTF-8 data detected! [378 Sat, 01 Feb 2020 13:29:44 -0800] https://www.ibm.com/connect/ibm/sitemap.xml response code 404 [377 Sat, 01 Feb 2020 13:29:44 -0800] Using Processor...XmlProcessor for https://www.ibm.com/internal/cms/sitemap-cms.xml [376 Sat, 01 Feb 2020 13:29:44 -0800] Converting from encoding ASCII... [375 Sat, 01 Feb 2020 13:29:44 -0800] Using Processor...XmlProcessor for https://www.ibm.com/homepage_sitemap.xml [374 Sat, 01 Feb 2020 13:29:44 -0800] Converting from encoding ASCII... [373 Sat, 01 Feb 2020 13:29:44 -0800] Using Processor...HtmlProcessor for https://www.ibm.com/cn-zh [372 Sat, 01 Feb 2020 13:29:44 -0800] UTF-8 data detected! [371 Sat, 01 Feb 2020 13:29:44 -0800] Start process pages... Current Memory:2996264 [370 Sat, 01 Feb 2020 13:29:44 -0800] Downloading complete [369 Sat, 01 Feb 2020 13:29:44 -0800] ...Downloads per hours: 1.3665760689157E-5 [368 Sat, 01 Feb 2020 13:29:44 -0800] ....done. [367 Sat, 01 Feb 2020 13:29:44 -0800] Scheduling again.. [366 Sat, 01 Feb 2020 13:29:44 -0800] Sort downloaded/not downloaded 7.1525573730469E-6 [365 Sat, 01 Feb 2020 13:29:44 -0800] ..getPages call complete.. [364 Sat, 01 Feb 2020 13:29:44 -0800] Get Page Content time 0.077270984649658 [363 Sat, 01 Feb 2020 13:29:44 -0800] FetchUrl initial processing of page 0 of 6 Current memory usage:2993368 [362 Sat, 01 Feb 2020 13:29:44 -0800] Page Request time 0.75996899604797 [361 Sat, 01 Feb 2020 13:29:43 -0800] Init Get Pages 0.0026259422302246 [360 Sat, 01 Feb 2020 13:29:43 -0800] Done getting list of 6 to download... [359 Sat, 01 Feb 2020 13:29:43 -0800] Fetch url list to download time 0.0028190612792969 [358 Sat, 01 Feb 2020 13:29:43 -0800] Percent idle time per hour: 99.55% [357 Sat, 01 Feb 2020 13:29:43 -0800] Time to check Scheduler 0.01891303062439 [356 Sat, 01 Feb 2020 13:29:43 -0800] Minimum fetch loop time is: 5 seconds [355 Sat, 01 Feb 2020 13:29:43 -0800] a:20:{s:1:"b";i:1580592565;s:1:"k";i:1580592583;s:1:"Y";s:2: "ad";s:2:"dz";s:2:"-1";s:2:"ea";s:2:"-1";s:1:"D";s:1:"1";s:2 :"az";s:2:"ax";s:2:"ba";s:1:"2";s:2:"cn";i:1;s:2:"ao";a:0:{} s:1:"Z";s:0:"";s:2:"bq";a:37:{i:0;s:7:"unknown";i:1;s:3:"bmp ";i:2;s:2:"gz";i [354 Sat, 01 Feb 2020 13:29:43 -0800] First 256 bytes of first token in response: [353 Sat, 01 Feb 2020 13:29:43 -0800] Making schedule request: http://localhost:8080/?c=fetch&a=schedule&time=15805 92583&session=75d340934b1d1ec59ea5bb6b4d1a6dcf&robot _instance=0-0-localhost-1580591967&machine_uri=/&cra wl_time=1580592565&check_crawl_time=1580592583 [352 Sat, 01 Feb 2020 13:29:43 -0800] Done curl exec [351 Sat, 01 Feb 2020 13:29:43 -0800] Set curl options for single page request [350 Sat, 01 Feb 2020 13:29:43 -0800] Init curl request of a single page [349 Sat, 01 Feb 2020 13:29:43 -0800] Checking http://localhost:8080/ for a new schedule. [348 Sat, 01 Feb 2020 13:29:43 -0800] MAIN LOOP CASE 4 -- WEB SCHEDULER [347 Sat, 01 Feb 2020 13:29:43 -0800] Crawl time stayed same! [346 Sat, 01 Feb 2020 13:29:43 -0800] End Name Server Check [345 Sat, 01 Feb 2020 13:29:43 -0800] Start of response was:a:7:{s:1:"a";i:1;s:1:"b";i:1580592565;s:2:"dq";i:5;s:2:" dv";s:4484:"YTo3OntpOjA7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoi TkFNRSI7czo2OiJEUlVQQUwiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5 OiJTSUdOQVRVUkUiO3M6MTI1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhy ZWYsICcvc2l0ZXMvYWxs [344 Sat, 01 Feb 2020 13:29:43 -0800] Done curl exec [343 Sat, 01 Feb 2020 13:29:43 -0800] Set curl options for single page request [342 Sat, 01 Feb 2020 13:29:43 -0800] Init curl request of a single page [341 Sat, 01 Feb 2020 13:29:43 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592583&session=75d340934b1d1ec59ea5bb6b4d1a6dcf& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=1580592565 [340 Sat, 01 Feb 2020 13:29:43 -0800] http://localhost:8080/ to see if active crawl time has changed. [339 Sat, 01 Feb 2020 13:29:43 -0800] Checking name server: [338 Sat, 01 Feb 2020 13:29:33 -0800] No data. Sleeping... [337 Sat, 01 Feb 2020 13:29:33 -0800] Percent idle time per hour: 99.53% [336 Sat, 01 Feb 2020 13:29:33 -0800] Time to check Scheduler 0.0054500102996826 [335 Sat, 01 Feb 2020 13:29:33 -0800] Minimum fetch loop time is: 5 seconds [334 Sat, 01 Feb 2020 13:29:33 -0800] a:1:{s:1:"a";i:2;} [333 Sat, 01 Feb 2020 13:29:33 -0800] First 256 bytes of first token in response: [332 Sat, 01 Feb 2020 13:29:33 -0800] Making schedule request: http://localhost:8080/?c=fetch&a=schedule&time=15805 92573&session=a204671d811c0b26210300de570be0ab&robot _instance=0-0-localhost-1580591967&machine_uri=/&cra wl_time=1580592565&check_crawl_time=1580592573 [331 Sat, 01 Feb 2020 13:29:33 -0800] Done curl exec [330 Sat, 01 Feb 2020 13:29:33 -0800] Set curl options for single page request [329 Sat, 01 Feb 2020 13:29:33 -0800] Init curl request of a single page [328 Sat, 01 Feb 2020 13:29:33 -0800] Checking http://localhost:8080/ for a new schedule. [327 Sat, 01 Feb 2020 13:29:33 -0800] MAIN LOOP CASE 4 -- WEB SCHEDULER [326 Sat, 01 Feb 2020 13:29:33 -0800] Crawl time stayed same! [325 Sat, 01 Feb 2020 13:29:33 -0800] End Name Server Check [324 Sat, 01 Feb 2020 13:29:33 -0800] Start of response was:a:7:{s:1:"a";i:1;s:1:"b";i:1580592565;s:2:"dq";i:5;s:2:" dv";s:4484:"YTo3OntpOjA7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoi TkFNRSI7czo2OiJEUlVQQUwiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5 OiJTSUdOQVRVUkUiO3M6MTI1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhy ZWYsICcvc2l0ZXMvYWxs [323 Sat, 01 Feb 2020 13:29:33 -0800] Done curl exec [322 Sat, 01 Feb 2020 13:29:33 -0800] Set curl options for single page request [321 Sat, 01 Feb 2020 13:29:33 -0800] Init curl request of a single page [320 Sat, 01 Feb 2020 13:29:33 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592573&session=a204671d811c0b26210300de570be0ab& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=1580592565 [319 Sat, 01 Feb 2020 13:29:33 -0800] http://localhost:8080/ to see if active crawl time has changed. [318 Sat, 01 Feb 2020 13:29:33 -0800] Checking name server: [317 Sat, 01 Feb 2020 13:29:33 -0800] Switching archive... [316 Sat, 01 Feb 2020 13:29:33 -0800] New name: C:/Users/Forrest-m/yioop/yioop2/yioop/work_directory/cache/0 -Archive1580592565 [315 Sat, 01 Feb 2020 13:29:33 -0800] Percent idle time per hour: 99.53% [314 Sat, 01 Feb 2020 13:29:33 -0800] Crawl time is now 1580592565 [313 Sat, 01 Feb 2020 13:29:33 -0800] MAIN LOOP CASE 1 -- SWITCH CRAWL OR NO CURRENT CRAWL [312 Sat, 01 Feb 2020 13:29:33 -0800] Crawl time has changed! [311 Sat, 01 Feb 2020 13:29:33 -0800] End Name Server Check [310 Sat, 01 Feb 2020 13:29:33 -0800] Minimum fetch loop time is: 5 seconds [309 Sat, 01 Feb 2020 13:29:33 -0800] New Crawl Time Found: 1580592565 [308 Sat, 01 Feb 2020 13:29:33 -0800] ...Removed 0 cullable URLS from to crawl lists in time: 5.0067901611328E-6 [307 Sat, 01 Feb 2020 13:29:33 -0800] Culling noncrawlable urls after change in crawl parameters; To Crawl Count: 0, To Crawl Again: 0 [306 Sat, 01 Feb 2020 13:29:33 -0800] Checking if urls in to crawl lists need to be culled [305 Sat, 01 Feb 2020 13:29:33 -0800] Allowed/Disallowed Urls have changed [304 Sat, 01 Feb 2020 13:29:33 -0800] Start of response was:a:17:{s:1:"a";i:1;s:1:"b";i:1580592565;s:2:"dq";i:5;s:2: "aa";a:3:{i:0;s:21:"http://www.yahoo.com/";i:1;s:23:"http:// www.youtube.com/";i:2;s:22:"http://www.google.com/";}s:2:"cb ";s:3:"MIX";s:2:"cc";s:16:"MixArchiveBundle";s:2:"ba";s:1:"2 ";s:2:"az";s:2:"ax"; [303 Sat, 01 Feb 2020 13:29:33 -0800] Done curl exec [302 Sat, 01 Feb 2020 13:29:33 -0800] Set curl options for single page request [301 Sat, 01 Feb 2020 13:29:33 -0800] Init curl request of a single page [300 Sat, 01 Feb 2020 13:29:33 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592573&session=a204671d811c0b26210300de570be0ab& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=0 [299 Sat, 01 Feb 2020 13:29:33 -0800] http://localhost:8080/ to see if should start crawling [298 Sat, 01 Feb 2020 13:29:33 -0800] Checking name server: [297 Sat, 01 Feb 2020 13:29:23 -0800] No data. Sleeping... [296 Sat, 01 Feb 2020 13:29:23 -0800] Percent idle time per hour: 99.50% [295 Sat, 01 Feb 2020 13:29:23 -0800] MAIN LOOP CASE 1 -- SWITCH CRAWL OR NO CURRENT CRAWL [294 Sat, 01 Feb 2020 13:29:23 -0800] Crawl time has changed! [293 Sat, 01 Feb 2020 13:29:23 -0800] End Name Server Check [292 Sat, 01 Feb 2020 13:29:23 -0800] Minimum fetch loop time is: 5 seconds [291 Sat, 01 Feb 2020 13:29:23 -0800] Crawl Time Changing to 0 -- No Crawl [290 Sat, 01 Feb 2020 13:29:23 -0800] Start of response was:a:6:{s:1:"a";i:1;s:1:"b";i:0;s:2:"dv";s:4484:"YTo3OntpOj A7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoiTkFNRSI7czo2OiJEUlVQQU wiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5OiJTSUdOQVRVUkUiO3M6MT I1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhyZWYsICcvc2l0ZXMvYWxsL3 RoZW1lcycpIG9yIGNvbn [289 Sat, 01 Feb 2020 13:29:23 -0800] Done curl exec [288 Sat, 01 Feb 2020 13:29:23 -0800] Set curl options for single page request [287 Sat, 01 Feb 2020 13:29:23 -0800] Init curl request of a single page [286 Sat, 01 Feb 2020 13:29:23 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592563&session=7660a37d7b214330eaaae435dd196803& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=0 [285 Sat, 01 Feb 2020 13:29:23 -0800] http://localhost:8080/ to see if should start crawling [284 Sat, 01 Feb 2020 13:29:23 -0800] Checking name server: [283 Sat, 01 Feb 2020 13:29:13 -0800] No data. Sleeping... [282 Sat, 01 Feb 2020 13:29:13 -0800] Percent idle time per hour: 99.48% [281 Sat, 01 Feb 2020 13:29:13 -0800] MAIN LOOP CASE 1 -- SWITCH CRAWL OR NO CURRENT CRAWL [280 Sat, 01 Feb 2020 13:29:13 -0800] Crawl time has changed! [279 Sat, 01 Feb 2020 13:29:13 -0800] End Name Server Check [278 Sat, 01 Feb 2020 13:29:13 -0800] Minimum fetch loop time is: 5 seconds [277 Sat, 01 Feb 2020 13:29:13 -0800] Crawl Time Changing to 0 -- No Crawl [276 Sat, 01 Feb 2020 13:29:13 -0800] Start of response was:a:6:{s:1:"a";i:1;s:1:"b";i:0;s:2:"dv";s:4484:"YTo3OntpOj A7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoiTkFNRSI7czo2OiJEUlVQQU wiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5OiJTSUdOQVRVUkUiO3M6MT I1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhyZWYsICcvc2l0ZXMvYWxsL3 RoZW1lcycpIG9yIGNvbn [275 Sat, 01 Feb 2020 13:29:13 -0800] Done curl exec [274 Sat, 01 Feb 2020 13:29:13 -0800] Set curl options for single page request [273 Sat, 01 Feb 2020 13:29:13 -0800] Init curl request of a single page [272 Sat, 01 Feb 2020 13:29:13 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592553&session=983a4513b564a2781e3ce34fd063b35a& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=0 [271 Sat, 01 Feb 2020 13:29:13 -0800] http://localhost:8080/ to see if should start crawling [270 Sat, 01 Feb 2020 13:29:13 -0800] Checking name server: [269 Sat, 01 Feb 2020 13:29:03 -0800] No data. Sleeping... [268 Sat, 01 Feb 2020 13:29:03 -0800] Percent idle time per hour: 99.45% [267 Sat, 01 Feb 2020 13:29:03 -0800] MAIN LOOP CASE 1 -- SWITCH CRAWL OR NO CURRENT CRAWL [266 Sat, 01 Feb 2020 13:29:03 -0800] Crawl time has changed! [265 Sat, 01 Feb 2020 13:29:03 -0800] End Name Server Check [264 Sat, 01 Feb 2020 13:29:03 -0800] Minimum fetch loop time is: 5 seconds [263 Sat, 01 Feb 2020 13:29:03 -0800] Crawl Time Changing to 0 -- No Crawl [262 Sat, 01 Feb 2020 13:29:03 -0800] Start of response was:a:6:{s:1:"a";i:1;s:1:"b";i:0;s:2:"dv";s:4484:"YTo3OntpOj A7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoiTkFNRSI7czo2OiJEUlVQQU wiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5OiJTSUdOQVRVUkUiO3M6MT I1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhyZWYsICcvc2l0ZXMvYWxsL3 RoZW1lcycpIG9yIGNvbn [261 Sat, 01 Feb 2020 13:29:03 -0800] Done curl exec [260 Sat, 01 Feb 2020 13:29:03 -0800] Set curl options for single page request [259 Sat, 01 Feb 2020 13:29:03 -0800] Init curl request of a single page [258 Sat, 01 Feb 2020 13:29:03 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592543&session=1a5cd2dd86dc73aa35061a099424a59e& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=0 [257 Sat, 01 Feb 2020 13:29:03 -0800] http://localhost:8080/ to see if should start crawling [256 Sat, 01 Feb 2020 13:29:03 -0800] Checking name server: [255 Sat, 01 Feb 2020 13:28:53 -0800] No data. Sleeping... [254 Sat, 01 Feb 2020 13:28:53 -0800] Percent idle time per hour: 99.42% [253 Sat, 01 Feb 2020 13:28:53 -0800] MAIN LOOP CASE 1 -- SWITCH CRAWL OR NO CURRENT CRAWL [252 Sat, 01 Feb 2020 13:28:53 -0800] Crawl time has changed! [251 Sat, 01 Feb 2020 13:28:53 -0800] End Name Server Check [250 Sat, 01 Feb 2020 13:28:53 -0800] Minimum fetch loop time is: 5 seconds [249 Sat, 01 Feb 2020 13:28:53 -0800] Crawl Time Changing to 0 -- No Crawl [248 Sat, 01 Feb 2020 13:28:53 -0800] Start of response was:a:6:{s:1:"a";i:1;s:1:"b";i:0;s:2:"dv";s:4484:"YTo3OntpOj A7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoiTkFNRSI7czo2OiJEUlVQQU wiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5OiJTSUdOQVRVUkUiO3M6MT I1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhyZWYsICcvc2l0ZXMvYWxsL3 RoZW1lcycpIG9yIGNvbn [247 Sat, 01 Feb 2020 13:28:53 -0800] Done curl exec [246 Sat, 01 Feb 2020 13:28:53 -0800] Set curl options for single page request [245 Sat, 01 Feb 2020 13:28:53 -0800] Init curl request of a single page [244 Sat, 01 Feb 2020 13:28:53 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592533&session=a85d3f79d357edeaee1465922279974e& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=0 [243 Sat, 01 Feb 2020 13:28:53 -0800] http://localhost:8080/ to see if should start crawling [242 Sat, 01 Feb 2020 13:28:53 -0800] Checking name server: [241 Sat, 01 Feb 2020 13:28:43 -0800] No data. Sleeping... [240 Sat, 01 Feb 2020 13:28:43 -0800] Percent idle time per hour: 99.38% [239 Sat, 01 Feb 2020 13:28:43 -0800] MAIN LOOP CASE 1 -- SWITCH CRAWL OR NO CURRENT CRAWL [238 Sat, 01 Feb 2020 13:28:43 -0800] Crawl time has changed! [237 Sat, 01 Feb 2020 13:28:43 -0800] End Name Server Check [236 Sat, 01 Feb 2020 13:28:43 -0800] Minimum fetch loop time is: 5 seconds [235 Sat, 01 Feb 2020 13:28:43 -0800] Crawl Time Changing to 0 -- No Crawl [234 Sat, 01 Feb 2020 13:28:43 -0800] Start of response was:a:6:{s:1:"a";i:1;s:1:"b";i:0;s:2:"dv";s:4484:"YTo3OntpOj A7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoiTkFNRSI7czo2OiJEUlVQQU wiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5OiJTSUdOQVRVUkUiO3M6MT I1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhyZWYsICcvc2l0ZXMvYWxsL3 RoZW1lcycpIG9yIGNvbn [233 Sat, 01 Feb 2020 13:28:43 -0800] Done curl exec [232 Sat, 01 Feb 2020 13:28:43 -0800] Set curl options for single page request [231 Sat, 01 Feb 2020 13:28:43 -0800] Init curl request of a single page [230 Sat, 01 Feb 2020 13:28:43 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592523&session=b5a294296a09d3fe4835230a95160684& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=0 [229 Sat, 01 Feb 2020 13:28:43 -0800] http://localhost:8080/ to see if should start crawling [228 Sat, 01 Feb 2020 13:28:43 -0800] Checking name server: [227 Sat, 01 Feb 2020 13:28:33 -0800] No data. Sleeping... [226 Sat, 01 Feb 2020 13:28:33 -0800] Percent idle time per hour: 99.34% [225 Sat, 01 Feb 2020 13:28:33 -0800] MAIN LOOP CASE 1 -- SWITCH CRAWL OR NO CURRENT CRAWL [224 Sat, 01 Feb 2020 13:28:33 -0800] Crawl time has changed! [223 Sat, 01 Feb 2020 13:28:33 -0800] End Name Server Check [222 Sat, 01 Feb 2020 13:28:33 -0800] Minimum fetch loop time is: 5 seconds [221 Sat, 01 Feb 2020 13:28:33 -0800] Crawl Time Changing to 0 -- No Crawl [220 Sat, 01 Feb 2020 13:28:33 -0800] Start of response was:a:6:{s:1:"a";i:1;s:1:"b";i:0;s:2:"dv";s:4484:"YTo3OntpOj A7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoiTkFNRSI7czo2OiJEUlVQQU wiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5OiJTSUdOQVRVUkUiO3M6MT I1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhyZWYsICcvc2l0ZXMvYWxsL3 RoZW1lcycpIG9yIGNvbn [219 Sat, 01 Feb 2020 13:28:33 -0800] Done curl exec [218 Sat, 01 Feb 2020 13:28:33 -0800] Set curl options for single page request [217 Sat, 01 Feb 2020 13:28:33 -0800] Init curl request of a single page [216 Sat, 01 Feb 2020 13:28:33 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592513&session=ce865a5ee510cd181dcf4b16c5215244& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=0 [215 Sat, 01 Feb 2020 13:28:33 -0800] http://localhost:8080/ to see if should start crawling [214 Sat, 01 Feb 2020 13:28:33 -0800] Checking name server: [213 Sat, 01 Feb 2020 13:28:23 -0800] No data. Sleeping... [212 Sat, 01 Feb 2020 13:28:23 -0800] Percent idle time per hour: 99.29% [211 Sat, 01 Feb 2020 13:28:23 -0800] MAIN LOOP CASE 1 -- SWITCH CRAWL OR NO CURRENT CRAWL [210 Sat, 01 Feb 2020 13:28:23 -0800] Crawl time has changed! [209 Sat, 01 Feb 2020 13:28:23 -0800] End Name Server Check [208 Sat, 01 Feb 2020 13:28:23 -0800] Minimum fetch loop time is: 5 seconds [207 Sat, 01 Feb 2020 13:28:23 -0800] Crawl Time Changing to 0 -- No Crawl [206 Sat, 01 Feb 2020 13:28:23 -0800] Start of response was:a:6:{s:1:"a";i:1;s:1:"b";i:0;s:2:"dv";s:4484:"YTo3OntpOj A7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoiTkFNRSI7czo2OiJEUlVQQU wiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5OiJTSUdOQVRVUkUiO3M6MT I1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhyZWYsICcvc2l0ZXMvYWxsL3 RoZW1lcycpIG9yIGNvbn [205 Sat, 01 Feb 2020 13:28:23 -0800] Done curl exec [204 Sat, 01 Feb 2020 13:28:23 -0800] Set curl options for single page request [203 Sat, 01 Feb 2020 13:28:23 -0800] Init curl request of a single page [202 Sat, 01 Feb 2020 13:28:23 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592503&session=93c30178937cb93c7b19e425a09fc98a& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=0 [201 Sat, 01 Feb 2020 13:28:23 -0800] http://localhost:8080/ to see if should start crawling [200 Sat, 01 Feb 2020 13:28:23 -0800] Checking name server: [199 Sat, 01 Feb 2020 13:28:13 -0800] No data. Sleeping... [198 Sat, 01 Feb 2020 13:28:13 -0800] Percent idle time per hour: 99.24% [197 Sat, 01 Feb 2020 13:28:13 -0800] MAIN LOOP CASE 1 -- SWITCH CRAWL OR NO CURRENT CRAWL [196 Sat, 01 Feb 2020 13:28:13 -0800] Crawl time has changed! [195 Sat, 01 Feb 2020 13:28:13 -0800] End Name Server Check [194 Sat, 01 Feb 2020 13:28:13 -0800] Minimum fetch loop time is: 5 seconds [193 Sat, 01 Feb 2020 13:28:13 -0800] Crawl Time Changing to 0 -- No Crawl [192 Sat, 01 Feb 2020 13:28:13 -0800] Start of response was:a:6:{s:1:"a";i:1;s:1:"b";i:0;s:2:"dv";s:4484:"YTo3OntpOj A7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoiTkFNRSI7czo2OiJEUlVQQU wiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5OiJTSUdOQVRVUkUiO3M6MT I1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhyZWYsICcvc2l0ZXMvYWxsL3 RoZW1lcycpIG9yIGNvbn [191 Sat, 01 Feb 2020 13:28:13 -0800] Done curl exec [190 Sat, 01 Feb 2020 13:28:13 -0800] Set curl options for single page request [189 Sat, 01 Feb 2020 13:28:13 -0800] Init curl request of a single page [188 Sat, 01 Feb 2020 13:28:13 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592493&session=c4f3ce5702567f5d2267406308444735& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=0 [187 Sat, 01 Feb 2020 13:28:13 -0800] http://localhost:8080/ to see if should start crawling [186 Sat, 01 Feb 2020 13:28:13 -0800] Checking name server: [185 Sat, 01 Feb 2020 13:28:03 -0800] No data. Sleeping... [184 Sat, 01 Feb 2020 13:28:03 -0800] Percent idle time per hour: 99.17% [183 Sat, 01 Feb 2020 13:28:03 -0800] MAIN LOOP CASE 1 -- SWITCH CRAWL OR NO CURRENT CRAWL [182 Sat, 01 Feb 2020 13:28:03 -0800] Crawl time has changed! [181 Sat, 01 Feb 2020 13:28:03 -0800] End Name Server Check [180 Sat, 01 Feb 2020 13:28:03 -0800] Minimum fetch loop time is: 5 seconds [179 Sat, 01 Feb 2020 13:28:03 -0800] Crawl Time Changing to 0 -- No Crawl [178 Sat, 01 Feb 2020 13:28:03 -0800] Start of response was:a:6:{s:1:"a";i:1;s:1:"b";i:0;s:2:"dv";s:4484:"YTo3OntpOj A7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoiTkFNRSI7czo2OiJEUlVQQU wiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5OiJTSUdOQVRVUkUiO3M6MT I1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhyZWYsICcvc2l0ZXMvYWxsL3 RoZW1lcycpIG9yIGNvbn [177 Sat, 01 Feb 2020 13:28:03 -0800] Done curl exec [176 Sat, 01 Feb 2020 13:28:03 -0800] Set curl options for single page request [175 Sat, 01 Feb 2020 13:28:03 -0800] Init curl request of a single page [174 Sat, 01 Feb 2020 13:28:03 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592483&session=590f45689626d4baf716da8cb0db010c& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=0 [173 Sat, 01 Feb 2020 13:28:03 -0800] http://localhost:8080/ to see if should start crawling [172 Sat, 01 Feb 2020 13:28:03 -0800] Checking name server: [171 Sat, 01 Feb 2020 13:27:53 -0800] No data. Sleeping... [170 Sat, 01 Feb 2020 13:27:53 -0800] Percent idle time per hour: 99.10% [169 Sat, 01 Feb 2020 13:27:53 -0800] MAIN LOOP CASE 1 -- SWITCH CRAWL OR NO CURRENT CRAWL [168 Sat, 01 Feb 2020 13:27:53 -0800] Crawl time has changed! [167 Sat, 01 Feb 2020 13:27:53 -0800] End Name Server Check [166 Sat, 01 Feb 2020 13:27:53 -0800] Minimum fetch loop time is: 5 seconds [165 Sat, 01 Feb 2020 13:27:53 -0800] Crawl Time Changing to 0 -- No Crawl [164 Sat, 01 Feb 2020 13:27:53 -0800] Start of response was:a:6:{s:1:"a";i:1;s:1:"b";i:0;s:2:"dv";s:4484:"YTo3OntpOj A7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoiTkFNRSI7czo2OiJEUlVQQU wiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5OiJTSUdOQVRVUkUiO3M6MT I1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhyZWYsICcvc2l0ZXMvYWxsL3 RoZW1lcycpIG9yIGNvbn [163 Sat, 01 Feb 2020 13:27:53 -0800] Done curl exec [162 Sat, 01 Feb 2020 13:27:53 -0800] Set curl options for single page request [161 Sat, 01 Feb 2020 13:27:53 -0800] Init curl request of a single page [160 Sat, 01 Feb 2020 13:27:52 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592472&session=3e69ee488f8cdfcf68aed743ca475abb& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=0 [159 Sat, 01 Feb 2020 13:27:52 -0800] http://localhost:8080/ to see if should start crawling [158 Sat, 01 Feb 2020 13:27:52 -0800] Checking name server: [157 Sat, 01 Feb 2020 13:27:42 -0800] No data. Sleeping... [156 Sat, 01 Feb 2020 13:27:42 -0800] Percent idle time per hour: 100.00% [155 Sat, 01 Feb 2020 13:27:42 -0800] MAIN LOOP CASE 1 -- SWITCH CRAWL OR NO CURRENT CRAWL [154 Sat, 01 Feb 2020 13:27:42 -0800] Crawl time has changed! [153 Sat, 01 Feb 2020 13:27:42 -0800] End Name Server Check [152 Sat, 01 Feb 2020 13:27:42 -0800] Minimum fetch loop time is: 5 seconds [151 Sat, 01 Feb 2020 13:27:42 -0800] Crawl Time Changing to 0 -- No Crawl [150 Sat, 01 Feb 2020 13:27:42 -0800] Start of response was:a:6:{s:1:"a";i:1;s:1:"b";i:0;s:2:"dv";s:4484:"YTo3OntpOj A7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoiTkFNRSI7czo2OiJEUlVQQU wiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5OiJTSUdOQVRVUkUiO3M6MT I1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhyZWYsICcvc2l0ZXMvYWxsL3 RoZW1lcycpIG9yIGNvbn [149 Sat, 01 Feb 2020 13:27:42 -0800] Done curl exec [148 Sat, 01 Feb 2020 13:27:42 -0800] Set curl options for single page request [147 Sat, 01 Feb 2020 13:27:42 -0800] Init curl request of a single page [146 Sat, 01 Feb 2020 13:27:42 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592462&session=8f383b3a6214f0a1383dec2f134128ac& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=0 [145 Sat, 01 Feb 2020 13:27:42 -0800] http://localhost:8080/ to see if should start crawling [144 Sat, 01 Feb 2020 13:27:42 -0800] Checking name server: [143 Sat, 01 Feb 2020 13:27:32 -0800] No data. Sleeping... [142 Sat, 01 Feb 2020 13:27:32 -0800] Percent idle time per hour: 100.00% [141 Sat, 01 Feb 2020 13:27:32 -0800] MAIN LOOP CASE 1 -- SWITCH CRAWL OR NO CURRENT CRAWL [140 Sat, 01 Feb 2020 13:27:32 -0800] Crawl time has changed! [139 Sat, 01 Feb 2020 13:27:32 -0800] End Name Server Check [138 Sat, 01 Feb 2020 13:27:32 -0800] Minimum fetch loop time is: 5 seconds [137 Sat, 01 Feb 2020 13:27:32 -0800] Crawl Time Changing to 0 -- No Crawl [136 Sat, 01 Feb 2020 13:27:32 -0800] Start of response was:a:6:{s:1:"a";i:1;s:1:"b";i:0;s:2:"dv";s:4484:"YTo3OntpOj A7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoiTkFNRSI7czo2OiJEUlVQQU wiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5OiJTSUdOQVRVUkUiO3M6MT I1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhyZWYsICcvc2l0ZXMvYWxsL3 RoZW1lcycpIG9yIGNvbn [135 Sat, 01 Feb 2020 13:27:32 -0800] Done curl exec [134 Sat, 01 Feb 2020 13:27:32 -0800] Set curl options for single page request [133 Sat, 01 Feb 2020 13:27:32 -0800] Init curl request of a single page [132 Sat, 01 Feb 2020 13:27:32 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592452&session=43060c574d58bae3d7127a7cc3457bf4& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=0 [131 Sat, 01 Feb 2020 13:27:32 -0800] http://localhost:8080/ to see if should start crawling [130 Sat, 01 Feb 2020 13:27:32 -0800] Checking name server: [129 Sat, 01 Feb 2020 13:27:22 -0800] No data. Sleeping... [128 Sat, 01 Feb 2020 13:27:22 -0800] Percent idle time per hour: 100.00% [127 Sat, 01 Feb 2020 13:27:22 -0800] MAIN LOOP CASE 1 -- SWITCH CRAWL OR NO CURRENT CRAWL [126 Sat, 01 Feb 2020 13:27:22 -0800] Crawl time has changed! [125 Sat, 01 Feb 2020 13:27:22 -0800] End Name Server Check [124 Sat, 01 Feb 2020 13:27:22 -0800] Minimum fetch loop time is: 5 seconds [123 Sat, 01 Feb 2020 13:27:22 -0800] Crawl Time Changing to 0 -- No Crawl [122 Sat, 01 Feb 2020 13:27:22 -0800] Start of response was:a:6:{s:1:"a";i:1;s:1:"b";i:0;s:2:"dv";s:4484:"YTo3OntpOj A7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoiTkFNRSI7czo2OiJEUlVQQU wiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5OiJTSUdOQVRVUkUiO3M6MT I1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhyZWYsICcvc2l0ZXMvYWxsL3 RoZW1lcycpIG9yIGNvbn [121 Sat, 01 Feb 2020 13:27:22 -0800] Done curl exec [120 Sat, 01 Feb 2020 13:27:22 -0800] Set curl options for single page request [119 Sat, 01 Feb 2020 13:27:22 -0800] Init curl request of a single page [118 Sat, 01 Feb 2020 13:27:22 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592442&session=80138ccbe49bfb74b9a4a0834b7e56c0& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=0 [117 Sat, 01 Feb 2020 13:27:22 -0800] http://localhost:8080/ to see if should start crawling [116 Sat, 01 Feb 2020 13:27:22 -0800] Checking name server: [115 Sat, 01 Feb 2020 13:27:12 -0800] No data. Sleeping... [114 Sat, 01 Feb 2020 13:27:12 -0800] Percent idle time per hour: 100.00% [113 Sat, 01 Feb 2020 13:27:12 -0800] MAIN LOOP CASE 1 -- SWITCH CRAWL OR NO CURRENT CRAWL [112 Sat, 01 Feb 2020 13:27:12 -0800] Crawl time has changed! [111 Sat, 01 Feb 2020 13:27:12 -0800] End Name Server Check [110 Sat, 01 Feb 2020 13:27:12 -0800] Minimum fetch loop time is: 5 seconds [109 Sat, 01 Feb 2020 13:27:12 -0800] Crawl Time Changing to 0 -- No Crawl [108 Sat, 01 Feb 2020 13:27:12 -0800] Start of response was:a:6:{s:1:"a";i:1;s:1:"b";i:0;s:2:"dv";s:4484:"YTo3OntpOj A7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoiTkFNRSI7czo2OiJEUlVQQU wiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5OiJTSUdOQVRVUkUiO3M6MT I1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhyZWYsICcvc2l0ZXMvYWxsL3 RoZW1lcycpIG9yIGNvbn [107 Sat, 01 Feb 2020 13:27:12 -0800] Done curl exec [106 Sat, 01 Feb 2020 13:27:12 -0800] Set curl options for single page request [105 Sat, 01 Feb 2020 13:27:12 -0800] Init curl request of a single page [104 Sat, 01 Feb 2020 13:27:12 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592432&session=4e9e27fbf6684ba5493ad9bb5aad95ad& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=0 [103 Sat, 01 Feb 2020 13:27:12 -0800] http://localhost:8080/ to see if should start crawling [102 Sat, 01 Feb 2020 13:27:12 -0800] Checking name server: [101 Sat, 01 Feb 2020 13:27:02 -0800] No data. Sleeping... [100 Sat, 01 Feb 2020 13:27:02 -0800] Percent idle time per hour: 100.00% [99 Sat, 01 Feb 2020 13:27:02 -0800] MAIN LOOP CASE 1 -- SWITCH CRAWL OR NO CURRENT CRAWL [98 Sat, 01 Feb 2020 13:27:02 -0800] Crawl time has changed! [97 Sat, 01 Feb 2020 13:27:02 -0800] End Name Server Check [96 Sat, 01 Feb 2020 13:27:02 -0800] Minimum fetch loop time is: 5 seconds [95 Sat, 01 Feb 2020 13:27:02 -0800] Crawl Time Changing to 0 -- No Crawl [94 Sat, 01 Feb 2020 13:27:02 -0800] Start of response was:a:6:{s:1:"a";i:1;s:1:"b";i:0;s:2:"dv";s:4484:"YTo3OntpOj A7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoiTkFNRSI7czo2OiJEUlVQQU wiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5OiJTSUdOQVRVUkUiO3M6MT I1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhyZWYsICcvc2l0ZXMvYWxsL3 RoZW1lcycpIG9yIGNvbn [93 Sat, 01 Feb 2020 13:27:02 -0800] Done curl exec [92 Sat, 01 Feb 2020 13:27:02 -0800] Set curl options for single page request [91 Sat, 01 Feb 2020 13:27:02 -0800] Init curl request of a single page [90 Sat, 01 Feb 2020 13:27:02 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592422&session=1d249ffe9cebb664ee4c00f30637a707& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=0 [89 Sat, 01 Feb 2020 13:27:02 -0800] http://localhost:8080/ to see if should start crawling [88 Sat, 01 Feb 2020 13:27:02 -0800] Checking name server: [87 Sat, 01 Feb 2020 13:26:52 -0800] No data. Sleeping... [86 Sat, 01 Feb 2020 13:26:52 -0800] Percent idle time per hour: 100.00% [85 Sat, 01 Feb 2020 13:26:52 -0800] MAIN LOOP CASE 1 -- SWITCH CRAWL OR NO CURRENT CRAWL [84 Sat, 01 Feb 2020 13:26:52 -0800] Crawl time has changed! [83 Sat, 01 Feb 2020 13:26:52 -0800] End Name Server Check [82 Sat, 01 Feb 2020 13:26:52 -0800] Minimum fetch loop time is: 5 seconds [81 Sat, 01 Feb 2020 13:26:52 -0800] Crawl Time Changing to 0 -- No Crawl [80 Sat, 01 Feb 2020 13:26:52 -0800] Start of response was:a:6:{s:1:"a";i:1;s:1:"b";i:0;s:2:"dv";s:4484:"YTo3OntpOj A7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoiTkFNRSI7czo2OiJEUlVQQU wiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5OiJTSUdOQVRVUkUiO3M6MT I1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhyZWYsICcvc2l0ZXMvYWxsL3 RoZW1lcycpIG9yIGNvbn [79 Sat, 01 Feb 2020 13:26:52 -0800] Done curl exec [78 Sat, 01 Feb 2020 13:26:52 -0800] Set curl options for single page request [77 Sat, 01 Feb 2020 13:26:52 -0800] Init curl request of a single page [76 Sat, 01 Feb 2020 13:26:52 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592412&session=70808a379eea2ba295a525374a38780e& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=0 [75 Sat, 01 Feb 2020 13:26:52 -0800] http://localhost:8080/ to see if should start crawling [74 Sat, 01 Feb 2020 13:26:52 -0800] Checking name server: [73 Sat, 01 Feb 2020 13:26:42 -0800] No data. Sleeping... [72 Sat, 01 Feb 2020 13:26:42 -0800] Percent idle time per hour: 100.00% [71 Sat, 01 Feb 2020 13:26:42 -0800] MAIN LOOP CASE 1 -- SWITCH CRAWL OR NO CURRENT CRAWL [70 Sat, 01 Feb 2020 13:26:42 -0800] Crawl time has changed! [69 Sat, 01 Feb 2020 13:26:42 -0800] End Name Server Check [68 Sat, 01 Feb 2020 13:26:42 -0800] Minimum fetch loop time is: 5 seconds [67 Sat, 01 Feb 2020 13:26:42 -0800] Crawl Time Changing to 0 -- No Crawl [66 Sat, 01 Feb 2020 13:26:42 -0800] Start of response was:a:6:{s:1:"a";i:1;s:1:"b";i:0;s:2:"dv";s:4484:"YTo3OntpOj A7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoiTkFNRSI7czo2OiJEUlVQQU wiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5OiJTSUdOQVRVUkUiO3M6MT I1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhyZWYsICcvc2l0ZXMvYWxsL3 RoZW1lcycpIG9yIGNvbn [65 Sat, 01 Feb 2020 13:26:42 -0800] Done curl exec [64 Sat, 01 Feb 2020 13:26:42 -0800] Set curl options for single page request [63 Sat, 01 Feb 2020 13:26:42 -0800] Init curl request of a single page [62 Sat, 01 Feb 2020 13:26:42 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592402&session=16309361390339cf64ff4d5913b97b16& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=0 [61 Sat, 01 Feb 2020 13:26:42 -0800] http://localhost:8080/ to see if should start crawling [60 Sat, 01 Feb 2020 13:26:42 -0800] Checking name server: [59 Sat, 01 Feb 2020 13:26:32 -0800] No data. Sleeping... [58 Sat, 01 Feb 2020 13:26:32 -0800] Percent idle time per hour: 100.00% [57 Sat, 01 Feb 2020 13:26:32 -0800] MAIN LOOP CASE 1 -- SWITCH CRAWL OR NO CURRENT CRAWL [56 Sat, 01 Feb 2020 13:26:32 -0800] Crawl time has changed! [55 Sat, 01 Feb 2020 13:26:32 -0800] End Name Server Check [54 Sat, 01 Feb 2020 13:26:32 -0800] Minimum fetch loop time is: 5 seconds [53 Sat, 01 Feb 2020 13:26:32 -0800] Crawl Time Changing to 0 -- No Crawl [52 Sat, 01 Feb 2020 13:26:32 -0800] Start of response was:a:6:{s:1:"a";i:1;s:1:"b";i:0;s:2:"dv";s:4484:"YTo3OntpOj A7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoiTkFNRSI7czo2OiJEUlVQQU wiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5OiJTSUdOQVRVUkUiO3M6MT I1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhyZWYsICcvc2l0ZXMvYWxsL3 RoZW1lcycpIG9yIGNvbn [51 Sat, 01 Feb 2020 13:26:32 -0800] Done curl exec [50 Sat, 01 Feb 2020 13:26:32 -0800] Set curl options for single page request [49 Sat, 01 Feb 2020 13:26:32 -0800] Init curl request of a single page [48 Sat, 01 Feb 2020 13:26:32 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592392&session=357d25448c59d4fb6bd83b84fa87112b& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=0 [47 Sat, 01 Feb 2020 13:26:32 -0800] http://localhost:8080/ to see if should start crawling [46 Sat, 01 Feb 2020 13:26:32 -0800] Checking name server: [45 Sat, 01 Feb 2020 13:26:22 -0800] No data. Sleeping... [44 Sat, 01 Feb 2020 13:26:22 -0800] Percent idle time per hour: 100.00% [43 Sat, 01 Feb 2020 13:26:22 -0800] MAIN LOOP CASE 1 -- SWITCH CRAWL OR NO CURRENT CRAWL [42 Sat, 01 Feb 2020 13:26:22 -0800] Crawl time has changed! [41 Sat, 01 Feb 2020 13:26:22 -0800] End Name Server Check [40 Sat, 01 Feb 2020 13:26:22 -0800] Minimum fetch loop time is: 5 seconds [39 Sat, 01 Feb 2020 13:26:22 -0800] Crawl Time Changing to 0 -- No Crawl [38 Sat, 01 Feb 2020 13:26:22 -0800] Start of response was:a:6:{s:1:"a";i:1;s:1:"b";i:0;s:2:"dv";s:4484:"YTo3OntpOj A7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoiTkFNRSI7czo2OiJEUlVQQU wiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5OiJTSUdOQVRVUkUiO3M6MT I1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhyZWYsICcvc2l0ZXMvYWxsL3 RoZW1lcycpIG9yIGNvbn [37 Sat, 01 Feb 2020 13:26:22 -0800] Done curl exec [36 Sat, 01 Feb 2020 13:26:22 -0800] Set curl options for single page request [35 Sat, 01 Feb 2020 13:26:22 -0800] Init curl request of a single page [34 Sat, 01 Feb 2020 13:26:22 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592382&session=3c61b94957bac8d4dc7aa635f290bd8b& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=0 [33 Sat, 01 Feb 2020 13:26:22 -0800] http://localhost:8080/ to see if should start crawling [32 Sat, 01 Feb 2020 13:26:22 -0800] Checking name server: [31 Sat, 01 Feb 2020 13:26:12 -0800] No data. Sleeping... [30 Sat, 01 Feb 2020 13:26:12 -0800] Percent idle time per hour: 100.00% [29 Sat, 01 Feb 2020 13:26:12 -0800] MAIN LOOP CASE 1 -- SWITCH CRAWL OR NO CURRENT CRAWL [28 Sat, 01 Feb 2020 13:26:12 -0800] Crawl time has changed! [27 Sat, 01 Feb 2020 13:26:12 -0800] End Name Server Check [26 Sat, 01 Feb 2020 13:26:12 -0800] Minimum fetch loop time is: 5 seconds [25 Sat, 01 Feb 2020 13:26:12 -0800] Crawl Time Changing to 0 -- No Crawl [24 Sat, 01 Feb 2020 13:26:12 -0800] Start of response was:a:6:{s:1:"a";i:1;s:1:"b";i:0;s:2:"dv";s:4484:"YTo3OntpOj A7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoiTkFNRSI7czo2OiJEUlVQQU wiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5OiJTSUdOQVRVUkUiO3M6MT I1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhyZWYsICcvc2l0ZXMvYWxsL3 RoZW1lcycpIG9yIGNvbn [23 Sat, 01 Feb 2020 13:26:12 -0800] Done curl exec [22 Sat, 01 Feb 2020 13:26:12 -0800] Set curl options for single page request [21 Sat, 01 Feb 2020 13:26:12 -0800] Init curl request of a single page [20 Sat, 01 Feb 2020 13:26:12 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592372&session=7b734cd14923a10a7b35697892d4c771& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=0 [19 Sat, 01 Feb 2020 13:26:12 -0800] http://localhost:8080/ to see if should start crawling [18 Sat, 01 Feb 2020 13:26:12 -0800] Checking name server: [17 Sat, 01 Feb 2020 13:26:02 -0800] No data. Sleeping... [16 Sat, 01 Feb 2020 13:26:02 -0800] Percent idle time per hour: 0.00% [15 Sat, 01 Feb 2020 13:26:02 -0800] MAIN LOOP CASE 1 -- SWITCH CRAWL OR NO CURRENT CRAWL [14 Sat, 01 Feb 2020 13:26:02 -0800] Crawl time has changed! [13 Sat, 01 Feb 2020 13:26:02 -0800] End Name Server Check [12 Sat, 01 Feb 2020 13:26:02 -0800] Minimum fetch loop time is: 5 seconds [11 Sat, 01 Feb 2020 13:26:02 -0800] Crawl Time Changing to 0 -- No Crawl [10 Sat, 01 Feb 2020 13:26:02 -0800] Start of response was:a:6:{s:1:"a";i:1;s:1:"b";i:0;s:2:"dv";s:4484:"YTo3OntpOj A7YTo3OntzOjI6IklEIjtzOjE6IjEiO3M6NDoiTkFNRSI7czo2OiJEUlVQQU wiO3M6ODoiUFJJT1JJVFkiO3M6MToiMCI7czo5OiJTSUdOQVRVUkUiO3M6MT I1OiIvaHRtbC9oZWFkLypbY29udGFpbnMoQGhyZWYsICcvc2l0ZXMvYWxsL3 RoZW1lcycpIG9yIGNvbn [9 Sat, 01 Feb 2020 13:26:02 -0800] Done curl exec [8 Sat, 01 Feb 2020 13:26:02 -0800] Set curl options for single page request [7 Sat, 01 Feb 2020 13:26:02 -0800] Init curl request of a single page [6 Sat, 01 Feb 2020 13:26:02 -0800] Request was:http://localhost:8080/?c=fetch&a=crawlTime&time= 1580592362&session=01e4671272f5d0226fe71992f6cdc4c4& robot_instance=0-0-localhost-1580591967&machine_uri=/&am p;crawl_time=0 [5 Sat, 01 Feb 2020 13:26:02 -0800] http://localhost:8080/ to see if should start crawling [4 Sat, 01 Feb 2020 13:26:02 -0800] Checking name server: [3 Sat, 01 Feb 2020 13:26:02 -0800] PHP Version in use: 7.3.0 [2 Sat, 01 Feb 2020 13:26:02 -0800] In Fetch Loop [1 Sat, 01 Feb 2020 13:26:02 -0800] 0-Fetcher using channel 0 Initialize logger.. [0 Sat, 01 Feb 2020 13:26:02 -0800]my6.5.6.patch
mypos1.patch
term_weight.txt.gz
mypos.patch
mypos1.1.patch
myposner1.patch
myposner1-2.patch
qa1.patch