Added sender/receiver windows to track sent schedules and incoming web page data. This ensures that data posted from fetchers is only uploaded when previous schedules have been received, to solve the issue of out-of-order indexing. Full documentation: http://www.cs.sjsu.edu/faculty/pollett/masters/Semesters/Spring23/gargi/index.php?SlidingWindow.html

slidingwindow.patch

Hi Gargi,

Thanks for the patch! Going through the patch I notice a fair number of places where the line length is over 80. Can you fix these? Also,

+ null, $num_fetchers, $_REQUEST["active_fetchers"]);

You are using uncleaned form data which can be a security hole. Can you clean it before using it? Also,

$new_recv_object = new L\ReceiverMessage($seq_num, time(),

+ $diff_origin, $filename); I would avoid variable names using new or object and also don't use abbreviations like recv. For control structures like: + for ($i=0;$i<count($window_info['RECEIVER']);$i++) { make sure to follow the coding guidelines: + for ($i = 0; $i < count($window_info['RECEIVER']); $i++) { I would also tend to make a variable for count($window_info['RECEIVER']) to avoid doing a dereference each time through the loop -- I know it doesn't save much time, but it feels cleaner to me.

Can you make a new patch fixing these minor issues?

Best, Chris

Thanks so much for your suggestions Professor! I've updated the patch, hope this is better!

Thanks, Gargi

sw.patch

Hi, Gargi, I applied your patch and pushed it after some minor modifications to make it more match Yioop coding guidelines (do a git pull and diff to see what I changed). Did you test your code to make sure it works correctly when Yioop is being run under apache (which is the usual mode) rather than using it's own web server? I was a little bit confused by the code below and its effects if Yioop was run under Apache. Can you maybe clarify what it would do?

Best, Chris

           if ($name == 'QueueServer') {
               CrawlDaemon::saveWindowData();
           } else if ($name == 'index') {
               $global_stop_file = C\WORK_DIRECTORY .
                   "/data/global_stop.txt";
               file_put_contents($global_stop_file, time());
               CrawlDaemon::saveWindowData();
               chmod($global_stop_file, 0777);
               exit();
           }

Hi Professor,

That snippet of code is meant to handle the situation wherein either the queue server is turned off or we stop the running instance (php index.php stop) without gracefully stopping the crawl. I added the function saveWindowData() to CrawlDaemon.php which would shift the SlidingWindow.txt, next_seq_num.txt (where the QueueServer keeps writing the next sequence number to be assigned to a fetch schedule), and any created FetchSchedule.txt files to cache/WindowData<crawl_timestamp>. So I added a call to saveWindowData() in the stop() function in CrawlDaemon, in case either the QueueServer or index file were stopped.

I apologize, I faced some issues setting up apache on my end, and so I was under the assumption that what would work on yioop as its own server would also be applicable for apache and didn't test it completely.

Best, Gargi

Okay. Maybe just test it under Apache now, and post a new patch (for just that part as I've already applied your most recent patch) if there is an issue.

These changes were incorporated in version 9.4. Thanks for the patch!