I have another childhood dream to report. I have always wanted a Graph Based Summarizer on the Yioop search engine. It would be nice if the Yioop search engine had one.

Steps to reproduce: Click Page Options and select the Summarizer dropdown. Graph Based is not in the list of languages to choose from.

I have added the my.patch file that contains the code for the Graph Based Summarizer and the modifications to incorporate it into the search engine.

Hey Charles,

I am impressed by how quickly you have got this close to done. Below are some comments on the first iteration of your patch. Address them and make a new patch. It probably take one or two iterations before we get to something I'll apply on the live site.

(1) I see a couple of places with stray tab and white space issues. For example, line 56 of the patch has a tab, line 256, 412, etc have an extra space at the end of the line..

(2) There is commented out code such as line 170 of the patch which should be removed.

(3) Function braces should be vertically aligned for getSummary, findLargestIndex, etc.

(4) Variables name should such as $termFrequencies should be renamed $term_frequencies and minor typos like $subtaction should be fixed to $subtraction.

(5) On the other hand, the method mult_mut_vec should be renamed multiplyMatrixVector.

(6) You might want to add the u modifier to some of your regular expressions for things like splitting sentences so they are Unicode friendly.

(7) The method getSentenceRanks is using a for loop 10 times to do the matrix multiply of the adjacency matrix against p. I think it would be better to try to do this with a while loop and try to test if the sum of square differences between the current p and the previous rounds is smaller than some small constant like say 0.01.

Hey Dr. Pollett,

Your compliment of being impressed means a lot to me.  I take pride in being able to juggle a full-time job, a wife and three kids plus school.  It is difficult but I manage to push on through.

Now for the code comments. I will take care of 1 - 6 no problem. I thought I caught all of the comments, tabs and braces but I guess not. I touched a lot of code debugging adding it to the search engine and writing the summarizer.

However, I was having issues that relate to 7 which is why I chose to use 10 iterations instead. I left the squared difference code in there just in case so it should not be hard to put back. I do not feel finding the optimal threshold will be so easy. No matter how low I set the threshold, 0.00001 for example, it would always converge after 1 or 2 iterations. The code in the paper used 10 iterations so I thought it would a be sufficient number, at least for a first try, rather than a random amount based on the squared difference. It could be because I am using a small dataset but I have am not sure. If it is absolutely important that I use the squared difference, I would like to spend more time evaluating the results with more data so I can enumerate an optimal threshold.

Let me know what you think and I will have the patch ready shortly after. Thanks

Hey Charles, The algorithm is O(n^2) not counting the outer while loop / for loop. The Page Rank and Beyond book that I suggested has some stuff on choices for the outer loop that seems to indicate that 10 is okay (p79). As 10 is a constant, the whole algorithm would be quadratic. I was trying to look up if there were math results on the rate of convergence of the algorithm besides what's known through experimentation, but couldn't find any in that book. So for now, just work on 1-6 and make a new patch.

Best, Chris

Thanks for your reply. Comments 1 - 6 are all addressed. I screwed up my repository and had to start over with a new clone of the repository. I did a fresh pull this morning and added my changes wholesale in my.patch2.

Hey Charles,

Getting back into the swing of school after break. I will try to look at your patch sometime Monday night. Grading till then.

Best, Chris

Hey Charles,

Okay, it passed my late at night read the code test. I will test it some more tomorrow and if it still seems fine will push it. I couldn't quite figure out why the patch had anything with word_filter_plugin.php in it? Also, since you create the graph_base_summarizer.php file why is Mangesh the @author?

Best, Chris

I am not sure why word_filter_plugin.php has changes. I did a new pull and that was not a file I changed. Fortunately, it really is not a change. It looks like the line is added and removed so there should not be problem with committing it.

Implemented with 68a2447f9. Thanks for your work!

Chris

Attachments:

my.patch

my.patch2