In attachment, provided lexicon.txt - used by part of speech tagging.

Added path for wordnet feature

Steps to reproduce: Here are the steps for WordNet feature:

1. Download WordNet 2.1 from https://wordnet.princeton.edu/wordnet/download/current-version/ 2. Install it in your system. 3. Open Yioop! console and go to Page Option. 4. Click on WordNet and add WordNet Directory path for example - "C:\WordNet" 5. Click on Save.

Suggestions given by Prof. Pollett in meeting 1.Change variable names as per coding guidelines 2.Braces for classes and function will be vertically aligned. 3.Generate the patch without including lexicon.txt which is used for part of speech tagging. Send that file over the mail. 4.Show "WordNet" only except "WordNet Feature"

I made the changes and generated patch is provided in attachment. Lexicon.txt is sent over the mail.

Hey Shailesh, Don't delete old patches. I want a record of all the patches you submit. The patch size seems kinda huge. Can you try to make a smaller patch? It shouldn't be making so many changes to locales. Maybe you need to do a git pull before making the patch.

PartofSpeechTagger --> PartOfSpeechTagger

Also, don't use abbreviations like calc or Freq in function or variable names.

Best, Chris

Hello Professor,

As per our discussion, I created a new patch without making any changes in locale. Now the patch size seems smaller.

Changes for Locale are as follows: pageoption_element_wordnet_feature = "WordNet" pageoption_element_wordnet_dir = "WordNet Directory" search_view_wordnet_results = "WordNet Results" search_view_bm25score = "WordNet %s"

We need to add lexicon.txt which i sent you by an email previously.

Thanks, Shailesh

Hello Professor,

I made changes on new Yioop after pushing Centroid Summarization changes by Mangesh. Since I didn't find PHP_OS global variable, I used PHP_OS in my implementation.

Changes for Locale are as follows: pageoption_element_wordnet_feature = "WordNet" pageoption_element_wordnet_dir = "WordNet Directory" search_view_wordnet_results = "WordNet Results" search_view_bm25score = "WordNet %s"

We need to add lexicon.txt which i sent you by an email previously. I tested my patch on cloned yioop and its working fine.

Thanks, Shailesh

Hello Professor,

Please find the updated patch for WordNet. I generated the new patch as new changes to Summarization made some conflicts while applying my patch.

Changes for Locale are as follows: pageoption_element_wordnet_feature = "WordNet" pageoption_element_wordnet_dir = "WordNet Directory" search_view_wordnet_results = "WordNet Results" search_view_bm25score = "WordNet %s"

We need to add lexicon.txt which i sent you by an email previously. I tested my patch on cloned yioop and its working fine.

Thanks Shailesh

Hi Shailesh,

From the wordnet site itself it looks like you can download tab delimited tables of the wordnet data. Is it possible just to import these into a database and have a php script query it to avoid having to do shell_exec,parse, etc?

Chris

Hello Professor,

According to my understanding, after studying WordNet documentation and their database structure, they provided the information for database files, but the extraction process is using lot of different functions from different files which we need to implement in php (Currently they are implemented in C).

Thanks Shailesh

Hi Shailesh,

Looking at the wordnet tables they don't seem that complicated. Your code as it stands won't handle more than a couple users at a time because you have to write temporary files on every query and you use the same file name when you do it. It would be way better to just import the tables into a database and query that. You also can't write files to the BASE_DIR in general as you should assume that is read only.

Chris

Hello Professor,

I made new patch so that we don't need to use temporary output file and now code can handle queries from a couple of users at a time.

Thanks Shailesh

Hello Professor,

Please find the updated patch for WordNet.

Suggested changes : 1. To check for proper indentation in new files. 2. Rename WORDNET_DIR to WORDNET_EXEC. 3. Change filename to remove capital letters from them.

Changes for Locale are as follows: pageoption_element_wordnet_feature = "WordNet" pageoption_element_wordnet_dir = "WordNet Directory" search_view_wordnet_results = "WordNet Results" search_view_bm25score = "WordNet %s"

I tested my patch on cloned yioop and its working fine.

Thank You, Shailesh Padave

Part of speech tagger moved to en-US locale along with lexicon.txt file in my7.patch

Hi Shailesh,

I still could not get this to work stably so I am backing it out on my local machine (trying to get exec's to work in a cross platform way is very hard). For your presentation please just use your local copy. There are still a lot of issues in your code as far as naming of variables, spacing, long lines etc. Besides using camel-case for variable names rather than underscore, you should avoid typing info in names. For example, $terms rather than $query_array. You should further really avoid abbreviations in variable names. The way the part of speech tagger was require'd still was not flexible with regard to making PoS taggers for other languages "en-US" is hard-coded rather than trying to compute things dynamically. You are reading a quite large file "lexicon.txt" and I was worried on the impact of this on query performance. You seem to be repeatedly computing explode(" ", $query) in methods which call each other. This and similar code which repeats computations you have already done I also suspect will tend to slow things down.

I am going to try to release a new version of Yioop in the next few days. Let's try for a new patch after that since I don't want to add something last minute that I can't immediately figure out how to do quick fixes on.

Chris

Hello Professor,

I will make the necessary changes and i will upload the patch by tonight.

Thanks Shailesh

Hello Professor,

As per your suggestions, I made the changes in my files about variable names. Part of speech tagging code will generate a path for locale files dynamically.

I removed all the Notices and warnings. Kindly review my patch and let me know. I removed the repetition of code for explode(" ",$query).

Thank You Shailesh Padave

Hello Professor,

As per your suggestions, I made the changes in my files.

Changes are as follows: 1. For Linux and Mac system, ask user to add path for wordnet exec dir till bin folder. 2. Changes in variable name

count_non_zero_score to num_non_zero_score
results_wordnetscore to results_wordnet_score

Changes for Locale are as follows: pageoption_element_wordnet_feature = "WordNet" pageoption_element_wordnet_dir = "WordNet Directory" search_view_wordnet_results = "WordNet Results" search_view_bm25score = "WordNet %s"

I tested my patch on cloned yioop and its working fine. We need to add the lexicon.txt file in locale of en-US

Thank You Shailesh

Hello Professor,

I made new patch and made following changes: 1. Changed name of BM25ScoreCalculator to WordNetScoreCalculator. 2. Modified heading tag for WordNet Directory label.

Changes for Locale are as follows: pageoption_element_wordnet_feature = "WordNet" pageoption_element_wordnet_dir = "WordNet Directory" search_view_wordnet_results = "WordNet Results" search_view_bm25score = "WordNet %s"

I tested my patch on cloned yioop and its working fine.

Thanks Shailesh

Hey Shailesh,

I really like the idea you had in my office of writing a script to invoke wordnet to export all the data to two DB tables, and then, from Yioop only use those two tables to do the WordNet stuff. I'd really prefer to apply that patch.

Best, Chris

Hello Professor,

I will work on those scripts for creating tables and insert the required data in database. But I am confused about for which database should i create those scripts for create and insert. I am working on the scripts to extract required data from wordnet. I will send you the script as soon as possible.

Thanks Shailesh Padave

I don't understand your question. It should be for the default database for yioop. All queries, etc should go through yioop current database layer.

Best, Chris

Okay Professor. I will take a look at queries which are used in yioop previously.

Thanks Shailesh Padave

Hello Professor,

The idea I gave you will only work for basic forms of words like "computer, run", if user provides any query like "running" then wordnet is doing some other processing to make it to base form and then using it. So we could not do query expansion on words except than base forms.

So is it okay to go with executable of wordnet.

Thanks Shailesh

As you build your table using wordnet store the stemmed term into your table rather than the word. For example, look up lazy in wordnet, but stem and store lazi when you put it in your table. Then when you look up laziness sometime later at query time you run the stemmer as we are already doing get lazi and should be able to find the correct row in the table.

When making your tables have a column for locale_tag as well. This will allow people to write importers to yioop for systems geared toward other languages as well.

Okay.

Hello Professor,

I said that I can upload new patch by end of the day but just now I uploaded all the data for verb, noun, adjective and adverb in my-sql database. Still the words from exception list are not uploaded in database(which will increase the database size). I tried to fetch the data from my-sql database for a single word and it took about 1.58 seconds, as the size of database is too large.

In the previous implementation, processing time was about 0.3 seconds. So I am worried that if new implementation with database takes 1.58 seconds just to grab the information then including further processing it may complete in 2 seconds. I tried with PostgresSql database as well. It is also taking same time. This will affect the efficiency.

I searched over internet for processing time for queries and they takes more time for fetching data, Whereas, pointer in C are more faster. WordNet implementation uses pointers in C and thats why its much more faster in previous implementation.

So is it ok to upload a new patch after our meeting by tomorrow?

Thanks Shailesh

Hello Professor,

I pulled new code and generated new patch containing following changes: 1. Changed name of BM25ScoreCalculator to WordNetScoreCalculator. 2. Modified heading tag for WordNet Directory label as per coding guidelines.

Changes for Locale are as follows: pageoption_element_wordnet_feature = "WordNet" pageoption_element_wordnet_dir = "WordNet Directory" search_view_wordnet_results = "WordNet Results" search_view_bm25score = "WordNet %s"

I tested my patch on cloned yioop and its working fine. We need to add lexicon.txt file in locale/en-US/resources folder.

Kindly revert in any case of concern.

Thanks Shailesh

Hey Shailesh,

I got it to work, so I am closing this ticket. I will push the code later today when I get a chance.

Best, Chris

That's great.

Thanks for your quick reply on patch.

Thanks, Shailesh

Hi Shailesh,

The following code never takes less than a second:

   function __construct($lexicon)
   {
       $fh = fopen($lexicon, 'r');
       $start_time = microtime();
       while($line = fgets($fh)) {
           $tags = explode(' ', $line);
           $this->dictionary[strtolower(array_shift($tags))] = $tags;
       }
       fclose($fh);
       echo changeInMicrotime($start_time);
   }

So the part of speech tagger is slow.

Ya professor. I tried to do some changes on the other part of code to make it more efficient. After that, i found tagger takes about 0.2 seconds for tagging on an average.

Thanks Shailesh

Professor, I could not find any way to make it more faster. Can you please share if there is any other way to make it more faster?

Thanks Shailesh

Hello Professor,

I just checked in my system and for any query, its taking 0.2 seconds. I would like to confirm once, is lexicon.txt file is there in locale/en-US/resources?

Also after doing git pull, I could not find lexicon.txt file in locale/en-US/resources. I think this may be the reason for part of speech tagging being slow.

Thanks Shailesh

Hi Shailesh,

Your getCosineRank function is nonsense. $x and $normA don't do anything. It also takes nonstandard arguments. So it was always returning 0. So any results you were getting based on it were not correct.

Chris

Hello Professor,

I tried to run this code separately and i got cosine similarity ranking correctly. The code is as follows:

<?php function cosinusTokens(array $setenceA, array $setenceB) {

   $dotProduct = $cnt_sentA = $cnt_sentB = 0;
   $uniqueTokensA = $uniqueTokensB = array();
   $uniqueMergedTokens = array_unique(array_merge($setenceA, $setenceB));

   foreach ($setenceA as $token) $uniqueTokensA[$token] = 0;
   foreach ($setenceB as $token) $uniqueTokensB[$token] = 0;

   foreach ($uniqueMergedTokens as $token) {
       $x = isset($uniqueTokensA[$token]) ? 1 : 0;
       $y = isset($uniqueTokensB[$token]) ? 1 : 0;
       $dotProduct += $x * $y;
       $cnt_sentA += $x;
       $cnt_sentB += $y;
   }

echo ($cnt_sentA * $cnt_sentB) != 0

       ? $dotProduct / sqrt($cnt_sentA * $cnt_sentB)
       : 0;
   return ($cnt_sentA * $cnt_sentB) != 0
       ? $dotProduct / sqrt($cnt_sentA * $cnt_sentB)
       : 0;

} cosinusTokens(array('this', 'is', 'your', 'home'), array('this', 'is', 'my', 'home')); ?>

Cosine similarity score - 0.75

Thanks Shailesh

Hello Professor,

I think wordnet does not have usage of similar words in sentences for the queries given by you. That may be the reason we are getting 0 as cosine score. I tested it with different queries which has sentences in wordnet output and the function gave me cosine score.

Thanks Shailesh

Hello Professor,

I just fixed the issue for cosine similarity ranking score. You was correct. $x and $normA was my mistake.

I forgot to rename them to resA and cnt_sentA.

Here is the updated function for Cosine ranking:

   static function getCosineRank(array $sentenceA, array $sentenceB)
   {
       $result = $cnt_sentA = $cnt_sentB = 0;
       $arrayA = $arrayB = array();
       $resA = $resB = 0;
       $mergedWords = array_unique(array_merge($sentenceA, $sentenceB));
       foreach ($sentenceA as $word) {
           $arrayA[$word] = 0;
       }
       foreach ($sentenceB as $word) {
           $arrayB[$word] = 0;
       }
       foreach ($mergedWords as $word) {
           $resA = isset($arrayA[$word]) ? 1 : 0;
           $resB = isset($arrayB[$word]) ? 1 : 0;
           $result += $resA * $resB;
           $cnt_sentA += $resA;
           $cnt_sentB += $resB;
       }
       $score = ($cnt_sentA * $cnt_sentB) != 0 ?
                   $result / sqrt($cnt_sentA * $cnt_sentB) : 0;
       return $score;
   }

Sorry for the trouble.

Thank You Shailesh

Hello Professor,

In my15.patch, I did modification to cosine similarity ranking function.

Kindly review it.

Thanks Shailesh

Fixed with f35a262. Code simplified and re-organized so as to potentially be used for other languages.

Attachments:

my.patch

my1.patch

my2.patch

my3.patch

my4.patch

my5.patch

my7.patch

my8.patch

my9.patch

my11.patch

my12.patch

my14.patch

my15.patch