There are several techniques used by popular search engines to increase the speed and accuracy of results retrieved for a user query. One of such techniques is combining pair of words which always occur together in the same sequence. We refer to such pairs as Bigrams. For example “honda accord”, “hopton wafers”, “hp newwave” etc are bigrams. However if these words are not in the same sequence or separated by other words between them they act as independent words. Bigrams can be treated as single words while creating the inverted index for documents during the indexing phase. Similarly when the user query have these pair of words we treat them as single words to fetch documents relevant to the search. This technique speeds up the retrieval of documents and getting user desired results at the top. With this patch Yioop supports bigrams in the search query. We create a list of all pair of words which could qualify as bigrams as detailed in another patch titled "creating bigrams file". During the indexing phase the search engine checks the presence of all the bigrams in a given document by comparing each pair of consecutive words against the list of available bigrams. Then based on this comparison it creates an inverted index for both the bigrams as well as individual words in the posting list. During the query phase input query string is again checked against the list of bigrams. If bigram is found we pull documents corresponding to bigram using the inverted index.

Patch has many hard-coded magic numbers and strings. Also has dead code (commented out code). Please get me resubmit a clean patch

I have attached the new patch with several modifications. The process of creating the text file and filter file is more automated. The php file (create_bigrams_and_filter.php) to generate the text file and filter file is part of the patch. This file can only be run from command line as a tool and accepts the input xml file and language as command line arguments. The naming of text file and filter file is based solely on the input language. The english version is generated using "en" as language. All the files generated or required must be in the bigrams folder. I am sending you the already generated filter and text file for english language through email. If any changes or description is required please let me know.

Patch resubmitted. Modifications in this patch 1)const variables used to create paths in bigrams.php 2)bigrams folder removed as it made no sense creating a folder to store single bigrams class file. bigrams class file moved to lib. 3)Tool to create text and filter file moved to config. 4)text and filter files now generated inside search_filter folder of work directory 5)Code length reduced to comply with maximum column width 80

I took a stab at cleaning up your code. Please look at how I changed the variable and constant naming, added spaces after commas, etc. Your code is now part of trunk, however, I still don't think it is usable yet. In order for it to stay in trunk, you need to clearly describe in the source code how you can create a bigram filter with your code. That means say which wiki xml files to download, how they should be prepared, then what commands to run from the command line, and so on in as much detail as possible. You also need to add more descriptive comments before your methods rather than just repeat the name of the method in words. Your code still has this magic number 10000 hard-coded in it with no explanation. You need to phpdoc each constant. Ideally, your code should not expect the end user to do more than download a wiki dump of some sort and place it into the folder you specify. Do not make the end user have to uncompress things.

New Patch uploaded with changes suggested.

Implemented b1225795

Attachments:

0001-Bigrams-patch.patch

Bigram-support-new.patch