diff --git a/en-US/pages/about.thtml b/en-US/pages/about.thtml index e3e2c8f..c4a3293 100755 --- a/en-US/pages/about.thtml +++ b/en-US/pages/about.thtml @@ -54,17 +54,19 @@ with localization: Mary Pollett, Jonathan Ben-David, Thanh Bui, Sujata Dongre, Animesh Dutta, Youn Kim, Akshat Kukreti, Vijeth Patil, Chao-Hsin Shih, and Sugi Widjaja. Thanks to -Ravi Dhillon Tanmayee Potluri, Shawn Tice for creating patches for Yioop! -issues.Several of my master's students have done projects -related to Yioop!: Amith Chandranna, Priya Gangaraju, and Vijaya Pamidi. -Amith's code related to an Online version of the HITs algorithm +Ravi Dhillon Tanmayee Potluri, Shawn Tice, and Sandhya Vissapragada for +creating patches for Yioop! issues.Several of my master's students have done +projects related to Yioop!: Amith Chandranna, Priya Gangaraju, Vijaya Pamidi and +Vijaya Sinha. Amith's code related to an Online version of the HITs algorithm is not currently in the main branch of Yioop!, but it is obtainable from <a href="http://www.cs.sjsu.edu/faculty/pollett/masters/<?php ?>Semesters/Spring10/amith/index.shtml">Amith Chandranna's student page</a>. -Vijaya developed a Firefox web traffic extension for Yioop! +Vijaya Pamidi developed a Firefox web traffic extension for Yioop! Her code is also obtainable from <a href="http://www.cs.sjsu.edu/faculty/<?php ?>pollett/masters/Semesters/Fall10/vijaya/index.shtml">Vijaya Pamidi's -master's pages</a>. Priya's code served as the +master's pages</a>. <a href="http://www.cs.sjsu.edu/faculty/pollett/<?php +?>masters/Semesters/Spring11/amith/index.shtml">Vijaya Sinha's Project</a> +concerned using Open Street Map data in Yioop!. Priya's code served as the basis for the plugin feature currently in Yioop! The following other students have created text processors for Yioop!: Nakul Natu (pptx), Vijeth Patil (epub), and Tarun Pepira (xslx). diff --git a/en-US/pages/documentation.thtml b/en-US/pages/documentation.thtml index b8b2bde..e04b34d 100755 --- a/en-US/pages/documentation.thtml +++ b/en-US/pages/documentation.thtml @@ -1,5 +1,5 @@ <div class="docs"> -<h1>Yioop! Documentation v 0.84</h1> +<h1>Yioop! Documentation v 0.86</h1> <h2 id='toc'>Table of Contents</h2> <ul> <li><a href="#intro">Introduction</a></li> @@ -12,7 +12,7 @@ <li><a href="#crawls">Managing Crawls</a></li> <li><a href="#mixes">Mixing Crawl Indexes</a></li> <li><a href="#page-options">Options for Pages that are Indexed</a></li> - <li><a href="#filter">Search Filter</a></li> + <li><a href="#editor">Results Editor</a></li> <li><a href="#machines">GUI for Managing Machines and Servers</a></li> <li><a href="#localizing">Localizing Yioop! to a New Language</a></li> <li><a href="#framework">Building a Site using Yioop! as Framework</a> @@ -338,16 +338,20 @@ downloads of pages.</li> <li>It has a web interface to select seed sites for crawls and to set what sites crawls should not be crawled.</li> - <li>It obeys robots.txt file including the Crawl-delay and Sitemap - directives. It supports the robots meta tag.</li> + <li>It obeys robots.txt file including Google and Bing extensions such + as the Crawl-delay and Sitemap directives as well as * and $ in allow and + disallow. It further supports robots meta tag NONE, NOINDEX, NOFOLLOW, + NOARCHIVE, and NOSNIPPET and anchor tags with rel="nofollow" + attributes. It also supports X-Robots-Tag HTTP headers.</li> <li>Yioop! supports crawl quotas for web sites. i.e., one can control the number of urls/hour downloaded from a site.</li> <li>Yioop! can detect website congestion and slow down crawling a site that it detects as congested.</li> <li>It supports open web crawls, but through its web interface one can configure it also to crawl only specifics site, domains, or collections - of sites and domains. </li> - <li>IT supports dynamically changing the allowed and disallowed + of sites and domains. One can customize a crawl using regex in disallowed + directives to crawl a site to a fixed depth.</li> + <li>It supports dynamically changing the allowed and disallowed sites while a crawl is in progress.</li> <li>It supports dynamically injecting new seeds site via a web interface into the active crawl.</li> @@ -367,6 +371,8 @@ many fetchers.</li> <li>It determines search results using a number of iterators which can be combined like a simplified relational algebra.</li> + <li>Yioop can be configured to display word suggestion as a user + types a query.</li> <li>Since version 0.70, Yioop indexes are positional rather than bag of word indexes, and a index compression scheme called Modified9 is used.</li> @@ -392,6 +398,9 @@ <li>Using web archives, crawls can be mirrored amongst several machines to speed-up serving search results. This can be further sped-up by using memcache or filecache.</li> + <li>Yioop! supports the ability to filter out urls from search + results after a crawl has been performed. It also has the ability + to edit summary information that will be displayed for urls.</li> <li>A given Yioop! installation might have several saved crawls and it is very quick to switch between any of them and immediately start doing text searches.</li> @@ -590,7 +599,7 @@ section <a href="#embedding">Embedding Yioop!</a>, you can also look in the examples folder at the file search_api.php to see an example of how to use it. <b>If you intend to use Yioop! in a configuration with multiple queue servers (not fetchers), then -you the RSS check box needs to be checked.</b></p> +the RSS check box needs to be checked.</b></p> <p>The <b>Database Set-up</b> fieldset is used to specify what database management system should be used, how it should be connected to, and what user name and password should be used for the connection. At present sqlite2 @@ -611,10 +620,12 @@ which has privileges on all activities. Since different databases associated with a Yioop! installation might have different user accounts set-up after changing database information you might have to sign in again. </p> -<p>The <b>Search Auxiliary Links Displayed</b> fieldset is used to tell -you which links you would like to have presented on the search landing and -search results pages. The Signin checkbox controls whether to display the -link to the page for users to sign in to Yioop! The Cache checkbox toggles +<p>The <b>Search Page Elements and Links</b> fieldset is used to tell +you which element and links you would like to have presented on the search +landing and search results pages. The Word Suggest check box controls whether +a drop down of word suggestions should be presented by Yioop! when a user +starts typing in the Search box.The Signin checkbox controls whether to display +the link to the page for users to sign in to Yioop! The Cache checkbox toggles whether a link to the cache of a search item should be displayed as part of each search result. The Similar checkbox toggles whether a link to similar search items should be displayed as part @@ -732,13 +743,20 @@ config.php controls a number of parameters about how data is stored, how, and how often, the queue_server and fetchers communicate, and which file types are supported by Yioop! createdb.php can be used to create a bare instance of the Yioop! database with a root admin user having no password. This script is -not strictly necessary as the database should be creatable via the admin panel. -The file default_crawl.ini is copied to WORK_DIRECTORY after you set this -folder in the admin/configure panel. There it is renamed as crawl.ini and -serves as the initial set of sites to crawl until you decide to change these. -bigram_builder.php is a tool which can be used to extract from the urls in a -Wikipedia dump for a language two word pairs that should be treated as -a logical unit during a crawl. +not strictly necessary as the database should be creatable via the admin panel; +however, it can be useful if the database isn't working for some reason. +Also, in the configs folder is the file default_crawl.ini. This file is +copied to WORK_DIRECTORY after you set this folder in the admin/configure panel. +There it is renamed as crawl.ini and serves as the initial set of sites to crawl +until you decide to change these. The file token_tool.php is a tool which can +be used to help in term extraction during crawls and for making trie's +which can be used for word suggestions for a locale. To help word extraction +this tool can generate in a locale folder (see below) a word gram bloom filter. +Word grams are sequences of words that should be treated as a unit, for example, +Honda Accord. token_tool.php can use either a raw Wikipedia page count dump +file, or an actual Wikipedia dump file to extract from titles or redirects +these word grams. For trie construction this tool can use a file that lists +words one on a line. </dd> <dt>controllers</dt><dd>The controllers folder contains all the controller classes used by the web component of the Yioop! search engine. Most requests @@ -790,9 +808,18 @@ Gettext approach, but this could be easily mitigated using a memory cache such as <a href="http://memcached.org/">memcached</a> or <a href="http://php.net/manual/en/book.apc.php">apc</a>. Besides the file configure.ini, there is a statistics.txt file which has info about what -percentage of the id's have been translated. Finally, although not used in the -default Yioop! system, it is possible for a given locale folder to have a -sub-folder pages with translation of static pages used by a Yioop! installation. +percentage of the id's have been translated. In addition to configure.ini and +statistics.txt, the locale folder for a language contains two sub-folders: +pages, containing static html (with extension .thtml) files which might need +to be translated, and resources. The resources folder contains files: +suggest-trie.txt.gz, a <a href="http://en.wikipedia.org/wiki/Trie" +>Trie data structure</a> used for search bar word suggestions and tokenizer.php +which either specifies the number of characters for this language to +constitute a char gram or contains a stemmer class used to stem terms for +this language. This folder might also contain a Bloom filter file with a name +like all_word_grams.ftr which would be used to do word gramming of sequences of +words that should be treated as a unit, for example, "Honda Accord" or +"Bill Clinton". </dd> <dt>models</dt><dd>This folder contains the subclasses of Model used by Yioop! Models are used to encapsulate access to secondary storage. @@ -884,9 +911,15 @@ eventually be scheduled to crawl. Within each of these three kinds of folders there are typical many sub-folders, one for each day data arrived, and within these subfolders there are files containing the respective kinds of data.</dd> <dt>search_filters</dt><dd>This folder is used to store text files -containing global after crawl search filter data. The global search +containing global after crawl search filter and summary data. The global search filter allows a user to specify after a crawl is done that certain -urls be removed from the search results.</dd> +urls be removed from the search results. The global summary data +can be used to edit the summaries for a small number of web pages whose +summaries seem inaccurate or inappropriate. For example, some sites +like Facebook only allow big search engines like Google to crawl them. +Still there are many links to Facebook, so Facebook on an open web crawl +will appear, but with a somewhat confused summary based only on link text; +the results editor allows one to give a meaningful summary for Facebook.</dd> <dt>temp</dt><dd>This is used for storing temporary files that Yioop! creates during the crawl process. For example, temporary files used while making thumbnails.</dd> @@ -1400,7 +1433,13 @@ php fetcher.php start 5 </pre> <p>would all fall under this domain. A site can be specified as scheme://domain/path. For example, https://www.somewhere.com/foo/ . - Such a site includes https://www.somewhere.com/foo/anything_more .</p> + Such a site includes https://www.somewhere.com/foo/anything_more . + Yioop! also recognizes * and $ within urls. So http://my.site.com/*/*/ + would match http://my.site.com/subdir1/subdir2/rest and + http://my.site.com/*/*/$ would require the last symbold in the url + to be '/'. This kind of pattern matching can be useful in the + Allowed To Crawl Sites area to restrict the depth of a crawl to + within a url to a certain fixed depth.</p> <p>In the Disallowed Sites/Sites with Quotas, a number after a # sign indicates that at most that many pages should be downloaded from that site in any given hour. For example, @@ -1418,8 +1457,9 @@ php fetcher.php start 5 which wasn't in a downloaded document, but which is added to the inverted-index as if it had been in the document. The addition of user-defined meta-words is specified by giving a pattern matching rule - based on the url. For instance, in the figure above, the word column has - buyart and the url pattern column has: + based on the url. Unlike the sites field, for these fields we allow more + general regular expressions .For instance, in the figure above, the word + column has buyart and the url pattern column has: <pre> http://www.ucanbuyart.com/(.+)/(.+)/(.+)/(.+)/ </pre> @@ -1578,8 +1618,27 @@ OdpRdfArchiveBundle <p>It should be pointed out that the settings on this form (except the weight fields) only affect future crawls -- they do not affect any crawls that have already occurred or are on going.</p> - <h2 id='filter'>Search Filter</h2> - <p>The disallowed sites crawl option allows a user to specify they + <h2 id='editor'>Results Editor</h2> + <p>Sometimes after a large crawl one finds that there are some results + that appear that one does not want in the crawl or that the + summary for some result is lacking. The Result Editor activity allows + one to fix these issues without having to do a completely new crawl. + It has three main forms: An edited urls forms, a url editing form, + and a filter websites form.</p> + <p>If one has already edited the summary for + a url, then the drop-down in the edited urls form will list this url. One + can select it and click load to get it to display in the url editing + form. The purpose of the url editing form is to allow a user to change + the title and description for a url that appears on a search results + page. It does not affect whether the page is looked up for a given query, + only its final appearance. By filling out the three fields of the + url editing form, or by loading values into them through the previous form + and changing them, then clicking save updates the appearance of the summary + for that url. To return to using the default summary, one only fills + out the url field, leaves the other two blank, and saves. + </p> + <p>To understand the filter websites form, recall the disallowed sites + crawl option allows a user to specify they don't want Yioop! to crawl a given web site. After a crawl is done though one might be asked to removed a website from the crawl results, or one might want to remove a website from the crawl results because it @@ -1587,10 +1646,9 @@ OdpRdfArchiveBundle make the job of doing such filtering faster while one is waiting for a replacement crawl where the site has been disallowed, one can use a search filter.</p> -<img src='resources/SearchFilter.png' alt='The Search Filter form'/> - <p>Clicking on the Search Filter activity brings one to a screen - as above. Here one can specify a list of hosts which should be - excluded from the search results. The sites listed in the +<img src='resources/ResultsEditor.png' alt='The Results Editor form'/> + <p>Using the filter websites form one can specify a list of hosts which + should be excluded from the search results. The sites listed in the Sites to Filter text area are required to be hostnames. Using a filter, any web page with the same host name as one listed in the Sites to Filter will not appear in the search results. So for example, @@ -1713,7 +1771,7 @@ OdpRdfArchiveBundle <h3>Adding a stemmer or supporting character n-gramming for your language</h3> - <p>Depending on the language you are localizing to, it make sense + <p>Depending on the language you are localizing to, it may make sense to write a stemmer for words that will be inserted into the index. A stemmer takes inflected or sometimes derived words and reduces them to their stem. For instance, jumps and jumping would be reduced to @@ -1726,19 +1784,19 @@ OdpRdfArchiveBundle be stemmed before being looked up in the index. Currently, Yioop! comes with only an English language stemmer that uses the Porter Stemming Algorithm [<a href="#P1980">P1980</a>]. This stemmer is located in the - file lib/stemmers/en_stemmer.php . The [<a href="#P1980">P1980</a>] link + file WORK_DIRECTORY/locale/en-US/resources/tokenizer.php . + The [<a href="#P1980">P1980</a>] link points to a site that has source code for stemmers for many other languages (unfortunately, not written in PHP). It would not be hard to port these - to PHP and then add a file to the lib/stemmers folder. For instance, one - could add a file fr_stemmer.php containing a class FrStemmer with method - stem($word) if one wanted to add a stemmer for French. To get Yioop! to use - your stemmer you would then edit the file lib/phrase_parser.php . At the - start of this file there is a static associative array $STEMMERS. You - would add an entry for your stemmer to this array, for French this would - look like: 'fr' => 'FrStemmer' . + to PHP and then add modify the tokenizer.php file of the + appropriate locale folder. For instance, one + could modify the file + WORK_DIRECTORY/locale/fr-FR/resources/tokenizer.php + to contain a class FrStemmer with method + stem($word) if one wanted to add a stemmer for French. </p> <p>In addition to supporting the ability to add stemmers, Yioop also - supports another technique which can be used in lieu of a stemmer + supports a default technique which can be used in lieu of a stemmer called character n-grams. When used this technique segments text into sequences of n characters which are then stored in Yioop! as a term. For instance if n were 3 then the word "thunder" would be split @@ -1748,72 +1806,94 @@ OdpRdfArchiveBundle words in the text are often not separated with spaces. It is also useful for languages like German which can have long compound words. The drawback of n-grams is that they tend to make the index larger. - A list of languages which will be n-grammed by Yioop! can be - found in lib/phrase_parser.php in the global variable $CHARGRAMS. - This is an associative array of language tag => n to use for - char gramming. It should be noted if you decide to add a more - memory efficient stemmer for a language, then you should remove - the entry for the language tag from $CHARGRAMS. If you add a - language to Yioop! and want to use char gramming merely and an - additional entry to this array.</p> - <h3>Adding a bigram filter for your language</h3> + For Yioop built-in locales that do not have stemmer the file, the file + WORK_DIRECTORY/locale/LOCALE-TAG/resources/tokenizer.php has a line + of the form $CHARGRAMS['LOCALE_TAG'] = SOME_NUMBER; This number is + the length of string to use in doing char-gramming. If you add a + language to Yioop! and want to use char gramming merely add a tokenizer.php + to the corresponding locale folder with such a line in it.</p> + <h3>Using token_tool.php to improve search performance and relevance + for your language</h3> + <p>configs/token_tool is used to create suggest word dictionaries and 'n' + word gram filter files for the Yioop! search engine. To create either of + these items, the user puts a source file in Yioop's WORK_DIRECTORY/prepare + folder. Suggest word dictionaries are used to supply the content of the + dropdown of search terms that appears as a user is entering a query in + Yioop. To make a suggest dictionary one can use a command like:</p> + <pre> + php token_tool.php dictionary filename locale endmarker + </pre> <p> - Bigrams are pair of words which always occur together in the same - sequence in a user query, ex: "honda accord". Yioop! can treat these - pair of words as a single word to increase the speed and relevance - of retrieval. The configs/bigram_build.php script can be used to - create a bigram filter file for the Yioop search engine to detect - such words in documents and queries. The input to this script is - an xml file which contains a large collection of such bigrams. One - common source of a large set of bigrams is an XML dump of Wikipedia. - Wikipedia dumps are available for downloaded online free of cost. The - bigrams filter file is specific to a language, therefore, the user has to - create a separate filter file for each language that is to use this - functionality. The configs/bigram_builder.php script can be run multiple - times to create different filter files by specifying a different input xml - files and a different language as command-line arguments. Xml dumps of - Wikipedia for different specific languages are available to download, and it - is these language specific dumps which serve as input to this script. + Here filename should be in the current folder or PREP_DIR and should consist + of one word per line, locale is the locale this suggest (for example, en-US) + file is being made for and where a file suggest-trie.txt.gz will be written, + and endmarker is the end of word symbol to use in the trie. For example, + $ works pretty well. </p> <p> - To illustrate the use bigram_build.php, here are the steps to use it - in the case of wanting to create an English language bigram filter file. + token_tool.php can also be used to make filter files. A filter file is used + to detect when words in a language should be treated as a unit when + extracting text during a crawl. For example, Bill Clinton is 2 word gram + which should be treated as unit because it is a particular person. + token_tool.php is run from the command line as: </p> + <pre> + php token_tool.php filter wiki_file lang locale n extract_type max_to_extract + </pre> <p> - <p><b>Step 1</b>: Go to <a href="http://dumps.wikimedia.org/enwiki/" - >http://dumps.wikimedia.org/enwiki/</a> and obtain a dump - of the English Wikipedia. This page lists all the dumps according - to the date they were taken. Choose any suitable date or the latest. A - link with a label such as 20120104/, represents a dump taken on - 01/04/2012. Click this link to go in turn to a - page which has many links based on type of content you are looking for. - We are interested in content titled - "Recombine all pages, current versions only". Beneath this we might find a - link with a name like:<br /> - <b>enwiki-20120104-pages-meta-current.xml.bz2</b><br /> - This is a bz2 compressed xml file containing all the english pages of - wikipedia. Download this file to the search_filters - folder of your Yioop! work directory associated with your profile. This - file is of the order of 7GB. The bigram_builder.php script though - uncompresses while making a filter so you should have around 100GB free - while creating the bigram filter (You can free up space after the process - if over). + where wiki_file is a wikipedia xml file or a bz2 compressed xml file whose + urls or wiki page count dump file which will be used to determine the + n-grams, lang is an Wikipedia language tag, locale is the IANA language + tag of locale to store the results for (if different from lang, for example, + en-US versus en for lang), n is the number of words in a row to consider, + extract_type is where from Wikipedia source to extract: + </p> + <pre> + 0 = title's, + 1 = redirect's, + 2 = page count dump wikipedia data, + 3 = page count dump wiktionary data. + </pre> - <p><b>Step 2</b>: Run this script from the php command line as follows:</p> + <h3>Obtaining data sets for token_tool.php</h3> + <p> + Many word lists are obtainable on the web for free with Creative Commons + licenses. A good starting point is:</p> + <pre> + <a href="http://en.wiktionary.org/wiki/Wiktionary:Frequency_lists" + >http://en.wiktionary.org/wiki/Wiktionary:Frequency_lists</a> + </pre> + <p>A little script-fu can generally take such a list and put it into the + format of one word/term per line which is needed by token_tool.php</p> + <p> + For filter files, Raw page count dumps can be found at:</p> + <pre> + <a href="http://dumps.wikimedia.org/other/pagecounts-raw/" + >http://dumps.wikimedia.org/other/pagecounts-raw/</a> + </pre> + <p>These probably give the best n-gram or all gram results, usually + in a matter of minutes; nevertheless, this tool does support trying to + extract similar data from Wikipedia dumps. This can take hours.</p> + <p>For Wikipedia dumps, one can go to</p> + <pre> + <a href="http://dumps.wikimedia.org/enwiki/" + >http://dumps.wikimedia.org/enwiki/</a> + </pre> + <p> + and obtain a dump of the English Wikipedia (similar for other languages). + This page lists all the dumps according to date they were taken. Choose any + suitable date or the latest. A link with a label such as 20120104/, + represents a dump taken on 01/04/2012. Click this link to go in turn to a + page which has many links based on type of content you are looking for. For + this tool you are interested in files under "Recombine all pages, current + versions only".</p> + <p> + Beneath this we might find a link with a name like:</p> <pre> - php bigram_builder enwiki-20120104-pages-meta-current.xml.bz2 en + enwiki-20120104-pages-meta-current.xml.bz2 </pre> <p> - This creates a bigram filter en_bigrams.ftr for English in the same - directory. Yioop! will automatically detect the filter file and use - it the next time you crawl as well as when anyone performs an English - language query. It should be noted that Yioop! works perfectly - well if you don't create any bigram filters; however, bigram filters - enhance Yioop!'s ability to return relevant results for some languages - as well as speeds up certain two word queries. If you have a site you - crawled before creating a bigram filter, then make a bigram filter - then data from this older site might actually be served less well because - attempted bigram lookups will fail. + which is a file that could be processed by this tool. </p> <p><a href="#toc">Return to table of contents</a>.</p> <h2 id='framework'>Building a Site using Yioop! as Framework</h2> @@ -1876,7 +1956,7 @@ OdpRdfArchiveBundle <pre> $mycontroller->mypluginnamePlugin </pre> - <p>Notice in each expresseion the name of the + <p>Notice in each expression the name of the particular model or plugin is lower case. Given this way of referring to models, a controller can invoke a models methods to get data out of the file system or from a database with expressions like:</p> diff --git a/en-US/pages/downloads.thtml b/en-US/pages/downloads.thtml index 41e330f..0afdd73 100755 --- a/en-US/pages/downloads.thtml +++ b/en-US/pages/downloads.thtml @@ -2,12 +2,10 @@ <h2>Yioop! Releases</h2> <p>The Yioop! source code is still at an alpha stage. </p> <ul> +<li><a href="http://www.seekquarry.com/viewgit/?a=archive&p=yioop&h=ac5ed18f5faeb3511f809edd8f67dd632c1c43d6&hb=2341626254d213230fce7306637f844caa2a57f4&t=zip" + >Version 0.86-ZIP</a></li> <li><a href="http://www.seekquarry.com/viewgit/?a=archive&p=yioop&h=2941365a784374115a3c64af686ce12c17d28cd8&hb=7d142c2546364229e4796f6a77cdda66285b6ed5&t=zip" >Version 0.84-ZIP</a></li> -</li> -<li><a href="http://www.seekquarry.com/viewgit/?a=archive&p=yioop&h=9da95cca6208cc9eb5b8897bd2966a0f4879dab3&hb=59650148a7df20b56d5cdf4ddb626c83ec281dbf&t=zip" - >Version 0.822-ZIP</a></li> -</li> </ul> <h2>Git Repository / Contributing</h2> <p>The Yioop! git repository allows anonymous read-only access. If you would to