I have implemented the centroid method for text summarization. It also generates the word cloud and displays it on the search page.

Suggestions by Dr.Pollett on first patch: Give braces for loops and functions according to coding guidelines. Change the crawl constant 'CLOUD' to 'WORD_CLOUD'. Change the color of word cloud.

Second patch attached with suggested changes.

Hey Mangesh,

Your second patch is kinda huge compared to the first. Can you regenerate it?

Best, Chris

Hello Dr.Pollett,

The patch was huge because of the locale changes. So I regenerated and uploaded the patch without the locale changes. Three changes in locale that you have to do manually are as follows: crawl_component_basic = "Basic" crawl_component_centroid = "Centroid" crawloptions_element_summarizer = "Summarizer:"

Thank you. Mangesh

Hello Dr. Pollett,

I have resolved the notices appearing when no summarizer is selected.

Changes after the previous patch: Added the centroid summarizer option in default_crawl.ini. Removed the work_dir from config.php which was set in my previous patch. Added summarizer_option variable in queue_server.php

Locale still needs to be changed manually.

Thank you. Mangesh

Hi Mangesh,

I added the patch you had above. I test the centroid stuff on just the main seekquarry page and I noticed there are several places where in the summary words that should be separated by space get squished together. For example, "queryingof" below. I was wondering if you investigate and fix this?

Best, Chris

Resources. Table of Contents. Overview Getting Started Introduction to Search Engines and Yioop Yioop Feature List Set-up Requirements Installation and Configuration Upgrading Yioop Summary of Files and Folders Search and User Interface Search Basics Search Operators Result Formats Settings and Signin Mobile Interface Managing Accounts Managing Users, Roles, and Groups Crawling and Customizing Results Performing and Managing Crawls Mixing Crawl Indexes Classifying Web Pages Page Indexing and Search Options Results Editor Search Sources GUI for Managing Machines and Servers Building Sites with Yioop Building a Site using Yioop as a Framework Embedding Yioop in an Existing Site Localizing Yioop to a New Language Advanced Topics Modifying Yioop Code Yioop Command-line Tools References. Overview. Getting Started. This document serves as a detailed reference for theYioop search engine. If you want to get started using Yioop now,you probably want to first read the InstallationGuides page. Nutch is the original application forwhich the Hadoop infrastructure described above was developed. The header can be used to storeauxiliary data structures into the list if desired. This concludes the discussion of how Yioop fits into the current andhistorical landscape of search engines and indexes. It also supports command-line search queryingof these archives.Yioop supports an indexing plugin architecture to make itpossible to write one's own indexing modules that do furtherpost-processing.Web and Archive CrawlingYioop supports open web crawls, but through its web interface one canconfigure it also to crawl only specifics site, domains, or collectionsof sites and domains. For Windows, Mac, and Linux, another easy way to get therequired software is to download a Apache/PHP/MySql suite such as XAMPP. Return to table of contents. point your web browser to where your Yioop folder is located, a. You should. Web, RSS, and API. section. Unlike. You can. There are a.

Hello Dr.Pollett,

I fixed that. The two words separated by \n were getting squished together.

Patch name: mangesh_wordspace.patch

Thank you. Mangesh

Hi Mangesh, I just directly changed the one line indicated by your patch. I also tried to reduce the memory footprint of your code. Can you try to see if the changes I made make sense? The code still seems to run slow. If I take the Bill Clinton entry from wikipedia it times out before computing the centroid when I try in Page Options test. Can you see if you see any obvious speed-ups.

Best, Chris

Hi Mangesh,

Your code breaks yioop's ability to be distributed to multiple machines. You added getSeedInfo to HtmlProcessor and TextProcessor which tries to read a local copy of the crawl.ini file. That information is not on the local machine though typically. I am going to try to come up with a fix, but I might have to back out your code.

Chris

Okay, I fixed the issue caused by the way you did getSeedInfo, I also improved up the sentence extractor you had. It might make sense to add a filter on sentence size to reduce the number of sentences that the summarizer considers on some pages.

I've added the sentence size thresholds mentioned above with commit a179fe07

Hello Dr.Pollett,

Thank you very much for fixing the issues. It really improved the performance of centroid method.

Thanks, Mangesh

Thanks Mangesh for the patch. I am closing this issue now. If you would like to add more to your code please reopen it

Incorporated with 207c1075fed

Attachments:

my1.patch

my2.patch

my.patch

mangesh.patch

mangesh_wordspace.patch