This is a first attempt at adding classification to Yioop. Binary classifiers (i.e., classifiers that try to decide whether a new web page is or is not an instance of some class, like spam) can be trained using the new Classifiers activity, and then used in a crawl by selecting the desired classifiers under Page Options. Training requires a training set, which you can build by selecting documents from an existing crawl index and labeling them as either positive or negative examples of the target class. During a crawl, if a classifier determines that a page belongs to its target class, then it adds a class:<target_class> meta word to the document, which can be searched for later. For example, if a web page is determined to be spam according to a "spam" classifier, then the page will have the class:spam meta word attached.

Hey Shawn,

Thanks for the awesome patch. Since my local copy was in quite a disastrous state it took a little arm twisting to apply (our admin_controller.php files were in conflict), but after that it worked. I will push stuff to trunk, once I get my own modifications a little more stable. I guess a next step would be to get a patch for the documentation so I could know how to use the classifiers.

Thanks again, Chris

Implemented by 2f6c5c516. Thanks for your work Shawn!

Attachments:

classification.patch