It is often useful to crawl data of different formats. This is not performed by traditional search engines. This patch extends Yioop to crawl log files and databases.

The following files have been updated:

admin_controller.php - To store data provided by user on the interface search.css - To format the look of the tables displayed crawloptions_element.php - To design UI crawl_constants.php - To introduce new constants log_archive_bundle_iterator.php - Newly added database_archive_bundle_iterator - Newly added

Additional information: Please find the patch attached !!!

Thanks for your time and effort Tanmayee. The code from the above patch won't make it into Version 0.92. However, this is a feature I would like to add at some point so I am keeping this issue open and will post a revised spec here. I will then add support for these features.

Thanks a lot Dr. Pollett. Sure, I will check for notes here regarding progress. I will make sure I see the new version of Yioop when these features are added.

A new text archive iterator class should be created and a new archive database iterator class should be made. These should be very minimalistic with no corresponding UI. The arc_description.ini file for the text archive iterator might look like:

arc_type = 'TextArchiveBundle'; description = 'Mail records to index'; compression = 'gzip' delimiter_regex = '@\nFrom \-@' encoding='UTF-8' glob = 'mail*.log.gz'

This tells the bundle iterator to loop over all files in the archive directory matching the file name pattern mail*.log.gz. compression can be one of gzip, zip, bzip, or plain and tells how the TextArchiveBundle should try to open the files whose name match the pattern. delimiter_regex is a string used to split the uncompressed contents of the file into records. Records returned by this iterator should have the following fields set: The self::URL field should point into the Yioop cache to where this record could be looked up. self::TITLE should be blank, self::DESCRIPTION should be the contents of the record, self::HTTP_CODE should be 200, and self::TYPE should text/plain, self::ENCODING should be set to the value of encoding.

For the database archive bundle iterator, a database resource is specified again with a folder in the archive's folder. For a particular database resource the arc_description.ini file might look like:

arc_type = 'DatabaseArchiveBundle'; description = 'Mantis issues'; pdo_datasourcename = 'mysql:host=localhost;dbname=bugtracker'; username = 'cpollett'; password = 'secret'; query = 'SELECT * FROM mantis_bug_text_table'; delimeter = '\n';

When this iterator runs new PHP PDO object would be created using pdo_datasourcename, username, and password. Records returned would be modified and returned by this iterator as records. In particular, in records output by this iterator the following fields would be set:

The self::URL field should point into the Yioop cache to where this record could be looked up. self::TITLE should be blank, self::DESCRIPTION should be the concatenation of the fields of the original record using delimeter, self::HTTP_CODE should be 200, and self::TYPE should text/plain, self::ENCODING should be set to the value of encoding.

It might make sense to allow add/deleting of database folders with arc_description.ini in them through a web interface, perhaps as part of Search Sources, Add Media Source?

More complicated mappings of text and database records should be done by implementing Issue 111.

This was implemented by Version 0.94

Attachments:

0001-Invisible_Web_Patch.patch

0001-Invisible_Web_Revised_Patch.patch

0001-Invisible_Web_Patch_Dec18.patch