Controller used to handle search requests to SeekQuarry search site. Used to both get and display search results.
$web_site : \seekquarry\yioop\controllers\seekquarry\yioop\library\WebSite
Stores a reference to the web server when Yioop runs in CLI mode, it acts as request router in non-CLI mode.
In CLI, mode it is useful for caching files in RAM as they are read
__construct(\seekquarry\yioop\controllers\seekquarry\yioop\library\WebSite $web_site = null)
In addition to calling the base class' constructor, set up FileCache objects if we're configured to do query caching
| \seekquarry\yioop\controllers\seekquarry\yioop\library\WebSite | $web_site | is the web server when Yioop runs in CLI mode, it acts as request router in non-CLI mode. In CLI, mode it is useful for caching files in RAM as they are read |
processRequest()
This is the main entry point for handling a search request.
ProcessRequest determines the type of search request (normal request , cache request, or related request), or if its a user is returning from the admin panel via signout. It then calls the appropriate method to handle the given activity.Finally, it draw the search screen.
redirectWithMessage(string $message, string $copy_fields = false, boolean $restart = false)
Does a 301 redirect to the given location, sets a session variable to display a message when get there.
| string | $message | message to write |
| string | $copy_fields | $_REQUEST fields to copy for redirect |
| boolean | $restart | if yioop is being run as its own server rather than under apache whether to restart this server. |
pagingLogic(\seekquarry\yioop\controllers\array& $data, mixed $field_or_model, string $output_field, integer $default_show, array $search_array = array(), string $var_prefix = "", array $args = null)
When an activity involves displaying tabular data (such as rows of users, groups, etc), this method might be called to set up $data fields for next, prev, and page links, it also makes the call to the model to get the row data sorted and restricted as desired. For some data sources, rather than directly make a call to the model to get the data it might be passed directly to this method.
| \seekquarry\yioop\controllers\array& | $data | used to send data to the view will be updated by this method with row and paging data |
| mixed | $field_or_model | if an object, this is assumed to be a model and so the getRows method of this model is called to get row data, sorted and restricted according to $search_array; if a string then the row data is assumed to be in $data[$field_or_model] and pagingLogic itself does the sorting and restricting. |
| string | $output_field | output rows for the view will be stored in $data[$output_field] |
| integer | $default_show | if not specified by $_REQUEST, then this will be used to determine the maximum number of rows that will be written to $data[$output_field] |
| array | $search_array | used to sort and restrict in the getRows call or the data from $data[$field_or_model]. Each element of this is a quadruple name of a field, what comparison to perform, a value to check, and an order (ascending/descending) to sort by |
| string | $var_prefix | if there are multiple uses of pagingLogic presented on the same view then $var_prefix can be prepended to to the $data field variables like num_show, start_row, end_row to distinguish between them |
| array | $args | additional arguments that are passed to getRows and in turn to selectCallback, fromCallback, and whereCallback that might provide user_id, etc to further control which rows are returned |
checkCSRFToken(string $token_name, string $user) : boolean
Checks if the form CSRF (cross-site request forgery preventing) token matches the given user and has not expired (1 hour till expires)
| string | $token_name | attribute of $_REQUEST containing CSRFToken |
| string | $user | user id |
whether the CSRF token was valid
checkCSRFTime(string $token_name, string $action = "") : boolean
Checks if the timestamp in $_REQUEST[$token_name] matches the timestamp of the last CSRF token accessed by this user for the kind of activity for which there might be a conflict.
This is to avoid accidental replays of postings etc if the back button used.
| string | $token_name | name of a $_REQUEST field used to hold a CSRF_TOKEN |
| string | $action | name of current action to check for conflicts |
whether a conflicting action has occurred.
clean(mixed $value, mixed $type, mixed $default = null) : string
Used to clean strings that might be tainted as originate from the user
| mixed | $value | tainted data |
| mixed | $type | type of data in value can be one of the following strings: bool, color, double, float, int, hash, or string, web-url; or it can be an array listing allowed values. If the latter, then if the value is not in the array the cleaned value will be first element of the array if $default is null |
| mixed | $default | if $value is not set default value is returned, this isn't used much since if the error_reporting is E_ALL or -1 you would still get a Notice. |
the clean input matching the type provided
convertArrayLines(array $arr, string $endline_string = "\n", boolean $clean = false) : string
Converts an array of lines of strings into a single string with proper newlines, each line having been trimmed and potentially cleaned
| array | $arr | the array of lines to be process |
| string | $endline_string | what string should be used to indicate the end of a line |
| boolean | $clean | whether to clean each line |
a concatenated string of cleaned lines
convertStringCleanArray(string $str, string $line_type = "url") : \seekquarry\yioop\controllers\$lines
Cleans a string consisting of lines, typically of urls into an array of clean lines. This is used in handling data from the crawl options text areas. # is treated as a comment
| string | $str | contains the url data |
| string | $line_type | does additional cleaning depending on the type of the lines. For instance, if is "url" then a line not beginning with a url scheme will have http:// prepended. |
an array of clean lines
parsePageHeadVarsView(object $view, string $page_name, string $page_data)
Used to set up the head variables for and page_data of a wiki or static page associated with a view.
| object | $view | View on which page data will be rendered |
| string | $page_name | a string name/id to associate with page. For example, might have 404 for a page about 404 errors |
| string | $page_data | this is the actual content of a wiki or static page |
parsePageHeadVars(string $page_data, $with_body = false) : array
Used to parse head meta variables out of a data string provided either from a wiki page or a static page. Meta data is stored in lines before the first occurrence of END_HEAD_VARS. Head variables are name=value pairs. An example of head variable might be: title = This web page's title Anything after a semi-colon on a line in the head section is treated as a comment
| string | $page_data | this is the actual content of a wiki or static page |
| $with_body |
the associative array of head variables or pair [head vars, page body]
initializeAdFields(\seekquarry\yioop\controllers\array& $data, boolean $ads_off = false)
If external source advertisements are present in the output of this controller this function can be used to initialize the field variables used to write the appropriate Javascripts
| \seekquarry\yioop\controllers\array& | $data | data to be used in drawing the view |
| boolean | $ads_off | whether or not ads are turned off so that this method should do nothing |
addDifferentialPrivacy(integer $actual_value) : integer
Adds to an integer, $actual_value, epsilon-noise taken from an L_1 gaussian source to centered at $actual_value to get a epsilon private, integer value.
| integer | $actual_value | number want to make private |
$fuzzy_value number after noise added
recordViewSession(integer $page_id, string $sub_path, string $media_name)
Used to store in a session which media list items have been viewed so we can put an indicator by them when the media list is rendered
| integer | $page_id | the id of page with media list |
| string | $sub_path | the resource folder on that page |
| string | $media_name | item to store indiicator into session for |
initializeResponseFormat() : array
Determines how this query is being run and return variables for the view
A query might be run as a web-based where HTML is expected as the output, an RSS query, an API query, or as a serial query from a name_server or mirror instance back to one of the other queue servers in a Yioop installation. A query might also request different numbers of pages back beginning at different starting points in the result.
consisting of (view to be used to render results, flag for whether html results should be used, int code for what kind of group of similar urls should be done on the results, number of search results to return, start from which result)
initializeSubsearches() : array
Determines if query results are using a subsearch, and if so initializes them, also it sets up list of subsearches to draw at top of screen.
(subsearches, no_query) where subsearches is itself an array of data about each subsearch to draw, and no_query is a bool flag used in the case of a news subsearch when no query was entered by the user but still want to display news
initializeUserAndDefaultActivity(\seekquarry\yioop\controllers\array& $data) : array
Determines the kind of user session that this search request is for
This function is called by @see processRequest(). The user session might be one without a login, one with a login so need to validate against to prevent CSRF attacks, just after someone logged out, or a bot session (googlebot, etc) so remove the query request
| \seekquarry\yioop\controllers\array& | $data | that will eventually be sent to the view. We might update with error messages |
consisting of (query based on user info, whether if a cache request highlighting should be userd, what activity user wants, any arguments to this activity)
initializeIndexInfo(boolean $web_flag, integer $raw, \seekquarry\yioop\controllers\array& $data) : array
Determines which crawl or mix timestamp should be in use for this query. It also determines info and returns associated with this timestamp.
| boolean | $web_flag | whether this is a web based query or one from the search API |
| integer | $raw | should validate against list of known crawls or an internal (say network) query that doesn't require validation (faster without). |
| \seekquarry\yioop\controllers\array& | $data | that will eventually be sent to the view. We set the 'its' (index_time_stamp) field here |
consisting of index timestamp of crawl or mix in use, $index_info an array of info about that index, and $save_timestamp timestamp of last savepoint, used if this query is being is the query for a crawl mix archive crawl.
restrictQueryByUserAgent(string $query) : string
Sometimes robots disobey the statistics page nofollow meta tag.
and need to be stopped before they query the whole index
| string | $query | the search request string |
the search request string if not a bot; "" otherwise
addSearchViewData(array $index_info, boolean $no_query, integer $raw, string $view, array $subsearches, \seekquarry\yioop\controllers\array& $data)
Prepares the array $data so the SearchView can draw search results
| array | $index_info | an array of info about that index in use |
| boolean | $no_query | true in the case of a news subsearch when no query was entered by the user but still want to display news |
| integer | $raw | $raw what kind of grouping of identical results should be done (0 is default, 1 and higher used for internal queries) |
| string | $view | name of view class search results are for |
| array | $subsearches | an array of data about each subsearch to draw to the view |
| \seekquarry\yioop\controllers\array& | $data | that will eventually be sent to the view for rendering. This method adds fields to the array |
mirrorHandle() : boolean
Only used for serial network queries Used to check if there are any mirrors of the current server.
If so, it tries to distribute the query requests randomly amongst the mirrors and itself. To determine if there are mirrors of the current server it looks in a mirror_table.txt file for machines that have notified this machine they are mirroring it.
whether or not a mirror of the current site handled it
processQuery(\seekquarry\yioop\controllers\array& $data, string $query, string $activity, string $arg, integer $results_per_page, integer $limit, integer $index_name, integer $raw, mixed $save_timestamp, boolean $limit_feeds = true)
Searches the database for the most relevant pages for the supplied search terms. Renders the results to the HTML page.
| \seekquarry\yioop\controllers\array& | $data | an array of view data that will be updated to include at most results_per_page many search results |
| string | $query | a string containing the words to search on |
| string | $activity | besides a straight search for words query, one might have other searches, such as a search for related pages. this argument says what kind of search to do. |
| string | $arg | for a search other than a straight word query this argument provides auxiliary information on how to conduct the search. For instance on a related web page search, it might provide the url of the site with which to perform the related search. |
| integer | $results_per_page | the maixmum number of search results that can occur on a page |
| integer | $limit | the first page of all the pages with the query terms to return. For instance, if 10 then the tenth highest ranking page for those query terms will be return, then the eleventh, etc. |
| integer | $index_name | the timestamp of an index to use, if 0 then default used |
| integer | $raw | ($raw == 0) normal grouping, $raw > 0 no grouping done on data. If $raw == 1 no summary returned (used with f=serial, end user probably does not want) In this case, will get offset, generation, etc so could later lookup |
| mixed | $save_timestamp | if this timestamp is nonzero, then save iterate position, so can resume on future queries that make use of the timestamp. $save_time_stamp may also be in the format of string timestamp-query_part to handle networked queries involving presentations |
| boolean | $limit_feeds | if true the number of feed shard items to allow in search results is limited to WordIterator::LIMIT_FEEDS_COUNT |
calculateControlWords(string $query, boolean $raw, boolean $is_mix, string $index_name) : array
Extracts from the query string any control words: mix:, m:, raw:, no: and returns an array consisting of the query with these words removed, and then variables for their values.
| string | $query | original query string |
| boolean | $raw | the $_REQUEST['raw'] value |
| boolean | $is_mix | if the current index name is that of a crawl mix |
| string | $index_name | timestamp of current mix or index |
($query, $raw, $use_network, $use_cache_if_possible, $guess_semantics)
makeMediaGroups( $pages) : array
Groups search result pages together which have thumbnails from an array of search pages. Grouped thumbnail pages stored at array index of first thumbnail found, non thumbnail pages stored where were before
| $pages | an array of search result pages to group those pages with thumbs within |
$pages after the grouping has been done
getTopPhrases(string $crawl_item, integer $num, integer $crawl_time) : array
Given a page summary extract the words from it and try to find documents which match the most relevant words. The algorithm for "relevant" is pretty weak. For now we pick the $num many words whose ratio of number of occurrences in crawl item/ number of occurrences in all documents is the largest
| string | $crawl_item | a page summary |
| integer | $num | number of key phrase to return |
| integer | $crawl_time | the timestamp of an index to use, if 0 then default used |
an array of most selective key phrases
extractActivityQuery() : array
This method is responsible for parsing out the kind of query from the raw query string
This method parses the raw query string for query activities. It parses the name of each activity and its argument
list of search activities parsed out of the search string
markChildren(object $node, array $words, object $dom) : object
Used in rendering a cached web page to highlight the search terms.
| object | $node | DOM object to mark html elements of |
| array | $words | an array of words to be highlighted |
| object | $dom | a DOM object for the whole document |
the node modified to now have highlighting
canonicalizeLinks(object $node, string $url) : object
Make relative links canonical with respect to provided $url for links appear within the Dom node.
| object | $node | dom node to fix links for |
| string | $url | url to use to canonicalize links |
updated dom node
queryRequest(string $query, integer $results_per_page, integer $limit, integer $grouping, integer $save_timestamp, boolean $limit_feeds = true) : array
Part of Yioop Search API. Performs a normal search query and returns associative array of query results
| string | $query | this can be any query string that could be entered into the search bar on Yioop (other than related: and cache: queries) |
| integer | $results_per_page | number of results to return |
| integer | $limit | first result to return from the ordered query results |
| integer | $grouping | ($grouping == 0) normal grouping of links with associated document, ($grouping > 0) no grouping done on data |
| integer | $save_timestamp | if this timestamp is nonzero, then save iterate position, so can resume on future queries that make use of the timestamp |
| boolean | $limit_feeds | if true the number of feed shard items to allow in search results is limited to WordIterator::LIMIT_FEEDS_COUNT |
associative array of results for the query performed
relatedRequest(string $url, integer $results_per_page, integer $limit, string $crawl_time, integer $grouping, integer $save_timestamp, boolean $limit_feeds = true) : array
Part of Yioop Search API. Performs a related to a given url search query and returns associative array of query results
| string | $url | to find related documents for |
| integer | $results_per_page | number of results to return |
| integer | $limit | first result to return from the ordered query results |
| string | $crawl_time | timestamp of crawl to look for related request |
| integer | $grouping | ($grouping == 0) normal grouping of links with associated document, ($grouping > 0) no grouping done on data |
| integer | $save_timestamp | if this timestamp is nonzero, then save iterate position, so can resume on future queries that make use of the timestamp |
| boolean | $limit_feeds | if true the number of feed shard items to allow in search results is limited to WordIterator::LIMIT_FEEDS_COUNT |
associative array of results for the query performed
cacheRequest(string $url, array $ui_flags = array(), string $terms = "", string $crawl_time) : string
Part of Yioop Search API. Performs a related to a given url search query and returns associative array of query results
| string | $url | to get cached page for |
| array | $ui_flags | array of ui features which should be added to the cache page. For example, "highlight" would way search terms should be highlighted, "history" says add history navigation for all copies of this cache page in yioop system. |
| string | $terms | space separated list of search terms |
| string | $crawl_time | timestamp of crawl to look for cached page in |
with contents of cached page
cacheRequestAndOutput(string $url, array $ui_flags = array(), string $terms = "", integer $crawl_time)
Used to get and render a cached web page
| string | $url | the url of the page to find the cached version of |
| array | $ui_flags | array of ui features which should be added to the cache page. For example, "highlight" would say search terms should be highlighted, "history" says add history navigation for all copies of this cache page in yioop system. "summaries" says add a toggle headers and extracted summaries link. "cache_link_referrer" says a link on a cache page referred us to the current cache request |
| string | $terms | from orginal query responsible for cache request |
| integer | $crawl_time | the timestamp of the crawl to look up the cached page in |
imageCachePage(string $url, array $cache_item, string $cache_file, $queue_servers) : string
Makes an HTML web page for an image cache item
| string | $url | original url of the image |
| array | $cache_item | details about the image item |
| string | $cache_file | string with image |
| $queue_servers | machines used by yioop for the current index cache item is from. Used to find out urls on which image occurred |
an HTML page with the image embedded as a data url
formatCachePage(array $cache_item, string $cache_file, string $url, string $summary_string, integer $crawl_time, array $all_crawl_times, string $terms, array $ui_flags)
Formats a cache of a web page (adds history ui and highlight keywords)
| array | $cache_item | details meta information about the cache page |
| string | $cache_file | contains current web page before formatting |
| string | $url | that cache web page was originally from |
| string | $summary_string | summary data that was extracted from the web page to be put in the actually inverted index |
| integer | $crawl_time | timestamp of crawl cache page was from |
| array | $all_crawl_times | timestamps of all crawl times currently in Yioop system |
| string | $terms | from orginal query responsible for cache request |
| array | $ui_flags | array of ui features which should be added to the cache page. For example, "highlight" would way search terms should be highlighted, "history" says add history navigation for all copies of this cache page in yioop system. return string of formatted cached page |
addKeywordLinks(string $web_page, array $cache_item) : string
Function used to add links for keyword searches in keyword_links array of $cache_item to the text of the $web_page we are going to display the cache of as part of a pache page request
| string | $web_page | to add links to |
| array | $cache_item | original cache item web page generated from |
modified web page
createSummaryAndToggleNodes(\seekquarry\yioop\controllers\DOMDocument $dom, string $text_align, \seekquarry\yioop\controllers\DOMElement $body, string $summary_string, array $cache_item) : \seekquarry\yioop\controllers\DOMElement
Creates the toggle link and hidden div for extracted header and summary element on cache pages
| \seekquarry\yioop\controllers\DOMDocument | $dom | used to create new nodes to add to body object for page |
| string | $text_align | whether rtl or ltr language |
| \seekquarry\yioop\controllers\DOMElement | $body | represent body of cached page |
| string | $summary_string | header and summary that were extraced |
| array | $cache_item | contains infor about the cached item |
a div node with toggle link and hidden div
createDomBoxNode(\seekquarry\yioop\controllers\DOMDocument $dom, string $text_align, string $more_styles = "", string $tag = "div") : \seekquarry\yioop\controllers\DOMElement
Creates a bordered tag (usually div) in which to put meta content on a page when it is displayed
| \seekquarry\yioop\controllers\DOMDocument | $dom | representing cache page |
| string | $text_align | whether doc is ltr or rtl |
| string | $more_styles | any additional styles for box |
| string | $tag | base tag of box (default div) |
of styled box
getCrawlItems(string $url, array $crawl_times, array $queue_servers) : \seekquarry\yioop\controllers\[$all_crawl_times,
Get crawl items based on queue server setting.
| string | $url | is the URL of the cached page |
| array | $crawl_times | is an array storing crawl times for all indexes |
| array | $queue_servers | is an array containing URLs for queue servers |
$all_crawl_items] is an array containing an array of crawl times and an array of their respective crawl items
historyUI(\seekquarry\yioop\controllers\long $crawl_time, array $all_crawl_times, \seekquarry\yioop\controllers\DOMElement $div_node, \seekquarry\yioop\controllers\DOMDocument $dom, string $terms, boolean $hist_ui_open, string $url) : \seekquarry\yioop\controllers\DOMElement
User Interface for history feature
| \seekquarry\yioop\controllers\long | $crawl_time | is the crawl time |
| array | $all_crawl_times | is an array storing all crawl time |
| \seekquarry\yioop\controllers\DOMElement | $div_node | is the section that contains the History UI |
| \seekquarry\yioop\controllers\DOMDocument | $dom | is the DOM of the cached page |
| string | $terms | is a string containing query terms |
| boolean | $hist_ui_open | is a flag to check if History UI should be open by default |
| string | $url | is the URL of the page |
the section containing the options for selecting year and month
toggleHistory(array $months, \seekquarry\yioop\controllers\DOMElement $div_node, \seekquarry\yioop\controllers\DOMDocument $dom)
The history toggle displays the year and month associated with the timestamp at which the page was cached.
| array | $months | used to store month names for which we have a cache |
| \seekquarry\yioop\controllers\DOMElement | $div_node | is the section that contains the History UI |
| \seekquarry\yioop\controllers\DOMDocument | $dom | is the DOM of the cached page |
createHistoryDataStructure(array $all_crawl_times, string $locale_type, string $url) : array
Creates a data structure for storing years, months and associated timestamp components
| array | $all_crawl_times | is an array storing all crawl time |
| string | $locale_type | is the locale tag |
| string | $url | is the URL for the cached page |
$results is an array storing years array, months array and the combined data structure for the History UI
createLinkDivs(array $time_ds, string $current_year, string $current_month, \seekquarry\yioop\controllers\DOMElement $d1, \seekquarry\yioop\controllers\DOMDocument $dom, string $url, array $years, boolean $hist_ui_open, string $terms, \seekquarry\yioop\controllers\long $crawl_time) : \seekquarry\yioop\controllers\DOMElement
Create divs for links based on all (year, month) combinations
| array | $time_ds | is the data structure for History UI |
| string | $current_year | is the year associated with the timestamp of the cached page |
| string | $current_month | is the month associated with the timestamp of the cached page |
| \seekquarry\yioop\controllers\DOMElement | $d1 | is the section that contains options for years and months |
| \seekquarry\yioop\controllers\DOMDocument | $dom | is the DOM for the cached page |
| string | $url | is the URL for the cached page |
| array | $years | is an array storing years associated with all indexes |
| boolean | $hist_ui_open | checks if the History UI state should be open |
| string | $terms | is a string containing the query terms |
| \seekquarry\yioop\controllers\long | $crawl_time | is the crawl time for the cached page |
$d1 is the section containing the options for selecting year and month
viewLinksByYearMonth(array $years, array $months, string $current_year, string $current_month, array $time_ds, \seekquarry\yioop\controllers\DOMDocument $dom) : \seekquarry\yioop\controllers\DOMElement
Display links based on selected year and month in History UI
| array | $years | is an array storing years associated with all indexes |
| array | $months | is an array storing months |
| string | $current_year | is the year associated with the timestamp of the cached page |
| string | $current_month | is the month associated with the timestamp of the cached page |
| array | $time_ds | is the data structure for History UI |
| \seekquarry\yioop\controllers\DOMDocument | $dom | is the DOM for the cached page |
$d1 is the section containing the options for selecting year and month
addCacheJavascriptTags(\seekquarry\yioop\controllers\DOMDocument $dom, \seekquarry\yioop\controllers\DomElement& $node)
Add to supplied node subnodes containing script tags for javascript libraries used to display cache pages
| \seekquarry\yioop\controllers\DOMDocument | $dom | used to create new nodes |
| \seekquarry\yioop\controllers\DomElement& | $node | what to add script node to |