Further improvements to language and safe search handling in the case of a site with lang=mul, adds a checkbox to Settings page for Safe Search, add a dropdown to Crawl Option to specify how robots.txt should be followed, a=chris

Author Chris Pollett <chris@pollett.org>
Author date 2019-06-10 07:Jun:th
Author local date 2019-06-10 00:Jun:th -0700
Committer Chris Pollett <chris@pollett.org>
Committer date 2019-06-10 07:Jun:th
Committer local date 2019-06-10 00:Jun:th -0700
Commit b55ace189c34a736b2801527ae6748390ec668cc
Tree 5d6684dc10419c04a91bea6e42288430e4a0a8f0
Parent 8d9bf9c865f4d1ba01fddee00a487f38e4c85a45
Further improvements to language and safe search handling in the case of a site with lang=mul, adds a checkbox to Settings page for Safe Search, add a dropdown to Crawl Option to specify how robots.txt should be followed, a=chris
Affected files:
src/configs/Config.php
src/configs/PublicHelpPages.php
src/configs/default_crawl.ini
src/controllers/SettingsController.php
src/controllers/components/CrawlComponent.php
src/data/public_default.db
src/executables/Fetcher.php
src/executables/QueueServer.php
src/library/CrawlConstants.php
src/library/FetchUrl.php
src/library/LocaleFunctions.php
src/library/PhraseParser.php
src/library/ScraperManager.php
src/library/processors/CompressedProcessor.php
src/locale/ar/configure.ini
src/locale/bn/configure.ini
src/locale/de/configure.ini
src/locale/en_US/configure.ini
src/locale/es/configure.ini
src/locale/fa/configure.ini
src/locale/fr_FR/configure.ini
src/locale/he/configure.ini
src/locale/hi/configure.ini
src/locale/in_ID/configure.ini
src/locale/it/configure.ini
src/locale/ja/configure.ini
src/locale/kn/configure.ini
src/locale/ko/configure.ini
src/locale/nl/configure.ini
src/locale/pl/configure.ini
src/locale/pt/configure.ini
src/locale/ru/configure.ini
src/locale/te/configure.ini
src/locale/te/statistics.txt
src/locale/th/configure.ini
src/locale/tr/configure.ini
src/locale/vi_VN/configure.ini
src/locale/vi_VN/statistics.txt
src/locale/zh_CN/configure.ini
src/models/CrawlModel.php
src/models/PhraseModel.php
src/views/SettingsView.php
src/views/elements/CrawloptionsElement.php
src/views/elements/PageoptionsElement.php
ViewGit