SeekquarrySeekquarry - Yioop Repo - Seekquarry

« Back Issue #85
  • 2012-04-29 23:45 Reported by administrator
  • 2012-04-30 00:25 Marked fixed
Status: Marked fixed
Priority: Medium

Add support for * and $ in robots.txt paths; add support for X-Robots-Tag

administrator 2012-04-29 23:45
  • and $ are common Google and Bing extensions, same with the X-Robots-Tag http header. It would also be nice to support NoArchive, and NoSnippet.
administrator 2012-04-30 00:25

Fixed with b436dc7. Also refactors robot processing out of fetcher

- Blog - Privacy - Terms - SeekquarryBot
- Developed at SeekQuarry -
(c) 2026 Seekquarry - Open Source Search Engine Software