# Page Source / History Visibility and Robots

Arc Post-Mortem Summary. Three parts covering wiki page visibility and site robots files. Part 1 fixed source and history visibility rules for wiki pages; Part 2 handled public page-list visibility; and Part 3 built per-domain robots.txt and humans.txt, covering their generation, serving, and the edit-page wiring.

Arc started 2026-06-16. Goal: let a group owner keep the public (and crawlers) out of wiki page source and history, and let each web-server domain publish its own robots.txt / humans.txt. Group-level CSS was considered and dropped — domain-level CSS already covers that need.

## Part 1 — Source / history visibility ✓ done (this patch)

1. ✓ OPTIONS becomes a flag bitfield; render engine gets its own column. Previously `OPTIONS = (render_engine << 1) + encrypted`, which left no spare bit. Now:
   1. ✓ New column `SOCIAL_GROUPS.RENDER_ENGINE` (fresh schema in `ProfileModel`).
   2. ✓ `OPTIONS` holds only yes/no flags: bit 0 `GROUP_OPTION_ENCRYPTED`, bit 1 `GROUP_OPTION_PAGE_SOURCE_ALLOWED` (new constants in `Config.php`).
   3. ✓ `GroupModel` pack/unpack updated: `addGroup`, `updateGroup`, `isGroupEncrypted`, `getRenderEngine` (reads the column), `getGroupById` (selects the column, unpacks both flag bits). Public method signatures unchanged, so callers do not move.
   4. ✓ Migration `upgradeDatabaseVersion106` (`DATABASE_VERSION` 105→106): adds the column, then in PHP (portable across all five PDO back ends) copies each group's render engine out of OPTIONS and rewrites OPTIONS to keep the encrypted bit and turn the page-source-allowed bit on, so existing behaviour is preserved.
2. ✓ Group-level master. `PAGE_SOURCE_ALLOWED` (the bit) controls public view of both source and history. Default allowed. Set from a new "Public Source & History" Allowed / Not Allowed dropdown in the group edit form (`ManagegroupsElement`), wired through `PAGE_SOURCE_CODES` and the `updateGroup` field list.
3. ✓ Page-level override. The existing per-page `public_source` head var now gates history as well as source. Either the group flag or the page var being off denies the public; editors and owners always see both.
4. ✓ Enforcement. The history endpoint now returns 404 to the public user when the group disallows, or when the current page's `public_source` is off. `NO_HISTORY_SOURCE` also fires on the group flag, so the Source and History links disappear from the page for the public.
5. ✓ Page settings cue. When the group disallows, the page-settings `public_source` checkbox is disabled and shows an "(off at group level)" note, so an editor is not misled into thinking a page can opt back in.
6. ✓ Four locale strings added to `src/locale/en_US/configure.ini`.
7. ✓ Revision (2026-06-17): hide the buttons, not just the bodies. Source, history and discussion controls now disappear when the user may not view source, keyed off the same `NO_HISTORY_SOURCE` signal (page var or group flag off, owners exempt): the floated source/discuss buttons (`GroupbarElement`), the page-mode menu's source/history/discussion entries (`GroupmenuElement`), and the page-listing "view source" button (`WikiElement`, gated on the group flag).
8. ✓ Revision (2026-06-17): a page's discussion thread follows its source. When source is disallowed for the public user, the discussion thread itself returns 404, not just its hidden link — enforced where the feed path already resolves a thread back to its wiki page via `getPageInfoByThread`.

Arc folders for a shared day are now suffixed in the order they happened (2026-06-16A/B/C); existing 2026-06-16 arcs were relabelled accordingly. Past Z, suffixes double (AA, AB, …).

## Part 2 — public page-list visibility ✓ done (2026-06-17)

1. ✓ New OPTIONS flag bit `GROUP_OPTION_PAGE_LIST_ALLOWED` (bit 2, value 4) controlling whether the public may see a group's list of wiki pages. Default allowed.
2. ✓ Migration `upgradeDatabaseVersion107` (`DATABASE_VERSION` 106→107): turns the bit on for every existing group in PHP, so the listing stays visible as before.
3. ✓ `GroupModel` packs and unpacks the bit in `addGroup`, `updateGroup` and `getGroupById` alongside the other flags.
4. ✓ Set from a new "Public Page List" Allowed / Not Allowed dropdown in the group edit form (`ManagegroupsElement`), wired through `PAGE_LIST_CODES` and the field list.
5. ✓ Enforcement: the "pages" listing endpoint returns 404 to the public user when the group disallows it, and the "create / find pages" menu link is hidden from the public (`GroupmenuElement`). Members and owners are unaffected.
6. ✓ Two locale strings plus the dropdown label added.
7. ✓ Fix (2026-06-17): the create-group form reads `page_source_allowed` and `page_list_allowed` from the default group array, so both keys are seeded (allowed) there to stop undefined-key warnings on the Create Group page.

## Part 3 — per-domain robots.txt / humans.txt ▶ NEXT

Design agreed 2026-06-17 from Chris's spec. Each secure-route domain gets its own generated robots.txt and a free-form humans.txt, edited from the Web Server settings. Decisions: store in the per-domain appearance blob (no migration); a disallowed category blocks every address that reaches it (both the clean path and the query form); toggles are global for now (not per-group); the old shipped src/robots.txt is removed since this supersedes it.

## Part 3a — generation & serving ✓ done (2026-06-17)

1. ✓ `RobotsTxtGenerator` library class (pure logic): turns a domain's saved yes/no choices into robots.txt text. Disallow-all yields a blanket `Disallow: /`; otherwise it opens for all crawlers, adds a crawl-delay line, a Disallow block for every address leading to a turned-off category (both clean path like `/group` and query form like `/*?*c=group`), then the admin's extra directives. An unconfigured domain defaults to everything crawlable except search-result addresses.
2. ✓ Serving: index.php route map now sends `robots.txt` to `routeRobotsTxt` and `humans.txt` to `routeHumansTxt`. Each finds the request host, reads that domain's saved config from the appearance blob via a `GroupModel`, and emits generated text through the WebSite object, exactly as the `mta-sts.txt` route does.
3. ✓ Removed the old static `src/robots.txt`; the generated default supersedes it.

## Part 3b — the edit page & wiring ✓ done (2026-06-17)

1. ✓ "Configure robots.txt and humans.txt" link beneath the Routing row in the Web Server domain panel, opening `serverSettings&arg=robots&domain=...`.
2. ✓ The edit page itself: a new element with a `CloseHelper` [x] floated opposite; the disallow-all checkbox (JS-hides the other robots inputs, humans stays); the crawl-delay dropdown (1/3/5/10/30/60); the search/feeds/wiki checkboxes with the three wiki sub-checkboxes (source & history, page feeds, page lists); the additional-robots and humans.txt textareas; and Save / Cancel / Preview (Preview opens a tab with the generated robots.txt).
3. ✓ `serverSettings` handling: render the page for `arg=robots`, save the choices into the domain's appearance blob (merging, so appearance overrides are untouched), and a preview path that runs the same generator.
4. ✓ Locale strings for the link, the control labels, and the buttons.
5. ✓ Polish (2026-06-17): the config link sits on its own line beneath Routing (full-width flex row); saving no longer asks for a server restart (the files are served from the DB); and Save / Cancel / Preview are all the same button element so they match in size. Cancel is now a submit that returns to Server Settings with no message.

1. ✓ Store each domain's robots.txt and humans.txt text alongside `DOMAIN_APPEARANCE` (the per-domain serialized override table from the v104 migration).
2. ✓ Add "configure robots.txt" / "humans.txt" links to the web-server domain edit panel (`ServersettingsElement` + `SystemComponent` domain routes), each opening a simple textarea editor.
3. ✓ Resolved (was an open question about whether crawlers hitting group pages should be steered by this robots.txt by default): the two are orthogonal. robots.txt only advises crawlers which paths to fetch; the group-level page-source master from Part 1 enforces what the public may view (server-side 403/404). One does not steer the other, so there is no per-group robots default to decide.