HTML edition, converted from Markdown 2026-06-19. Legend: ✓ done ☐ open ▮ partial ✗ dropped.

# Yioop v10 — Working TODO List

Last updated: 2026-07-13. Done items marked [x]. Phases 0–4 are complete; Phase 4.5 is in progress.

2026-06-15 progress pass (git log since 2026-06-13 13:30, through HEAD `7bfab72`): item 13 (outbound mail construction dedup) and item 47 (random logouts — database-backed session restore) closed; item 17 (page-resource folder UI) substantially landed (remainder noted under the item). New production/startup work folded in as items 69–71 and logged in Phase 2.5. No items removed.

## Atto integration (yioop now uses WebSite, MailSite, TurnSite from atto v2)

- 1. ☐ Use the new WebSocket support in `WebSite.php` for chat / live updates. When Yioop is being served by atto's `WebSite`, prefer WebSockets over SSE for the messaging refresh path. When Yioop is running under Apache/nginx (no WebSocket support guaranteed), fall back to SSE.

- 2. ☐ Replace the polling loop on text chat (`messagesStatus` → `type=status`, no active call) with push-based delivery. Currently SSE is used only when a call is active; plain text messaging still polls.

## Voice/video calling

- 3. ☐ Investigate and fix bugs in the initial video-chat connection establishment. Symptom: the call sometimes fails to establish on the first try. Suspect signaling order between SSE and the first ICE/SDP exchange in `messagesStatus` / `addCallEvent` / `nextCallEvent`, or the call-group state machine in `GroupModel`. Need a reproducer first.

- 4. ☐ Stand up `TurnSite.php` (from atto) as a TURN/STUN server for Yioop video chats. Wire the ICE-server config in `Config.php` (~line 1493) to be served by TurnSite when enabled.

- 5. ☐ Add Manage-Machines controls for TurnSite: on/off, status, logging.

## Chat UI refresh

- 6. ☐ Modernise the chat input row (`UsermessagesElement.php` lines 366–406): wrap emoji + text + attach into a single rounded "pill"; replace single-line `<input type="text">` with auto-growing multi-line `<textarea>` (Enter-to-send, Shift+Enter for newline); swap mic/send based on whether input has text; move file-input inside the pill.

- 7. ☐ Move tertiary chat-header actions behind a kebab menu.

- 8. ☐ Rethink the translation feature. Per-conversation translation dropdown is confusing UX. Either drop entirely or replace with a per-message context-menu affordance (iMessage / WhatsApp pattern).

- 9. ☐ Bubble + spacing polish. Sender-coloured "me" vs neutral "them" via the new CSS custom properties (item 24); time labels only between groups (already done — keep it); quoted-reply visual.

## Mail

- 10. ✓ Add Manage-Machines controls for `MailSite.php`: on/off, status, logging. Wired in `MachineController::statuses` (the `MailServer` case dispatches `CrawlDaemon::start` / `CrawlDaemon::stop` based on the `start` / `stop` action) and in `MachineController::log` (the `MailServer` case reads `LOG_DIR/MailServer.log`; the `Mail` case reads `LOG_DIR/<SmtpClient::MAIL_LOG_NAME>` for the per-send mail transcript). Status visibility is by membership in the `$statuses["MailServer"][-1]` map.

- 11. ✓ Webmail interface inside Yioop — built. A capability Yioop did not have before v10: the `userMail` activity reads and writes MailSite mail through the MailBackend abstraction — folders, message list, read/compose/reply, flags, move/delete — so the webmail surface is built and in use. MailSite exposes the webmail-friendly direct-call API it uses. Further enhancements are filed individually as concrete items rather than tracked as an open-ended "polish" task (example: swipe-left-to-delete a message on mobile).

- 12. ✓ Fix bulk-mail headers so Gmail/Yahoo/etc. stop rejecting `BulkEmailJob` output. Add Message-ID, MIME-Version, Content-Type, Content-Transfer-Encoding, Reply-To, Return-Path, and for bulk paths List-Unsubscribe + List-Unsubscribe-Post (RFC 8058). Also DKIM signing (RFC 6376). SPF and DMARC are DNS-side but document them in the install guide.

- 13. ✓ Refactor outbound mail to share message-construction code with MailSite where useful, but keep the SMTP-client role. Plan: extract RFC 5322 / MIME building into a shared helper; when local MailSite is enabled and the recipient is local, route via `MailSite::deliverMail()`; shrink the remaining outbound code to remote-relay only. Status: partial. The shared helper `library/MimeMessage.php` exists and is the builder used by both `SmtpClient::sendImmediate` (the legacy outbound path) and `userMailDispatchMailsiteSend` (the new MailSite-outbound path from item 55). Local- recipient routing via `MailSite::deliverMail()` is wired in `userMailDispatchMailsiteSend`. Remaining work: trim `SmtpClient` down to the pure remote-relay role now that message construction lives in `MimeMessage` and local delivery bypasses SMTP entirely; in particular, `SmtpClient::sendImmediate`'s convenience overload that still builds a multipart body from `to/subject/body/attachments` args can collapse to a thin call into `MimeMessage::build` plus the wire-protocol exchange. Closed 2026-06-14 (commit `0e85328`, "mail construction dedup and docs"): `SmtpClient::sendImmediate` now builds the wire message via `MimeMessage::build(...)` and `SmtpClient` is trimmed to the wire-protocol exchange; ~600 lines of duplicated multipart construction removed across `MimeMessage.php`/`SmtpClient.php`, and docblocks rewritten to plain English. Both outbound paths (`sendImmediate` and `userMailDispatchMailsiteSend`) now share one builder; local-recipient routing via `MailSite::deliverMail()` was already wired (item 55). Item 12's header work (RFC 5322 / DKIM / List-Unsubscribe) is separate and still open.

- 14. ✓ Rename `library/MailServer.php` → `library/SmtpClient.php`. Landed early as part of phase 0 (the rename was inexpensive and removed a long- standing misname -- the old name suggested a server-side listener, but the class is purely a client of an upstream SMTP server). Item 13 (refactor outbound to share construction code with MailSite) still pending for phase 2.

- 63. ✓ Configurable mail Banner (`MAIL_HOST_NAME`). New Server-Settings field labelled "Banner:" that sets the host name MailSite announces in the SMTP/IMAP greeting and uses as the EHLO/HELO identity on outbound delivery. Single source of truth `MailSiteFactory::mailHostName()` (= `MAIL_HOST_NAME`, else first `MAIL_DOMAINS`, else `php_uname('n')`, else `localhost`); `MailServer::serverName` and `SmtpClient`'s outbound HELO both delegate to it (HELO is now one consistent server identity instead of the per-message sender domain). `SystemComponent` folds a PTR row (resolved sending IP → banner host) into the Suggested DNS Records panel, and the field placeholder shows the computed default. Wiki Mail Services page updated with a Banner section + PTR note. Advances the item-12/36 deliverability story (the HELO/PTR/FCrDNS half); RFC-5322/DKIM/List-Unsubscribe header work in item 12 is still open.

## Crawling

- 15. ☐ Diagnose and fix the "crawl/index speed degrades over time" bug. Suspect Scheduler tier round-robin, QueueBundle / Bloom-filter growth, IndexDocumentBundle partition rotation, fetcher↔queue-server schedule round-trip cost as bundles get bigger. Need instrumentation first. Sub-items: a. Synthetic crawl test driver. Build a test mode that simulates large crawls under realistic queue-server / fetcher / name-server pressure without actually fetching from the web. Synthetic URLs are minted from a seeded deterministic generator; `Fetcher` swaps in a mock `getPages()` that returns synthetic pages with controllable size, fan-out, and content distributions; the rest of the pipeline (parsing, scheduling, indexing, distribution to queue servers, summary upload) runs unmodified. Lets us drive a multi-million-page crawl on a single workstation in minutes and observe where time and memory accumulate. Wire as a config flag (`SYNTHETIC_CRAWL_MODE`) plus a test executable in `executables/` that orchestrates the synthetic name-server + N synthetic fetchers. b. Instrumentation. Add per-stage timing and memory counters to the suspect paths (Scheduler tier walk, QueueBundle add/peek, dictionary update, partition flush, fetcher-to-queue-server upload). Emit to a structured log (CSV or JSON-lines) the synthetic flow can post-process. c. Diagnose and fix. Run (a) with (b), look at where per-page wall time starts climbing, fix what's found.

- 73. ☐ Crash-safe index writes and quick recovery from a corrupt index. This is the root-cause fix behind the Phase 4 out-of-memory investigation: a corrupt on-disk index made `getPostingsString` read garbage-length postings, which on the live server pushed memory over the limit. Phase 4 shipped the mitigations that keep a corrupt index from taking the running server down (a maximum readable posting length that skips oversized reads, memory-scaled index caches, and per-cache and max-posting-read instrumentation), but those are guards, not prevention. The prevention is here. Sub-items: a. Audit the index-bundle write path. Walk how `IndexDocumentBundle` is added to during a crawl (partition append, dictionary merge, postings write) and find every point where an interrupted or partial write could leave a half-written entry that a later read interprets as a garbage length or offset. b. Make the writes crash-safe. Ensure that if the queue server or dictionary process crashes, is interrupted with Ctrl-C, or is killed partway through, the index is never left corrupt: write-then-rename or a small journal/marker so a partially applied step is either completed or rolled back on restart, giving a quick, safe recovery to a known-good point rather than a corrupt bundle. c. Reproduce and verify with the synthetic crawl driver (item 15a): drive a crawl, kill the queue server or dictionary process at chosen points in the write path, and confirm the index comes back clean and readable every time.

## Wiki

- 16. ✓ Audit and rewrite the wiki-page parsing regexes. Current regexes are fragile and have potential security flaws (XSS via crafted markup, ReDoS). Build an adversarial test corpus and either harden the regexes or move to a proper tokenizer/parser. Overlaps with the security audit (item 27). Decided on the tokenizer and parser, written in both PHP (WikiParser) and JS (help.js), covering the mediawiki-style and markdown engines, with a new markdown help page. A golden regression corpus is in place; the PHP parser and the JavaScript help.js port are both done, the port covered by per-case JavaScript unit tests, and the old regex engine is gone. The adversarial probing found the nesting failures real but the XSS and ReDoS not reproducible with direct probes. Completed in Phase 4, closed 2026-07-10; see the Phase 4 arc plan.

- 17. ☐ Modernise the wiki page-resource folder UI (`WikiElement::renderResources` line 969 / `renderResource` line 1296; drag-drop in `basic.js` line 973):

  - Replace the fixed-`in`-width table with CSS Grid using `repeat(auto-fill, minmax(min, 1fr))` and a container query.

  - Add direct delete (🗑️) per row with confirm dialog. Keep clipboard for cross-page moves.

  - Replace text action buttons with icons. Keep `tl()` strings as `aria-label` + `title`.

  - Inline rename: click name to edit, Enter / blur to commit, Esc to cancel.

  - Drag-drop between same-page subfolders (folder rows + up-level breadcrumb become drop targets).

  - Visible drop-zone overlay when files are dragged from the OS.

  - Multi-select with Shift / Ctrl-click; keyboard shortcuts (Delete, F2, Ctrl+X / C / V, Esc). Status 2026-06-15: substantially landed across the resource drag-drop / mail-style action arc (commits `8f43299`…`7bfab72`). Done: per-row direct delete with confirm; text action buttons replaced by a Mail-style toolbar (✓ confirm-rename / ✕ delete / ⋯ more-menu with Clip Copy + Clip Cut, plus ⚙ wiki-code and Extract), `tl()` strings kept as label/title; inline rename (click name, Enter/blur commit, Esc cancel); drag-drop between same-page subfolders with the folder rows and a pinned Parent Folder row (incl. move-to-root) as drop targets; OS-file drop onto a folder with drop handling; clipboard retained for cross-page moves; multifile and whole-folder drag-drop upload (multifile `$_FILES` in `WebSite`, single clean version per folder via the file's own `headPutContents`), with an upload progress readout (transferred size + moving-average speed) and H2 inbound upload-window sizing; the VersionManager stranded-lock root-cause bug fixed (`copyFileToGroupPageResource` passed `$timestamp` into the `$lock` slot, leaving a lock that silently failed every later rename/version op — commit `bff69c9`); the edit controls render identically under Apache and the built-in server (the v10-introduced `is_writable()` UI gate was the offender; resolved via work_directory permissions, the patch not needed). Remaining: multi-select (Shift/Ctrl-click) + keyboard shortcuts (Delete/F2/Ctrl+X/C/V/Esc); the fixed-`in`- width table → CSS-Grid `auto-fill`/container-query rewrite (the grid layout exists but is not the auto-fill rewrite). Keep open for those two. Logged in Phase 2.5.

- 18. ☐ Flesh out the `{{category|...}}` / `{{category-list|...}}` mechanism. Named parameters; shipped style presets (`style=list|cards|grid|hero|ticker`); pagination; auto-stub category index pages; "Filed under" affordance via `show` flag; cross-group lookup; replace custom `<x-c-list>` tags with semantic HTML.

- 19. ☐ Add a wiki editor live preview pane. Split the editor: textarea on the left, server-rendered preview on the right, AJAX refresh on debounce. Renderer is the existing `WikiParser`. ~50 lines of JS, no deps.

- 20. ☐ Add a contenteditable WYSIWYG-lite layer over the textarea (incremental, after item 19). Toolbar buttons insert wiki markup at cursor — MediaWiki classic-toolbar approach scaled down. Textarea remains source of truth. ~500–1500 lines, no deps.

- 21. ☐ Asynchronous email-based witness flow for secret ballots. The existing `{{secret-ballot|w1|w2|...}}` mechanism (`SocialComponent::initializeBallot` line 5176; `SigninModel::createWitnessKeyPair` line 385, `createBallotFile` line 253, `countBallotFile` line 289) requires all witnesses to authenticate on the same form to initialise the ballot and again to count it. Allow each witness to participate via emailed signed links instead, while preserving the current security and anonymity properties — the server must never see witness passwords or be able to reconstruct the private key without witness collaboration. Approach (no new deps; uses sodium + hash_hmac that are already in use):

  - On ballot init, server emails each witness a one-shot link with a token = HMAC-SHA256(page_id|ballot_id|witness_id| stage|expiry|nonce) keyed by `AUTH_KEY`. Click + sign-in + witness password generates that witness's `$random_i` and `$hash_i` per the existing `createWitnessKeyPair` formula.

  - To preserve the "server never sees the hashes" property: each witness derives a per-ballot keypair from their password (e.g. `sodium_crypto_box_seed_keypair` of `hash($password ⋅ $witness ⋅ $ballot_id)`) and encrypts their `$hash_i` to themselves. Server stores opaque blobs.

  - Once all `n` witness blobs are present, server can publish the ballot's public key only after one final witness confirmation step that releases an aggregated hash chain (avoiding needing every witness on a single live form, but still requiring all to have acted). Several variants of this last step are possible; the simplest: the initiator witness collects the blobs in a final email-link round and computes the polynomial.

  - Count time uses the same email-link mechanism in reverse: each witness clicks a second emailed link, signs in, submits their password to decrypt their own blob; once all contribute, server reconstructs the private key as today.

  - Threat model unchanged from the current synchronous flow: requires all named witnesses to act; server never sees passwords or pre-aggregation hashes; voter anonymity identical (votes still encrypted with sealed-box to the ballot public key).

  - Tests must cover: token expiry, replay rejection, witness who never responds, witness password mismatch on second click, key-mismatch detection (the existing `KEY_MISMATCH` path).

## MediaUpdater / MediaJob distribution

- 22. ☐ Make more `MediaJob` subclasses distributable. Survey:

  - Distributable today: `BulkEmailJob`, `FeedsUpdateJob`, `VideoConvertJob` (full), `AudioTranscriptionJob` (partial).

  - Name-server-only today: `PodcastDownloadJob`, `WikiThumbDetailJob`, `DescriptionUpdateJob`, `AnalyticsJob`, `CullOldRawImpressionsJob`, `RecommendationJob`, `TrendingHighlightsJob`.

## Mobile detection / responsive layout

- 23. ☐ Migrate `if ($_SERVER["MOBILE"])` branches into CSS media queries. ~30 PHP files currently branch on this for layout. Use `@media (max-width: …)` and `@container` (Baseline 2023). File-by-file refactor.

- 24. ✗ Replace the UA string sniff with UA Client Hints. Dropped 2026-06-19. Client Hints (`Sec-CH-UA-Mobile`) are Chrome-only — Safari and Firefox do not send them, so the UA-string sniff would have stayed as the fallback anyway, leaving two mechanisms where there was one. And items 23 and 25 move the mobile decisions into CSS and emit the viewport unconditionally, removing most of the server-side need. A one-line UA fallback stays. (The earlier narrow sub-task of dropping `fennec` from the UA sniff was already done — commit `c590f5d`.)

- 25. ☐ Emit the viewport meta tag unconditionally — moved from Phase 0 to Phase 5. Was reverted because the surrounding CSS isn't yet flex-friendly: with `width=device-width`, the desktop layout extends past phone viewports and content gets cut off. Lands together with items 23, 26, 27, 29, 30, 31 so the layout actually adapts when the meta tag is honoured.

## CSS modernisation (also a Coding.pdf update — see item 36)

- 26. ☐ CSS custom properties for color, spacing, typography. `:root { --color-fg, --color-bg, --color-accent, --space-1 … --space-6, --font-size-base }`. Foundational — items 9, 17, 18 consume these.

- 27. ☐ Adopt CSS logical properties (`margin-inline-start`, `padding-block`, `inset-inline-end`, …) to replace the duplicated `.html-ltr` / `.html-rtl` selector pattern (106 duplications in `search.css` alone). Baseline since 2023. Note: the mail message-action row introduced this pattern as a first foothold during the Phase 2 work — `.mail-message- actions-trash-group` uses `border-inline-start` / `padding-inline-start` / `margin-inline-start` instead of paired `.rtl` overrides. Use as the reference for the wider sweep.

- 28. ☐ Built-in dark mode via `color-scheme: light dark`, `light-dark()`, `prefers-color-scheme`.

- 29. ☐ Drop physical-page units (`in`, `pt`) from screen CSS in favour of `rem` / `em` / `%` / `ch` / `vw` and `clamp()`. `search.css` currently has 276 `in`, 127 `pt`, 784 `px`, 4 `rem`/`em`.

- 30. ☐ Accessibility-aware media features: `prefers-reduced-motion`, `prefers-contrast`, `@media (hover: hover)`, `:focus-visible`. Verify WCAG 2.2 AA contrast in both themes; add axe-core or Lighthouse alongside the existing WAVE check.

## Security audit & hardening

- 31. ☐ Run a full security audit before v10 release.

  - Crypto. `crawlHash` (`Utility.php` line 1616) is `md5`- with-halves-XOR'd → 8 bytes (or ~12-char base64). It's wired into binary index formats — doc IDs, term IDs, posting lists, B+-tree dictionary keys — and 164 callsites across the tree, so a blanket replacement would break every existing index and quadruple storage. Add a sibling `crawlAuthHash($string)` returning `hash_hmac('sha256', $string, AUTH_KEY)` for security uses; migrate just the security-relevant callsites to it (CSRF token in `Controller::generateCSRFToken` line 649 + check at line 671; the `crawlHash("A")` length-test at line 914 needs the new function's length; audit the rest). Indexing callsites stay on `crawlHash` unchanged.

  - Constant-time comparisons. `checkCSRFToken` (`Controller.php` line 672) uses `==`. Switch to `hash_equals`.

  - `crawlCrypt` fallbacks. Remove `mcrypt_create_iv` and `mt_rand` branches once PHP floor lands at ≥ 8.0.

  - Session cookies. Verify `HttpOnly`, `Secure`, `SameSite`; add CSP and `X-Content-Type-Options: nosniff`.

  - I/O paths. Spot-check `$_REQUEST` for `Controller::clean()`; view echoes for escaping; file uploads for traversal / MIME / executable.

  - SQL. Verify all models use prepared statements.

  - Wiki regex parser. Item 16; correctness and ReDoS.
  - Resource controllers. Path traversal.

  - CSRF coverage. Every state-changing action calls `checkCSRFToken`.

  - Dependency surface. `composer.lock` and bundled JS/CSS for known CVEs.

- 32. ✓ Retire the image captcha; gate signup with proof-of-work plus cheap invisible checks and an email confirmation. Image captchas are a declining defence — cheap machine solvers and human captcha-farms beat them — and a burden for blind users, so the `IMAGE_CAPTCHA` mode and `CaptchaModel`'s image drawing are removed. In its place, all self-contained (no outside service), layered:

  1. Proof-of-work as the gate. Keep and strengthen the existing `HASH_CAPTCHA` mode: the browser runs `hash_captcha.js` / `sha1.js` to find a nonce that meets a difficulty level, so a real signup waits a fraction of a second while a bot pays that cost on every one of thousands of attempts. Tune the difficulty (`HASH_CAPTCHA_LEVEL`), make it the single captcha, and simplify the admin Captcha-Type control.
  2. Cheap invisible checks that cost a bot but never bother a person: a honeypot field, a submit-timing check (reject a form completed implausibly fast), and per-IP / per-session rate limits on signup and recovery. The honeypot must be hidden from assistive technology (screen readers such as JAWS / NVDA / VoiceOver): hidden from view, out of the tab order (`tabindex="-1"`), `aria-hidden="true"`, and `autocomplete="off"`, so a blind user never reaches it or is wrongly flagged for filling it.
  3. Email confirmation as the real gate. Require a confirmed email address before an account activates, so every fake account needs a fresh working mailbox. Rides on Yioop's own mail and dovetails with the item 33 email work.

  Touch points (verified 2026-06-19): the auth / suggest mode (global `C\CAPTCHA_MODE`) lives in `RegisterController` (the `switch (C\CAPTCHA_MODE)` branches in create / recover / resend / suggest), `SystemComponent` (the `CAPTCHA_MODES` options), `Config.php` (`IMAGE_CAPTCHA` constant), `SecurityElement` (Captcha-Type control), and the auth forms (`RegisterView`, `RecoverView`, `ResendEmailView`, `SuggestView`). Separately, the image captcha has a second life as the embeddable wiki `[{image-captcha}]` placeholder — `setupGraphicalCaptchaViewData` in the base `Controller.php`, the `user_captcha_text` / `captcha_text` verification in `SocialComponent`, plus `StaticController`, `WikiElement`, `StaticView` — which is not gated by the mode and needs its own decision (drop / replace with proof-of-work / keep image only there). `CaptchaModel` is removed only if that placeholder is dropped or replaced. Repointed 2026-06-19: was “modernise the image captcha”; Chris approved retiring the image method for proof-of-work + invisible layers + email gate, all self-contained.

- 33. ✓ Replace the recovery-question system. (a) Default new installs to `EMAIL_RECOVERY` (done). (b) Add email-based one-time login codes: a short, time-limited, single-use code (or signed link) mailed to the member's registered address, so they can sign in or recover without a saved password and without any external authenticator app. Yioop already sends mail, so this adds no outside dependency — it is the same kind of email one-time login Chris uses with other services. (c) For an offline / airgapped install that keeps a question mode, let users type their own question and answer; drop the `register_view_recoveryN_*` translation infrastructure and remove the existing `EMAIL_AND_QUESTIONS_RECOVERY` mode. (Dropped 2026-06-19: TOTP and printed single-use recovery codes. TOTP forces members to install a separate authenticator app, which cuts against Yioop staying self-contained; the printed codes were its sidekick. The email one-time code in (b) is the self-contained replacement.)

- 47. ✓ Stop random logouts at the environment level — deeper session-lifetime fix. The phase-0 `ini_set` hot-fix in `src/index.php` raises `session.gc_maxlifetime` / `session.cookie_lifetime` per request, but does not reach Debian / Ubuntu's `/etc/cron.d/php*` job. That cron sweeps `session.save_path` (default `/var/lib/php/sessionsXY` or `/tmp`) every 30 minutes using the value of `gc_maxlifetime` from `/etc/php/X.Y/cli/php.ini`, which is whatever the distro shipped — independent of what PHP-FPM / Apache sets per process. On at least one Yioop deployment this still causes randomly-timed logouts; the symptom is "I came back after lunch and was logged out" even though `AUTOLOGOUT` is much longer than the lunch.

  Pick one of the following, in increasing order of robustness:

  (a) Move session storage out of `/tmp` / `/var/lib/php`. Set `session.save_path` to `WORK_DIRECTORY/data/sessions` (chmod 700) before `session_start()`. The Debian cron job only sweeps its configured `save_path`; a Yioop-owned directory is invisible to it. This is the smallest change and probably fixes the observed deployment immediately. Document the directory in the INSTALL guide. ~10 lines in `src/index.php` + a `mkdir -p` in the installer.

  (b) Custom file-based session handler (`SessionHandlerInterface`) that stores in `WORK_DIRECTORY/data/sessions` and respects `AUTOLOGOUT` for its own GC. Strictly stronger than (a) — Yioop owns the entire lifecycle and no external process can delete files prematurely — but adds ~60 lines and a unit test.

  (c) Database-backed session handler storing into the existing Yioop SQL store (new `USER_SESSIONS` table: `SESSION_ID PK`, `USER_ID`, `DATA TEXT`, `LAST_ACCESS INT`, `EXPIRES INT`). Necessary for multi-machine / load-balanced deployments where local-disk sessions don't follow the user across servers. Adds row-level invalidation on logout, an admin "log out everyone" button, and a clean place to track sessions per user for re-authentication prompts and suspicious-activity alerts.

  Recommended sequencing: do (a) first (small change, likely fixes the reported deployment), then (c) as part of the broader auth work in items 32-35 (item 33's email-one-time-login work will want a per-session store anyway). Skip (b) unless (a) doesn't resolve it and (c) is too big for the v10 window.

  Closed 2026-06-14 via approach (c), the robust database-backed path (commits `9b4c27d`, `492ae95`, `ac1b7a1`, `ff39001`→`e1f41a1`, `fcc1dd5`, `4d0f71d`, `d79e387`). New `SESSION_USER` table (`SESSION_ID` PK / `USER_ID` / `EXPIRES`), `SESSION_ID_LEN`=64, `DATABASE_VERSION` 104→105 with `upgradeDatabaseVersion105`. The long-running atto `WebSite` keeps live sessions in an in-memory LRU heap (`SplHeap`, rebuilt on restart) and rehydrates a dropped/expired-from-memory session from the `SESSION_USER` row, so a person stays signed in across a server restart or memory eviction — the actual cause of the "logged out after lunch" reports under the long-lived server. Rehydrate is wired lazily in `index.php` / `Controller` / `AdminController`, persistence via new `UserModel` session methods + `ProfileModel`; sequence-restore is tolerant of a missing/stale row. This also gives the per-user session store items 32–35 want (re-authentication prompts, "log out everyone"). The lighter (a) `session.save_path` move is unnecessary now that Yioop owns the session lifecycle through the DB store, but is still worth a line in the INSTALL guide for Apache/php-fpm deployments where the distro cron sweeps `/var/lib/php`.

## Documentation (to be regenerated for v10)

- 34. ☐ Update the online wiki documentation (last full update was for v9.3, Dec 2022). Source PDFs to refresh: `yioop_docs.pdf`, `Ranking_Mechanisms.pdf`, `Yioop_Install_Guides.pdf`, `Yioop_Wiki_Syntax.pdf`, `Coding.pdf`.

- 35. ☐ Document the new auth flows (email one-time login codes; the recovery modes and user-typed recovery question; and the hardened account-registration / activation-mail flow — opaque self-expiring activation token with no address in the link, generic anti-enumeration replies on the register and resend forms, per-IP and per-email resend throttling, a `Reply-To` from `MAIL_REPLY_TO`, and the multipart text + HTML activation message). Wiki source is delivered as a `.txt` for the wiki UI, never edited into the autogenerated help pages directly. Moved out of the Phase 3 arc 2026-06-25 into this documentation push; the behaviour has shipped, only the write-up remains.

- 36. ☐ Document mail deliverability requirements (DKIM, SPF, DMARC, and now the Banner/`MAIL_HOST_NAME` host identity + PTR / forward-confirmed reverse DNS — see item 63). The Mail Services wiki page already gained a Banner section and a PTR note in Suggested DNS Records this session; the PDF deliverability docs still need the same. Moved 2026-06-19 into the final v10 documentation push (Phase 10); it documents standards already shipped.

- 37. ☐ Document the wiki resource-folder UI, the new category-list parameters and styles, the live preview / WYSIWYG modes, and the asynchronous secret-ballot witness flow.

- 38. ✓ Determined the exact current Yioop runtime requirements for v10 release. PHP minimum pinned at 8.1. Reconciled `composer.json` (now `>=8.1.0` with `ext-curl`, `ext-intl`, `ext-fileinfo`, `ext-libxml` required; `pdo_mysql` / `pdo_pgsql` moved to `suggest`), `INSTALL.txt` (copyright bumped, dedicated Requirements section added), `SystemComponent.php` runtime check (`PHP_VERSION_ID < 80100`), and 25 locale `system_component_php_version` strings. 6 pre-existing locale translation bugs (de, el_GR, he, ko, pt, th) discovered and fixed in passing. Commit `a1cd8f5`. (Update 2026-06-10: `ext-openssl` re-added to `composer.json` and `INSTALL.txt` — the mail/TLS work, STARTTLS, DKIM signing, and ACME all depend on it again, so it is a hard runtime requirement once more.)

- 39. ✓ Updated the Coding-guidelines wiki page for v10. Surgical scope per the agreed plan; details in Phase 1 block below.

## Mail backend / MailSite integration (Phase 2 incremental items)

- 48. ✓ Mail import CLI tool: migrate existing user mailboxes into MailSite storage. Use case: an existing Postfix + maildrop + dovecot stack delivers mail to per-user filesystem trees (Maildir under `/var/mail/<user>/Maildir/{cur,new,tmp}` or mbox under `/var/mail/<user>`). When a deployment switches to MailSite for IMAP service, existing mail needs to migrate without loss. Build `src/executables/MailImport.php` with sources Maildir (priority), mbox, and optional live-IMAP. Common pipeline: parse RFC 5322 bytes + flags + internal date, call `MailStorage::appendMessage($user, $folder, $bytes, $flags, $internal_date)`. Add `--dry-run` and `--resume` (Message-ID dedup) flags. Default destination: `FileMailStorage` rooted at `WORK_DIRECTORY/mail`. Unit-test against a sample Maildir tree under `tests/test_files/` and an mbox sample. No longer blocking step 5; can be sequenced independently now that the webmail surface works against delivered mail.

  *Status (updated 2026-06-07): substantially advanced; used in production to migrate the pollett.org `cpollett` mailbox. The tool is `src/executables/MailTool.php` (subcommand dispatch: `import`/`verify`/`reuid`/`cleanup`, alongside the earlier `migrate`). Maildir source landed earlier (see the historical note below). This session's import work:

  - Streaming, memory-flat import. An earlier version built an array of all message descriptors up front, which OOM'd on the real ~49k-message run. Replaced with a streaming `eachMessage` generator + lightweight `countMessages`; the `appendMessage` loop is itself memory-flat (verified a 64M limit holds across thousands of appends). The earlier partial crash could not auto-resume (it had written no journal), so the chosen recovery was wipe-and-reimport; the streaming path completed it cleanly.

  - Resumable + idempotent. Per-folder `import-<folder>.journal` records done source basenames, written before the mutation and kept after success, so a re-run skips already-imported messages rather than duplicating them, and a crash mid-run resumes where it left off. `reportImport` distinguishes copied vs skipped.

  - Per-folder subscribe + flags-snapshot priming. Import now subscribes each created folder (storage-layer `createFolder` does not subscribe the way IMAP CREATE does, and LSUB filters to subscribed folders, so Apple Mail would not show unsubscribed folders) and primes the flags snapshot via a new public `refreshFlagsSnapshot`. Verified a from-empty rebuild: an emptied user is fully reconstructed by import per folder.

  - Per-folder UIDs from the start. Fresh imports allocate per-folder UIDs (clients render UIDNEXT-1 as an estimated message count, so a shared global UID space made Apple show ~143,931 messages); no separate `reuid` pass is needed after a fresh import.

  - Progress + timestamps. A "Scanning..." line plus "copied N/total" every 1000, and an `outAt()` helper that prefixes `[HH:MM:SS]` on progress/milestone lines across import/reuid/cleanup/verify/migrate.

  - New `reuid` subcommand. Renumbers a folder's UIDs into a fresh per-folder space with a new UIDVALIDITY; crash-safe and resumable via a `reuid.journal`. For repairing mailboxes that predate per-folder UID allocation.

  - New `cleanup` subcommand. Removes orphaned `<digits>.flags`/`<digits>.date` companion files left by the pre-consolidation storage format (one `.eml` per message now; flags live in the index, with `flags.snapshot` as the compaction-time recovery fallback). Streams `readdir`, leaves `.eml`/metadata/subdirs, idempotent. Still open: mbox source (next slice) and optional live-IMAP source (deferred). Possible nicety: a one-command "import whole dovecot tree" wrapper, and extending `cleanup` to sweep stale import journals.*

  Status (updated 2026-06-10): closed. mbox source landed on the import side (a file source is read as an mbox, split on its "From " postmark lines, `>From`-unquoted, with flags from Status/X-Status and the internal date from the Date header or postmark), and the tool was trimmed to its durable command set. The pre-production-only subcommands `migrate`, `reuid`, and `cleanup` were removed — `migrate` converted a legacy percent-encoded folder layout, `reuid` repaired the old global UID space, and `cleanup` swept the orphaned `.flags`/`.date` companion files; all three addressed one-time pre-production states that no current or future install reaches, so they were deleted along with their `moveDirFiles`/`cleanupFolder` helpers rather than carried as dead maintenance surface. A new `export <user> <folder> <out_file>` subcommand writes a user's local mail folder to an mbox: each message gets a "From " postmark dated from its IMAP internal date, Status/X-Status headers carrying its stored flags (which live in the folder index, not the message bytes, so they must be written back for a later import to recover them), and ">From " quoting of body lines beginning "From "; the folder is streamed a message at a time so a large folder stays memory-flat. The postmark is written in the configured local timezone so it round-trips through the import side's `strtotime` parse (a UTC-written postmark was read back as local, shifting dates). Round-trip verified end to end: export then re-import preserves message count, flags, internal dates, and `>From` body lines exactly. The command set is now `import`/`export`/`verify`. Delivered as `yioop-mailtool-export-remove-commands.patch`.

  Earlier status (Maildir slice): New `src/library/MaildirSource.php` parser, `MailImport.php` CLI skeleton with subcommand dispatch (`maildir <path> <user> [--dry-run] [--resume]`), and `tests/MaildirSourceTest.php` covering parseFolderName / parseFlags / extractMessageId / messageMetaFromPath / skipFile / isValid / folders / messageFiles against a deterministic sample tree under `tests/test_files/` (frozen unix-timestamp mtimes set in setUp() since git does not preserve mtime through a patch). Subfolder hierarchy via Maildir++ dot convention (".Sent" -> "Sent", ".Archive.2025" -> "Archive/2025"). Flag mapping per Maildir++ section 4 (R/S/T/D/F -> \Answered/\Seen/\Deleted/\Draft/\Flagged; P/Passed dropped; "new/" messages get \Recent added). Dovecot metadata files skipped. tmp/ not enumerated. --resume dedupes by Message-ID against the destination folder.

- 49. ✓ Mail Services fieldset in Server Settings (commit `76bdec2`). 8 new `MAIL_*` Profile.php constants, fieldset with dropdown + sub-config visible only when MailSite is selected. No listener spawned; intent recorded only.

- 50. ✓ Mail button in SocialControlsElement + userMail activity stub (commit `e34e230`). Button gated on `C\MAIL_MODE !== 'disabled'`. Routes through `routeUserMail` to `SocialComponent::userMail()`, registered as a social activity in both AdminController and GroupController.

- 51. ✓ `src/library/ImapClient.php`: minimal RFC 3501 client for use by the Mail activity (commit `510d4b4`). Ported the ~200-line ImapClient class from atto's Anonymous WebMail demo, brought up to Yioop conventions.

- 52. ✓ Mail activity: external IMAP accounts UI (commit `f53ba8b`). Add/Edit/Delete account forms, account list, inbox listing, single-message view; libsodium credential encryption with master key in private DB; CRUD model with user-scoping. 13 test cases / 36 assertions, all pass.

- 53. ✓ Mail activity: MailSite-stored mail. Decision made in favor of direct in-process MailStorage calls (no socket-loopback layer) wrapped by a backend abstraction so IMAP and MailSite share the same handler surface. Implementation arc (12 patches over ~3 sessions):

  a. Backend abstraction. New `src/library/MailBackend.php` — abstract base with 21 methods: connection lifecycle (close), account identity (accountId, displayName, senderEmail, isSyntheticAccount), folder ops (listFolders, annotateUnreadCounts, createFolder, renameFolder, deleteFolder, purgeFolder), messages (listMessages, fetchMessage, setFlag, moveMessage, deleteMessage, appendMessage), and bulk variants (setFlagBulk, moveMessagesBulk, deleteMessagesBulk). Static factory `MailBackend::forAccountId($account_id, $data, $component)` routes by `MAILSITE_ACCOUNT_ID` (=0, synthetic) vs IMAP account row. Errors throw `MailBackendException` with localized messages.

  b. Concrete backends. `src/library/ImapMailBackend.php` wraps the existing `ImapClient` with lazy connect, cached selected-folder state, and `translateException()` that maps `ImapClient` `ERROR_*` tokens to localized `social_component_mail_err_*` strings. `src/library/MailSiteMailBackend.php` wraps the `MailSiteFactory::build()` direct-API path: in-process, no sockets. Both share the abstract interface contract.

  c. Handler migrations. Each user-visible Mail handler in `SocialComponent` migrated to the backend dispatch pattern: `MailBackend::forAccountId(…) + $backend->method(…) + $backend->close()`. Migrated: `userMailListFolders`, `userMailListMessages`, `userMailViewMessage`, `userMailFolderAction`, `userMailToggleFlag`, `userMailBulkAction`, `userMailArchiveToSent`, `userMailDownloadAttachment`, `userMailDownloadAllAttachments`, `userMailBuildReplyPrefill`, `userMailLoadMessages`. Each migration also dropped the corresponding inline-IMAP worker (`userMailFindSentFolder`, `userMailFindTrashFolder`, `userMailListAll`, `userMailBulkApply`, `userMailFolderApply`, `userMailFetchBody`, `userMailImapErrorMessage`).

  d. Send-archive bug fix. `ImapMailBackend::appendMessage` had two latent bugs (wrong arg order on the `ImapClient::appendMessage` call, wrong response-shape check) that didn't manifest until the first migrated handler actually exercised it; fixed when the migration of `userMailArchiveToSent` surfaced them.

  e. Connection sharing. Discovered that `imapListMessagesAdvanced` (the IMAP-specific sort/filter branch of `userMailListMessages`) opened a second connection per request, tripping Gmail's per-account concurrent-session limit. Added public-with-`@internal` `imapClient()` and `replaceImapClient($new_client)` accessors on `ImapMailBackend` so the advanced flow shares the backend's already-open client. Same pattern then applied to `userMailLoadMessages`'s IMAP branch.

  f. Attachment download Apache-vs-WebSite fix. `userMailStreamAttachment` and `userMailStreamAttachmentsZip` used bare PHP `header()` and `exit()`. Under Apache this works; under WebSite mode the response stream has already started by the time the handler runs, so `header()` warns and `exit()` kills the long-running server process. Switched to `$parent->web_site->header(…)` and `\seekquarry\atto\webExit()`. Same fix-pattern applied proactively to all three `userMessages` JSON `echo + exit()` handlers (load-more-messages, invalid-thread bailout, success-emit).

  g. UI affordances added during the migration. Per-folder unread counts in the sidebar (visible for both MailSite and IMAP); Junk folder in MailSite's default folder set (special-use `\Junk`); user-created MailSite folders surface in the sidebar (merge against `defaultFolders()` in `MailSiteMailBackend::listFolders`); Move action on the message view (dropdown self-submits `onchange`, ordered as `[↩] [↩«] [➡] [Move...▾] | [🗑]`, RTL-correct via `border-inline-start`); sidebar account-list scroll position persisted via `sessionStorage` so the active account stays in view after action redirects.

  h. Test coverage. Three new test files in `tests/`: `MailHeaderParserTest.php` (17 cases covering `parse`/`decodeMimeWord`/`splitAddresses` — pure-function parser); `MailSiteMailBackendTest.php` (13 cases against a `RamMailStorage`-backed MailSite installed via the new `MailSiteFactory::setTestOverride()` test seam; covers synthetic account shape, folder catalog and merge, append/fetch round-trip, flag toggling, move/delete soft- vs-hard, bulk variants). `ImapMailBackendTest.php` is deferred (needs an `ImapClient` socket-pair pattern like `ImapClientTest`; the IMAP backend wraps `ImapClient` which is already covered, so the marginal value vs the work to script wire-level responses is moderate). New `tests/ManyEmailExperiment.php` injects N synthetic messages (default 500) into FileMailStorage for a target user so the webmail UI can be exercised under realistic counts (scrolling, pagination, sort, filter, bulk select); knobs for date span, attachment ratio, seen/flagged ratios.

  i. Post-migration cleanup (deferred). A patch 12 pass would revert the public-`@internal` visibility leaks introduced by patches 5, 6, 18 (`userMailLog`, `userMailProbeFolderUnread`, `imapClient`, `replaceImapClient`) back to protected once the advanced inbox + load-more flows fold into the backend. Estimated ~-100 lines. Not blocking webmail use. Separately, the `MAIL_CLONE_JOB.LAST_ERROR` column drop that this arc made possible (per-message errors moved to `MAIL_CLONE_ERROR`) is fully shipped and verified 2026-06-07 — see the Step 5 note in the phased ordering.

  j. Remaining migration patch. `mail-backend-search` — search UI + handler migration — is the last handler-migration patch. Currently `userMailSearch` (when wired up) still uses inline IMAP SEARCH; will move through a new `$backend->search($folder, $query)` abstract once the search UI is in place. Estimated ~150 lines.

## Mail backend / MailSite — backlog surfaced by phase 2 work (mostly post-v10; item 55 landed during phase 2)

- 54. ✓ Cross-account move. Move a message from one mail account to another (e.g. IMAP → MailSite, or Gmail → Pollett-IMAP). Fetch-from-A + append-to-B + delete-from-A, with rollback on any step's failure. Done: the move UI exposes "Move to another account" alongside the existing within-account move dropdown, and an external IMAP account can be cloned wholesale into local MailSite storage.

- 55. ✓ MailSite outbound sending. Landed during phase 2 (commits `3d9967c`, `7c36dfb`, `f6a2826` on May 24-25) ahead of the post-v10 schedule. `userMailDispatchMailsiteSend` in `SocialComponent` builds the wire message once via `MimeMessage::build`, then classifies recipients by domain: local recipients (whose domain is in `MAIL_DOMAINS`) are delivered directly through `MailSite::deliverMail` (in-process, no SMTP loopback); external recipients are grouped by lowercase domain, each domain's MX records are resolved, and per-MX-host the outbound `SmtpClient` tries port 25 first with a port 587 fallback for networks that block 25. The session uses `opportunistic` STARTTLS (upgrades when the peer advertises it, plaintext otherwise) and no AUTH (peer-MTA mode rather than smarthost submission). After delivery the same bytes are archived to the sender's Sent folder via the MailSite backend's `appendMessage`. Implementation chose option (a) of the original two-options sketch (local MTA relay) but used peer-MTA outbound rather than handing off to an upstream relay; this avoids needing a configured smarthost on each install but means the deployment server must have outbound port 25 (or 587) reachable from its network. See item 60 (SmtpTunnel) for the workaround when a deployment's network egress blocks both.

- 56. ☐ MailSite spam classification. The Junk folder exists (added during the backend-arc work) but no classifier writes to it. Needs an SCL hook in `MailSite::deliverMail`, Mark/ Unmark Junk buttons in the message view, and a classifier (Bayesian against the user's existing Junk + non-Junk corpus, or DNSBL lookups for the upstream IP). Sequencing: ship Mark/Unmark Junk first (manual user training data), then Bayesian on top of the accumulated training set.

- 57. ☐ MailSite unread-count session caching. Per-folder unread probe currently re-runs every `listFolders` call (so the badges are always live). For users with many folders + large message counts the probe walks each folder's metadata; a short-TTL session cache (mirroring the IMAP cache pattern) would amortise the cost.

- 58. ☐ MailSite sort / filter / unread-only / flagged-only. The IMAP `imapListMessagesAdvanced` flow supports server-side SORT, SEARCH-based filters, and the unread- only / flagged-only toggles. MailSite currently runs `listMessages` newest-first only, no filters. The "honest branch" in `userMailListMessages` exists because of this capability gap. Implement equivalents in `MailSiteMailBackend::listMessages` (the storage already has the indices; just needs to expose them).

- 59. ✓ Test Mail Mode + Test Mode Fallback Port. Two related Server Settings toggles, both surfaced inside the Mail Services fieldset. `C\MAIL_TEST_MODE` (boolean, default false) controls two behaviors at once: (1) the compose-form address validation in `SocialComponent::userMailIsValidAddress` relaxes from PHP's strict filter_var FILTER_VALIDATE_EMAIL to a looser "non-empty-local@non-empty-domain" check, so test@127.0.0.1 and user@localhost are accepted in the To/Cc/Bcc fields; (2) `MailSiteFactory::build` skips the `onMailFrom`/`onRcptTo` FQDN-reject hook so MailSite's SMTP listener stops refusing non-FQDN senders/recipients at MAIL FROM and RCPT TO time. `C\MAIL_TEST_MODE_FALLBACK_PORT` (integer, default 0) names a fallback outbound port the MailSite-to-peer-MTA path tries after a TCP-level failure on port 25; consulted only when test mode is on and the port is nonzero. The port-input field in Server Settings is shown/hidden via the same `setDisplay` pattern the MAIL_MODE-dependent blocks use: `mail-test-mode.onchange` flips `mail-test-mode-port-row`. Replaces an earlier unconditional 25→587 fallback (the speculative path Postfix didn't actually accept unauthenticated submission on). Companion: the standalone `smtp_tunnel.php` (item 60) — set the fallback port to the tunnel's listen port to reach an upstream Postfix from a network that blocks 25. The null reverse-path `<>` is always accepted on MAIL FROM regardless of this setting. Persisted through `ProfileModel::$profile_fields` and the `SystemComponent::updateProfileFields` boolean-fields list.

- 60. ✓ `smtp_tunnel.php`: standalone byte-level TCP forwarder for port-25 egress workarounds. Single-file PHP CLI script, no Yioop dependencies and no namespace; lives outside the Yioop tree so it can be deployed on the upstream-SMTP host independently of any Yioop release. Single-process `stream_select`-driven TCP forwarder. Accepts plain TCP connections on a configurable listen port and bridges each one to a single fixed upstream `host:port`, copying bytes bidirectionally with no protocol awareness. Run on the SMTP-bearing host (typically pollett.org) listening on port 5252 (literally "2525 backwards"; well clear of the IANA well-known range so common egress filters do not block it) and forwarding to `127.0.0.1:25`. A Yioop install running on a network that blocks outbound port 25 then points its outbound `SmtpClient` at `pollett.org:5252` and reaches the upstream Postfix transparently. CLI: `php smtp_tunnel.php [listen_port] [forward_host] [forward_port] [allow_csv]` where the optional fourth argument is a comma-separated list of source IPs permitted to connect (empty means no restriction). Operationally the tunnel is the deployment-side complement to MailSite's outbound MTA (item 55): MailSite resolves MX and tries 25→587 from the client side; `smtp_tunnel.php` makes the listening side reachable when the client side is firewalled. Caveat: because the upstream Postfix sees the tunnel host as the peer, allowing this through requires either binding the upstream listener to loopback (the safe shape) or restricting the tunnel's allow-list to the operator's known cruise-ship egress IP -- otherwise the tunnel becomes a small open relay onto port 25.

- 61. ✓ PHP 8.5 dynamic-property declaration on SmtpClient. Declared `public $last_response_body` on `SmtpClient`; it was being assigned in the constructor and in `readResponseGetCode` without a declared property, surfacing as "Creation of dynamic property seekquarry\yioop\library\SmtpClient::$last_response_body is deprecated" on every mail send under PHP 8.5. A follow-up sweep should grep all classes for assigns to undeclared `$this->*` so we catch the rest before they hit users on 8.5 (the 8.2 deprecation is now noisy enough that one bare `mail-send` test surfaces it).

## Atto WebSite — non-blocking connection lifecycle

- 62. ✓ Make the `WebSite` TCP connection lifecycle fully non-blocking. Origin: observed bursty hangs on pollett.org, where `WebSite` serves TLS directly on 443 (no Apache in front). Root cause: the single-threaded accept/serve loop blocked whenever one client stalled during a synchronous step (TLS handshake, protocol detection, or a frame read), freezing every other connection. Surfaced and fixed in the 2026-06-07 session. Verified in production with two standalone client probes (`diag_accept_blocking.php`, `diag_h2_frame_stall.php`) that stall N connections at a chosen lifecycle stage and time a legitimate concurrent request. Sub-stages, each its own verified patch on `master`: a. Stage 1 — non-blocking accept + deferred TLS handshake. Ported MailSite's deferred-handshake pattern: `accept` sets the socket non-blocking immediately and returns secure connections in a `handshake` protocol state with a deadline; `stepHandshake` peeks for the `0x16` TLS record, steps `stream_socket_enable_crypto` across select wakes (done / failed / needs-more-IO), and promotes on completion. `HANDSHAKE_TIMEOUT` on `ConnectionAcceptor`; `cullDeadStreams` reaps pending handshakes by deadline (they have no modified-time, so the generic idle timeout would wrongly reap them instantly). Production: phase 3 went from FAILED + an `SSL: Broken pipe` storm to ~0.9x baseline. b. Stage 2 — deferred post-TLS protocol detection. The post-handshake HTTP/2-magic read was itself blocking (a client that finished TLS but sent no bytes blocked). `classifySecureFirstBytes` accumulates the first decrypted bytes in a per-connection detect buffer and decides h1/h2 non-blocking. Production ~1.0x. c. Stage 3a — bound the select wait by handshake deadline. With no timer alarms the main `stream_select` used a null (infinite) timeout, so a silent stalled handshake was only reaped when unrelated traffic woke the loop. Added `nextHandshakeDeadline()` and clamp the wait to the soonest pending deadline. Production ~0.9x. d. Stage 3b — non-blocking WebSocket frame reads. `parseWsRequest` had set the socket blocking and looped on blocking reads, so a slow WS client dribbling a partial frame blocked the loop. Replaced with a per-connection input buffer plus a pure `WebSocketFrame::parseBuffer` (returns frame + bytes consumed, or null for need-more); per-frame handling moved to `dispatchWsFrame`. Public WebSocket route API (`onMessage` / `onClose` / `onError` / `send` / `sendBinary` / `close`, and `WebSocketFrame::build`) unchanged, so the atto WebSocket example still works. Verified with a real socketpair: a frame split across 3 reads buffers without blocking and dispatches only when complete; two frames in one read both dispatch. e. Stage 3c — non-blocking HTTP/2 frame reads. Evidence first: `diag_h2_frame_stall.php` established real H2-over-TLS connections (full preface + SETTINGS) then stalled mid-frame; pre-fix this measured phase 3 at 146x baseline (838 ms) with only 2 stall connections openable before the loop choked. Fixed by a low-risk split: the ~190-line frame-processing body became `processH2Frame` (byte-for-byte unchanged), and a new `parseH2Request` reads available bytes non-blocking into a per-connection input buffer, extracts each complete frame (9-byte header + declared payload), and calls `processH2Frame` per frame, draining multiple buffered frames in one pass. Verified a real HPACK-encoded HEADERS request dispatches both whole and when fragmented across reads. Production: phase 3 dropped from 146x to ~0.4x baseline, with all 20 stall connections opening cleanly. f. Stage 4 — H3Listener review (no code). Code-reviewed the QUIC/HTTP-3 path: it is non-blocking by design — a single UDP socket multiplexes all connections by connection ID, `accept` drains datagrams with non-blocking `stream_socket_recvfrom`, per-datagram processing is bounded, and `reapStaleConnections` does RFC 9000 idle-timeout reaping, so one stalled H3 client cannot block others. The only blocking read in the file (`Tls13Engine::fillExact` / `readRecord`) is explicitly off the QUIC path (a standalone TLS-over-TCP helper per its own comment). No change needed.

      *Open follow-up (optional, not blocking): `initH2Request`'s
      read of the client's initial SETTINGS frame at H2
      connection-establishment is still blocking. It is a distinct,
      narrow window (only during H2 setup) and is not what the
      diagnostic measured; converting it would mean deferring
      establishment across select wakes like the handshake detect
      phase. The two diagnostics are worth keeping as regression
      checks. Note observed during testing: phase-3 latency spikes
      can also come from HDD-under-load contention on response
      serving (filesystem latency, independent of loop blocking) —
      re-run a few times and compare typical figures, not a single
      worst case.*

## Production reliability & streaming (atto WebSite / MailSite)

Production-driven work outside the originally planned phases, recorded as numbered items here and grouped under "Phase 2.5 — Ongoing production reliability" in the phased ordering below.

- 64. ✓ ACME certificate auto-renewal (2026-06-08). `AcmeRenewJob` MediaJob, ARI-gated via `AcmeManager::renewalWindow` with a `ONE_MONTH` expiry fallback, plus in-place `WebSite::reloadSecureCertificate` driven by a once-a-minute cert-watch timer in `index.php`, so the secure listeners pick up a renewed certificate without a restart.

- 65. ✓ Per-domain Appearance (2026-06-09/10). Each custom-route domain can override the full Appearance set (colors, logos, favicon, background image, theme, landing page, searchbar), with a global fallback when unset. New `DOMAIN_APPEARANCE` table (migration 104), overlaid at render time, edited through a domain picker in the Appearance activity; per-domain images live in `resources/<domain>/` served via a new pretty-URL route and applied across all 13 logo/favicon surfaces through a new `LogoHelper`.

- 66. ✓ Restart Server made reliable (2026-06-13). Restart was flaky — daemons sometimes killed-not-relaunched, the web server sometimes killed-not-respawned. Root cause: the replacement secure listener's bind-retry window (10 × 250 ms) was shorter than the exiting TLS server's privileged-port teardown, so the replacement exhausted its retries and exited, leaving no server; `BIND_ATTEMPTS` widened 10→40 in `WebSite`. Supporting fixes:

  - `interruptibleSleep($duration, $step, $wake_callback)` in `Utility.php`, with idle daemon waits in MediaUpdater / Mirror / QueueServer / Fetcher converted to wake promptly on a stop request instead of sleeping out a fixed interval.

  - `CrawlDaemon` lock files now store `"timestamp pid"`; `isProcessAlive` (POSIX `posix_kill($pid, 0)`) lets `start()` clear a stale lock and proceed while never killing a live daemon — removes the manual `rm`-the-lock footgun. `statuses()` skips stale locks.

  - A non-blocking boot-drain state machine in `index.php`: a 1-second repeating timer advances one daemon per tick through drain/relaunch, replacing a blocking `sleep` loop that had frozen the freshly-restarted event loop for seconds per daemon.

  - A poll-based restart-status page (h2/h3-safe, replacing a streaming page) with phase wording, including the `&amp;`-in-URL and CSRF-freshness fixes that had made the poll receive HTML instead of JSON.

  - The configure-extensions check gained OpenSSL + Sodium (required) and POSIX (optional — enables graceful daemon shutdown and stale-lock clearing).

  - Side fix: a Media-Source Add/Edit form showed a phantom unlabelled field because the show/hide JS referenced a nonexistent `path-terms` id instead of the real `path-regex`. Open follow-up: re-verify restart consistency across several back-to-back restarts (a clean post-restart `ps` was observed — web server present with the normal `secure-80-443 … PreviousSession.txt` launch arg, daemons idle, no spin); if the earlier intermittent lingering-old-process or missing-replacement outcomes recur, the next suspect is the boot-drain / relaunch state machine rather than the bind window.

- 67. ✓ HTTP/2 streaming delivery — window-sized spans (2026-06-13). Large media and images stalled in Firefox under H2: the transfer died at ~131 KB (`NS_ERROR_NET_PARTIAL_TRANSFER`), then re-requested in a cascade. Root cause: the server committed a whole large body to one stream and depended on the client extending that stream's flow-control window; Firefox advertises the default 65535 per-stream window and withholds per-stream `WINDOW_UPDATE` for a body it is not actively draining (the connection window stayed at megabytes — only the per-stream window was pinned at 0). H1 has no per-stream flow control (works); Safari advertises a 2 MB window (works). Fix: a `WebSite::peerStreamWindow()` accessor exposes the connection's per-stream initial window, and `ResourceController` serves each span — byte-range and large whole-file — at `min(MAX_RANGE_STREAM_LEN, that window)`, so every stream completes within the credit the client granted and closes, the client fetching the next span the way it does against a conventional server. On a 2 MB-window connection a response goes out in one or two large spans (no slowdown); on a 65535-window connection it serves ~64 KB spans. Interim hardening kept as correct on its own: park-a-flow-blocked-stream-instead-of-resetting (no stall-RST during paced playback) and holding an early per-stream `WINDOW_UPDATE` that arrives before the response registers the stream. Behaviour note: a large whole-file GET (no Range header) now returns a capped `206 Partial Content` + `Accept-Ranges` so the client range-fetches the rest — browsers and media elements handle this; spot-check any non-browser client that specifically expects a `200` for a direct large-file download. Follow-up resolved 2026-06-13 (commit `02f0561`, "large download full 200 on h1_1"): over HTTP/1.1 a large whole-file GET (no Range header) now serves a full `200 OK` streamed in chunks rather than the capped `206`, so non-browser clients that expect a `200` work; the window-sized-span `206` path stays for H2/H3 where the per-stream flow window requires it. Closes the spot-check above.

- 68. ✓ Event-loop CPU spins (2026-06-13). `MailSite` ran a forked child at ~100% CPU: the select loop added every in-progress TLS-handshake socket to the `stream_select` write set, and a connected socket is almost always writable, so select returned immediately every iteration while any handshake was pending — fixed by leaving handshake sockets in the read set only (a "would block" crypto step is waiting to read the peer's next record; a stalled handshake is bounded by its deadline in `cullDeadStreams`). atto's `GopherSite` spun at ~100% from a reversed timer subtraction (`microtime(true) - $next_alarm[0]` → negative → clamped to a 0 select timeout) plus an always-zero fractional-seconds term; both fixed (atto tag `v2.2.1`). The same fractional-seconds bug in `WebSite` (correct operand order, so no continuous spin, but sub-second timer waits collapsed to a 0 timeout) was fixed in passing; `MailSite`'s own timer math was already correct. `Fetcher` / `QueueServer` / `MediaUpdater` / `Mirror` were checked and are unaffected — sleep-based loops with no `stream_select` timeout arithmetic. Reviewed and intentionally not changed: `H3Listener` does not need the streaming or handshake fixes — it already paces bodies incrementally with large (16 MB) default per-stream windows, and QUIC does TLS in userspace over UDP so the `stream_select` handshake-spin has no analogue. Backport status: `MailSite` and `WebSite` work is backported to atto (`cpollett/atto`); the streaming span-sizing logic is Yioop-side (`ResourceController`) and does not backport.

- 69. ✓ First-run & single-secure-port startup robustness (2026-06-13). A freshly-cloned, not-yet-configured instance can now be started and brought up cleanly. Pieces: the crawl daemon survives an unconfigured WORK_DIRECTORY instead of dying on launch (`CrawlDaemon` lets the process live and warns rather than fatal, commit `e67c3c2`); a single-secure-port mode (just 443, self-signed cert auto-written) in `index.php` (`e1fc4fd`); maximum request length derived from PHP's `post_max_size` so large uploads are not silently truncated (`c180065`); the first-run Configure→running handoff polls a real restart-status URL built through `controllerUrl` with first-run constants in `Config.php`, replacing a streaming page (`70a7f97`, `c3eb12f`, `0d25e11`); and a guard that warns when a second index server is started with a port already in use (`ada3e8f`). This is the headless/browser bring-up path referenced in the work-directory setup notes.

- 70. ✓ Command-line `restart` / `status` for the running server (2026-06-13/14). `index.php` gained CLI `restart` and `status` subcommands so the live server can be restarted and inspected from a shell without the web UI: restart routes through the same restart-server path the UI uses (`66738f1`), monitors restart progress and reports phase status (`283d957`), names the services it relaunched (`241e740`), a `status` command summarises the web server + daemons (`236735f`), and bare/`--help` invocation prints usage (`0f63160`). Supporting fix: `CrawlDaemon::isProcessAlive` treats an `EPERM` from `posix_kill($pid, 0)` as alive-but-other- user rather than dead (`795957b`), so process-liveness is correct when the web user differs from the file owner — the same cross-user (`_www`/`www-data` vs the owning account) hazard the resource-UI Apache issue surfaced. Builds on item 66.

- 71. ✓ Headless `ConfigureTool` command-line mode (2026-06-13). `src/configs/ConfigureTool.php` gained a non-interactive one-command-per-invocation mode alongside the existing interactive `menu`, so a fresh install is configurable with no browser: `work-dir [path]`, `root-password <old> <new>`, `locale <tag>`, and the toggle commands `debug` / `search-access` / `page-elements` (with `+name`/`-name` to turn settings on/off), plus `name-server` and `robot`, each with an argument spec, count checking, and a usage listing (`71dc944`, `c9606ae`, `df25cdf`, `12c71cd`, `aaf8f3b`; `readInput`/`readPassword` consolidated in `Utility.php`). This is the seam used to stand up a working WORK_DIRECTORY + `Profile.php` + the two SQLite DBs headlessly (e.g. so `CodeTool` unit tests can run). Document in the install guide under item 34.

- 72. ✓ Page source/history/list visibility + per-domain robots.txt / humans.txt (2026-06-16/17). Two related controls built on the per-domain Appearance foundation (item 65) and the group OPTIONS bitfield. (a) Visibility: `SOCIAL_GROUPS.OPTIONS` was refactored from a packed `(render_engine<<1)+encryption` into a pure flag bitfield, with `render_engine` split out to its own `RENDER_ENGINE` column and new flags `GROUP_OPTION_PAGE_SOURCE_ALLOWED` / `GROUP_OPTION_PAGE_LIST_ALLOWED` (migrations v106 / v107). A group owner can now keep the public — and crawlers — out of wiki page source and history (one master gating both: the page-level `public_source` head var, the history endpoint, the source/discuss/history nav, and the discussion-thread feed all honor it) and out of the public page list. Owners stay exempt. (b) robots / humans: each web-server secure domain publishes its own files. `RobotsTxtGenerator` (a pure library class) builds robots.txt from per-domain choices stored in the `DOMAIN_APPEARANCE` blob (disallow -all, a crawl delay with a None option, per-category crawlable toggles for search / feeds / wiki with wiki sub-toggles for page source / feed / list, plus free-form additional directives); humans.txt is free-form text. `robots.txt` / `humans.txt` routes in `src/index.php` serve the requesting host's stored text (falling back to the built-in default), mirroring the `.well-known` / mta-sts route pattern; the old static `src/robots.txt` was removed. Edited through a new `RobotssettingsElement` page reached from each domain's Server-Settings routing panel, with a live Preview. Commits `af06edee`, `d4a22b44`, `7abf188a`, `73b76b77`, `b2c8b7e8`, `f70ded84`, `87ec0bd0`, `c842350`.

## Suggested phased ordering

Phase 0 — Code hygiene + immediate hot-fixes (complete)

- ✓ 40 (extend `CodeTool needsdocs`)

- ✓ 41 (sweep and add the missing docblocks the tool surfaces; 0 findings remaining across `src/`. Tests tree also covered: commit `048ce3a` swept docblocks across `tests/` to bring the unit-test suite up to the same standard the production tree now satisfies.)

- ✓ 31 partial: `crawlAuthHash` sibling added and CSRF token migrated to it; `checkCSRFToken` now uses `hash_equals` for the comparison and validates the new 54-char token layout; unit test for the new primitive in `UtilityTest::crawlAuthHashTestCase`. Indexing callsites of `crawlHash` untouched as planned.

  ✓ 31 partial: the security-relevant `crawlHash(AUTH_KEY . x)` callsites (secret-ballot FORM_HASH, csv_form_hash, page_hash, and the git app-code token) migrated to `crawlAuthHash`; the CSV form fingerprint strengthened from the 8-byte `crawlHash` fold to `hash("sha256", ...)`; AUTH_KEY now generated from `random_bytes`; the ballot verify and CSRF check compare with `hash_equals`. Two lingering `mt_rand` git-archive temp names converted to `random_int`. Arc-plan 4.1.a (crypto) and 4.1.b (constant-time) done; indexing callsites left on `crawlHash`.

  ✓ 31 partial: arc-plan 4.1.c (drop dead crypto fallbacks) — `crawlCrypt` was already on `random_bytes`, and the last two stray `mt_rand` git-archive temp names went to `random_int`. 4.1.d (cookie and header hardening) — session cookie set `HttpOnly`, `SameSite=Lax`, and `Secure` over HTTPS; CSP `frame-ancestors` and `nosniff` were already central. The cookie and header hardening lands in Phase 4.1, not Phase 5.

  ✓ 31 partial: arc-plan 4.1.d (response and cookie hardening) done — session cookie HttpOnly, SameSite, and Secure over HTTPS. 4.1.e (input and output paths) spot-checked: inputs go through Controller::clean, no view echoes a raw request value, upload base names use pathinfo basename, and resources are served read-only (readfile/fread, never include/eval) so uploaded scripts do not run; the one gap, a missing "`..`" strip in the resource-folder write path, was closed.

- ✓ 33(a): new installs default to `EMAIL_RECOVERY`. Surgical fix in `ProfileModel::updateProfile` — adds `RECOVERY_MODE` to `$not_null_fields`, plus a `shouldKeepZero` policy helper so an admin's deliberate `NO_RECOVERY = 0` setting survives saves. New `tests/ProfileModelTest.php` with 8 assertions.

- ✓ 14 (rename `MailServer` → `BulkEmailClient` and its file; callers, test, autoload-by-namespace all updated)

- ✓ 12 partial: Message-ID, MIME-Version, Content-Type, Content-Transfer-Encoding added to both branches of `MailServer::sendImmediate` (PHP `mail()` and the SMTP path). Helper `MailServer::messageIdDomain` factored out and unit- tested. Full DKIM / List-Unsubscribe / Reply-To / Return-Path still deferred to phase 2.

- ✓ 24 partial: `fennec` UA token dropped from the mobile sniff in `src/index.php` (commit `c590f5d`; Fennec was discontinued in 2020 when Mozilla replaced it with Fenix). Full Sec-CH-UA-Mobile migration with `Accept-CH` header is the remaining piece, deferred to phase 5.

- ✓ Ad-hoc: bumped `session.gc_maxlifetime` to match `AUTOLOGOUT` in `src/index.php` (commit `9557889`, `early-logout-fix` tarball). Default PHP `gc_maxlifetime` is 1440 seconds (24 minutes), which was triggering early logouts for users idle longer than that even though Yioop's intended session timeout per `AUTOLOGOUT` is much longer. The fix sets `session.gc_maxlifetime` and `session.cookie_lifetime` to `AUTOLOGOUT` before `session_start()` so PHP's GC matches Yioop's policy. Surfaced while testing the EMAIL_RECOVERY default; not in the original phase 0 list. Note: this fixed the obvious case but random logouts still occur on at least one deployment, indicating an environment-level cause this per-request `ini_set` cannot reach. See deferred item below in Phase 3 for the deeper fix.

- ✓ Ad-hoc: PHP 8.5 forward-compat cleanups (commit `6e46da2`). Dropped deprecated calls to `imagedestroy()` (GC is automatic since PHP 8.0; the function emits an E_DEPRECATED in 8.5) and to `xml_parser_free()` (same story). Replaced one `list($a, $b) = ...` destructure that no longer parsed cleanly under 8.5's stricter list grammar with the bracketed `[$a, $b] = ...` form. Surfaced when running yioop.com on 8.5 to validate the PHP 8.1 floor wasn't accidentally too low; the same fixes are forward-compat work for whenever the floor rises further.

- ✓ Ad-hoc: dropped empty `HiTokenizer::partsOfSpeechTestCase` scaffolding (commit `cc3776f`). The Hindi tokenizer's test class had a method named `partsOfSpeechTestCase` with no assertions — leftover scaffolding from a never-implemented POS-tagging feature. Removed rather than left as a 0-assertion passing case that misleads about coverage.

Phase 1 — Codebase floor (complete)

- ✓ 38 (PHP minimum pinned at 8.1; composer.json / INSTALL.txt / SystemComponent runtime check / 25 locale strings reconciled; 6 pre-existing locale translation bugs fixed in passing — commit `a1cd8f5`)

- ✓ 31 partial (`crawlCrypt` fallbacks for `mcrypt_create_iv` and `mt_rand` dropped now that PHP 8.1 guarantees `random_bytes` — commit `1630deb`. All 10 remaining `mt_rand` sites tree-wide converted to `random_int` for defense in depth — commit `924ad9e`. Two safety guards added: `AdvertisementModel` now early-returns `[]` when `TOTAL_AMOUNT < 1` instead of silently calling `mt_rand(1, 0)`; atto `WebSite` LRU eviction now skips when `$num_unmarked == 0` instead of silently passing `(0, -1)`. Both were pre-existing edge cases that `mt_rand` masked but `random_int` would `throw` on. The broader item 31 (full security audit) is still pending for phase 4.)

- ✓ Ad-hoc: locale backslash-corruption fix. Surfaced while doing the PHP-floor locale-string sweep: `LocaleModel::update Translation` (line 783) was unconditionally applying `addslashes` on every save, so any string with an apostrophe doubled its backslash count on every round-trip through Manage Locales. Some strings had accumulated hundreds of backslashes. Four commits: `09165ed` replaces the writer's `addslashes` with a proper `iniEscape` helper (root-cause fix); `83819ca` re-emits every `configure.ini` through the new path (497 lines across 11 files; pure encoding normalization, semantically transparent); `f05b751` repairs 31 cross-contaminated keys across 7 locales (de had Spanish strings, he had Japanese strings, pt was on PHP 7.1 from a stale machine-translation pass, etc.) and resets `he` to the en_US fallback for a native-speaker pass later (item 42); `8d7b816` strips the runaway escape chains that the normalization preserved (467 lines; `\\{2,}'` pattern collapses to `'`, audit-verified safe against legit regex content like `vi_VN`'s `\b` word-boundary). Cleared 518 corruption sites total. Pre-requisite for the broader item-42 translation audit.

- ✓ Ad-hoc: `strpos`/`substr` → `str_contains`/`str_starts_with`/ `str_ends_with` (145 sites across 49 files; commit `139f9e7`). Word-boundary `\bstrpos\b` anchor prevents accidental `mb_strpos` matches; paren-aware non-greedy capture avoids the cross-line matching trap; escape-aware length check for substr literals (`"\r\n"` with N=2 converts correctly). Skips `stripos` (semantic change without `mb_strtolower`) and `($pos = strpos(...)) !== false` assignment-in-condition forms (need the position).

- ✓ Ad-hoc: `switch` → `match` for simple-expression arms (10 sites across 11 files; commit `c4f07c1`). Audited all 96 switches; only 10 had value-returning arms suitable for `match`. PhraseModel letter table, nl/it/en_US Tokenizer vowel/cons, NWordGrams 4-arm via list-destructure, GroupModel EXIF orientation (preserves no-op default), Utility CSS unit conversion, PdfProcessor 2× char maps, LocaleModel direction, PartialZipArchive decompression. Two follow-ups (`f28cde9`, `a5fb3f3`, `7b40335`) collapsed column-aligned `=>` per the fragile-layout rule in code I wrote, plus a sweep of the atto files (back-ported to upstream atto).

- ✓ Ad-hoc: nullsafe `?->` for explicit null guards. 11 sites total across 4 files in 2 commits (`0bf2054`, `9141f29`). `if ($x) { $x->m(); }` → `$x?->m();` for fire-and-forget calls; `$x !== null ? $x->y : null` → `$x?->y`; `$x !== null && $x->y` → `$x?->y` (preserving falsy short-circuit semantics). All changes contained to atto WebSite / FeedArchiveBundle / FeedDocumentBundle / MailSite. Wider scan found no further clean candidates.

- ✓ Ad-hoc: constructor promotion across 45 classes. 4 commits (`f6dc46a`, `628dc49`, `4efe97b`, `e7963c3`). Batch 1 (8 classes): pure-assignment constructors where param names matched property names — StaticTurnAuthenticator, Element, Transport, Flag, Connection, ConnectionAcceptor, Listener, QuicStream. Batch 2 (14): mixed-assignment constructors (some params promoted, body keeps non-trivial setup) — Frame subclasses (HeaderFrame, DataFrame, RstStreamFrame, PushPromiseFrame, PingFrame, GoAwayFrame, WindowUpdateFrame, ContinuationFrame), WebSocket, FileAuthenticator, StringArray, HashTable, PriorityQueue (also removed redundant `$num_values` declaration that shadowed parent), SparseMatrix, MediaJob, LRUCache, Trie, SuffixTree, PageIterator, View. Batch 3 (15): Frame, SettingsFrame, H3Connection, BulkEmailClient, DocIterator, WordIterator, GroupIterator, MixArchiveBundleIterator, IndexShard, LSMTree, Tier, IndexDictionary, MessagesBundle, FileCache, VersionManager, Model. Batch 4 (8): WebSite (body restructured so `$this->base_path` is updated in place by the conditional default logic), WebArchiveBundle, IndexArchiveBundle, CrawlQueueBundle, DoubleIndexBundle (also fixed the PHP-8.2 dynamic-property deprecation for `$dir_name`), BloomFilterBundle, NetworkIterator, Bzip2BlockIterator. Skipped: name-mismatch cases (Component, Layout, WebSite Listener at line 6725) that would force visible API renames; constructors where the param is transformed before assignment (ContextTagger's `$lang`, StochasticTermSegmenter's `$lang`); cases where the param shadows a parent property (TextArchiveBundleIterator, DatabaseBundleIterator). Net effect: ~600 fewer lines across src/library, constructors now self-documenting.

- ✓ Ad-hoc: removed unused `Padding` trait from atto WebSite. Commit `36b6597` — 57 lines deleted. The trait had no `use Padding;` consumers anywhere; PADDED-flag handling lives inline in HeaderFrame/DataFrame `parseBody` methods. Pre-existing dead code from the v9.5.3 baseline AI-generated atto scaffolding. Back-ported to upstream atto.

- ✓ Ad-hoc: int-backed enums for QuicStream state machines. Commit `4728703`. Two file-scope enums in `H3Listener.php`: `RecvState` (Open=0, SizeKnown=1, Done=2, Reset=3) and `SendState` (Ready=0, Data=1, DataSent=2, Done=3, Reset=4). Replaces 9 untyped int constants on QuicStream; typed properties `public RecvState $recv_state` and `public SendState $send_state`; 12 internal references updated. Previously `RECV_OPEN = 0` and `SEND_READY = 0` were interchangeable ints; now distinct types catch cross-channel mix-ups at runtime. Surveyed wider codebase for enum candidates — CrawlConstants is too tightly coupled (inherited via `implements` by 56 classes) and other constant clusters are config values, not closed sets. QuicStream is the one clean win.

- ✓ 39 — Coding.pdf / Coding-guidelines wiki rewrite for the PHP 8.1 floor. Surgical scope: dropped the PHP-5.4-compat rule (PHP item 11) and replaced with an explicit PHP 8.1 floor plus prefer-modern-form guidance covering match, str_contains/starts/ ends, nullsafe ?->, constructor promotion, enums, and random_int. Updated PSR-12 reference to also cite PER Coding Style 3.0 (the PHP-8.x extension of PSR-12). Bumped copyright year in the example license block from 2023 to 2026 (two sites). Replaced the .html-ltr / .html-rtl mandate with logical-property guidance (preview of item 27) while keeping the legacy selectors documented as still-supported. Added a parallel "prefer media / container queries" note alongside the .mobile selector (preview of items 23-25). Added a new Security section between CSS and HTML codifying: no mt_rand for security-relevant values; explicit range guards before random_int when bounds derive from user-influenced data; hash_equals for secret comparisons; HMAC over server-side secret for auth tokens (not raw hash); password_hash / password_verify only via crawlCrypt; no @ error suppression; HTML/URL/SQL escaping rules; clean() the request supergloblals; CSRF token verification for state-changing endpoints; do not log credentials. Deliverable: full revised wiki source as `Coding-wiki-revised.txt` — pasted whole-file into the wiki editor; the wiki's built-in diff preserves the old version. Re-render Coding.pdf from the revised wiki page as the final step.

Phase 2 — Mail infrastructure (complete; the optional `mail-backend-search` handler migration and the item-13 outbound trim remain, plus item 12 below; item 11 is done and the deliverability docs moved to Phase 11; the patch-12 visibility cleanup and the `MAIL_CLONE_JOB.LAST_ERROR` column drop are both done — verified in code 2026-06-07) (Update 2026-06-14: the item-13 outbound trim is now done; only the optional `mail-backend-search` handler migration and item 12's RFC-5322/DKIM/List-Unsubscribe header work remain in this phase.) (Update 2026-06-19: item 12 is now done too (verified in code) — the email subscribe/unsubscribe arc landed the List-Unsubscribe headers on top of the existing `MimeMessage::build` (RFC 5322) and `SmtpClient::dkimSign` (DKIM); see item 12 above. Only the optional `mail-backend-search` handler migration remains open in this phase.) (Update 2026-06-19: Phase 2 is now COMPLETE. Its one remaining item, the optional `mail-backend-search` webmail search UI, is deferred Post-v10 — the backend primitive is already done (`MailBackend::search`, implemented by both `MailSiteMailBackend` and `ImapMailBackend` over the storage `searchMessages`); only the webmail search box, the `userMailSearch` handler, and results rendering (~150 lines) remain. It now lives with the other MailSite webmail follow-ups (54/56/57/58) under Post-v10.)

Step 1 (config-only, no behaviour change):

- ✓ 49 — Mail Services fieldset in Server Settings

Step 2 (UI affordance, no listener):

- ✓ 50 — SocialcontrolsElement Mail button + userMail activity stub

Step 3 (external IMAP first, validates the UI):

- ✓ 51 — ImapClient class
- ✓ 52 — Mail activity: external-IMAP accounts UI

Step 4 (fill MailSite storage from disk, still no listener):

- ✓ 48 — Mail import CLI tool (closed 2026-06-10). Streaming/resumable/idempotent import, per-folder subscribe + flags-snapshot priming, per-folder UIDs, progress + timestamps; used to migrate the production `cpollett` mailbox. Both Maildir and mbox sources land; mbox `export` added. The tool was trimmed to its durable command set `import`/`export`/`verify` — the pre-production-only `migrate`/`reuid`/`cleanup` subcommands (and their `moveDirFiles`/`cleanupFolder` helpers) were removed as one-time states no current or future install reaches. Optional live-IMAP source deferred (post-v10 nicety). See the status note under item 48.

Step 5 (MailSite-stored mail in the Mail activity):

- ✓ 53 — Mail activity: MailSite-stored mail (12-patch migration arc; details under item 53 above. The optional `mail-backend-search` handler migration (patch j) is still open. The patch-12 visibility cleanup (patch i) and the `MAIL_CLONE_JOB.LAST_ERROR` column drop are done — the drop ships as `upgradeDatabaseVersion96` (table-rebuild with a probe no-op guard), errors moved to the `MAIL_CLONE_ERROR` table in v95, fresh schema in `ProfileModel` omits the column, and `DATABASE_VERSION` is 102 so the migration runs; no code reads/writes the dropped column.)

Step 6 (the actual listener):

- ✓ 10 (Manage-Machines controls for MailSite: on/off, ports, status, logging — actually start/stop the listener; wired in `MachineController` `statuses` and `log` paths under the `MailServer` case).

Step 7+ (everything else):

- ✓ 55 (MailSite outbound sending — landed during phase 2 ahead of the post-v10 schedule. Local recipients delivered via `MailSite::deliverMail`; external recipients delivered as peer MTA via `SmtpClient` with per-domain MX resolution and port 25→587 fallback; opportunistic STARTTLS on the outbound leg; sent copy archived to Sent via the MailSite backend).

- ✓ 59 (Test Mail Mode + Test Mode Fallback Port: relaxes compose-form validation, skips MailSite FQDN-reject hook, enables an optional outbound port-25 fallback to a tunnel forwarder; replaces an earlier speculative 25→587 fallback).

- ✓ 60 (`smtp_tunnel.php` standalone byte-level TCP forwarder for reaching port 25 from a network egress that blocks it; lives outside the Yioop tree).

- ✓ 61 (PHP 8.5 dynamic-property declaration fix on `SmtpClient::$last_response_body`).

- ✓ 13 (refactor outbound mail to share construction code with MailSite when local-delivery applies; closed 2026-06-14 — `SmtpClient::sendImmediate` builds via `MimeMessage::build` and SmtpClient is trimmed to the wire exchange; see item 13).

- ✓ 12 full (RFC 5322 / DKIM / List-Unsubscribe headers) — done; verified in code 2026-06-19. RFC 5322 message construction is `MimeMessage::build`; DKIM signing is `SmtpClient::dkimSign` wrapped around that built message in `sendImmediate`; the List-Unsubscribe + List-Unsubscribe-Post headers come from `UnsubscribeToken::listUnsubscribeHeaders` / `listUnsubscribeHeadersForEmail`, attached on group/bulk send (`SocialComponent`) and on registration mail (`RegisterController`). Delivered by the 2026-06-17 email subscribe/unsubscribe arc (see `devlog/v10/2026-06-17 Email List Subscribe Unsubscribe/plan.html`), which also added the `MAIL_SUPPRESSION` store + send-path skip, the web unsubscribe endpoint (`ApiController::unsubscribe` + `UnsubscribeView`), the registration suppression gate + whole-site signed token with its forgery guard, and the `bot` mailbox reader (`BulkEmailJob::processBotMailbox` / `applyUnsubscribeToken`) that acts on the mailto form of the header. Still open from that arc are reader robustness (transient reconnect+retry, per-UID ratchet) and account aliases; mail-account drag-drop reorder was already done (verified 2026-06-07).

- ✓ 11 (webmail interface built — folders, list, read/compose/reply, flags, move/delete via MailBackend; further enhancements filed individually).

- ✓ 14 (rename `MailServer` → `SmtpClient`; done in phase 0).

Phase 2.5 — Ongoing production reliability (complete to date; production-driven, runs alongside the planned phases as live issues surface). Full per-item detail in the "Production reliability & streaming" section above; the sub-blocks below carry the finer-grained hot-fix detail.

- ✓ 62 — `WebSite` TCP connection lifecycle made fully non-blocking (accept + TLS handshake, post-TLS protocol detection, WebSocket frame reads, HTTP/2 frame reads; H3/QUIC reviewed and already non-blocking). Arose from the observed pollett.org bursty-hang bug. (2026-06-07)

Production hot-fixes (2026-06-08, pollett.org) — each shipped as a single-commit forward patch (see `yioop-acme-prod-and-bugs-handoff.md`):

- ✓ 64 — ACME certificate auto-renewal: `AcmeRenewJob` MediaJob (ARI-gated via `AcmeManager::renewalWindow`, ONE_MONTH expiry fallback) + in-place `WebSite::reloadSecureCertificate` on a once-a-minute cert-watch timer in `index.php`.

- ✓ Manage-Machines under the secure server: `WebSite` `processInternalRequest` inherits HTTPS/HOST/PORT so internal machine self-calls no longer 301-redirect to `https://0.0.0.0`.

- ✓ Secure-domains apex suggestion strips leading `mta-sts.`/`www.`; `MailSiteFactory::suggestedDnsRecords` skips `www.`/`mta-sts.` hosts whose apex is already listed.

- ✓ 63 — Mail Banner / `MAIL_HOST_NAME` + suggested PTR.

- ✓ ResourceController large-file serving: full-file and `serveRangeRequest` loops drain each block through the atto streaming flush (H2 DATA frame / H1 chunk) instead of the global flush the atto buffer ignores — fixes OOM serving a ~696 MB .mp4; `WebSite::writeStreamingHead_H1` strips Content-Length when forcing chunked; plus removal of the dead `if ($start = 0)` range-bound bug.

Streaming engine + features (2026-06-09/10):

- ✓ HTTP/2 + HTTP/3 streaming engine hardening: RFC 7540 outbound send-window engine, slice-bounded interleave, busy-spin/OOM drain-check fix, HPACK curly-apostrophe crash fix; H3 connection-level flow control + handshake-over-TCP complete-write loop.

- ✓ 65 — Per-domain Appearance.

Restart, delivery, spins (2026-06-13):

- ✓ 66 — Restart Server reliability (bind-retry window + supporting daemon/boot-drain/status fixes).

- ✓ 67 — HTTP/2 window-sized span delivery.

- ✓ 68 — Event-loop CPU spins (MailSite handshake, GopherSite + WebSite timer math).

Startup, sessions, CLI, resource UI (2026-06-13/15):

- ✓ 69 — First-run & single-secure-port startup robustness (daemon survives an unconfigured work directory; single 443 port mode; max-request-len from `post_max_size`; first-run restart status URL; warn on a second index server with an in-use port).

- ✓ 70 — Command-line `restart` / `status` for the running server (+ cross-user `isProcessAlive` EPERM fix; builds on 66).

- ✓ 71 — Headless `ConfigureTool` CLI (`work-dir`, `root-password`, `locale`, `debug`/`search-access`/`page-elements` toggles, `name-server`, `robot`) — configures a fresh install with no browser.

- ✓ 47 — Database-backed session restore (also the Phase 3 random-logout item): `SESSION_USER` table + in-memory LRU heap with DB rehydrate, DB version 104→105, so sessions survive a server restart / memory eviction.

- ✓ 13 — Outbound mail construction dedup (`SmtpClient::sendImmediate` → `MimeMessage::build`; SmtpClient trimmed to the wire exchange).

- ✓ large whole-file download returns a full `200` over HTTP/1.1 (closes the item-67 spot-check).

- ▮ 17 — Page-resource folder UI: drag-drop, multifile/folder upload with progress, mail-style action toolbar, inline rename, parent-row move (incl. to root), single-resource delete, and the VersionManager stranded-lock root-cause fix all landed; multi-select + keyboard shortcuts and the CSS-Grid auto-fill rewrite remain (kept open under item 17 / Phase 7).

Group visibility + per-domain robots (2026-06-16/17):

- ✓ 72 — Group page source / history / list visibility master (OPTIONS bitfield + `RENDER_ENGINE` column split, migrations v106 / v107) and per-domain robots.txt / humans.txt (`RobotsTxtGenerator` + `DOMAIN_APPEARANCE` storage + `src/index.php` host routes + a Server-Settings edit page with Preview).

Phase 3 — Auth modernisation (complete)

- ✓ 32, 33(b), 33(c)
- ✓ 47 (database-backed session restore — landed 2026-06-14, logged under Phase 2.5; the per-session store also serves the "log out everyone" / email-one-time-login needs of 32-35)

Phase 3.5 — Cooperative atto Audit and MailSite Daemon (complete; 2026-06-20 → 2026-06-21)

- ✓ Audited the atto servers for non-blocking behaviour and made the MailSite daemon's reactor loop cooperative in four slices (the `WebSite` server was already cooperative from Phase 2.5 item 62). Full per-slice detail in the Phase 3.5 arc plan.
- ✓ 50, 54, 55 (mail UI / outbound items completed alongside this arc)

Phase 3.75 — Authentication Controls and Password Policy (complete; 2026-06-25 → 2026-06-29)

- ✓ Rename the Recovery fieldset to Authentication; add a Yioop / LDAP sign-in-method dropdown.
- ✓ Yioop method: configurable password-rule checkboxes + a minimum-length field.
- ✓ Shared, improved password-strength check (JS + server) on every password entry point.
- ✓ Move LDAP settings into the panel (suffix, base DN, host); `LDAP_LOCAL_USER` becomes a `SigninModel` method; one email per account.
- ✓ Root-email LDAP probe before committing the switch (no admin lockout).
- ✓ Account Recovery label + recovery dropdown below the auth controls; LDAP forces No User Password Recovery Link.

Phase 4 — Security audit + wiki regex (closed 2026-07-10; started 2026-06-29)

- ✓ 31, 16. Phase 4.1 (auth, cookie, and header hardening) landed; see above. Phase 4.2 (item 16) then covered a tokenizer and parser for both the mediawiki-style and markdown engines, with a golden regression corpus in place. The PHP parser is done for both engines, the old regex engine is gone, markdown has its own help page, and the design is ported to JavaScript (help.js) with per-case JavaScript unit tests. Phase 4 closed 2026-07-10. See the Phase 4 arc plan for the item breakdown and for the “Other work” delivered alongside it.

Phase 4.5 — Settings refactor + live-site fixes (in progress; started 2026-07-10)

- ☐ Three threads that do not belong in the Phase 5 layout work. (a) Live-site crashes: the remaining yioop.com out-of-memory attention beyond the Phase 4 defensive clamp, and a pollett.org MailServer.php crash. (b) Split the single Server Settings screen into two activities: Servers keeps the machine-level groups (Name Server, Database, Web Server, Mail Services, Proxy), and a new User, Roles, Groups screen takes Account Registration, Monetization, Bot Configuration, and Git Wiki Pages, plus two new groups — Role Group Limits (per-role caps on group count and size, an account's cap being the maximum over its roles) and Moderation (the group-moderation constants, including whether moderation runs at all). (c) Small interface fixes, starting with the Manage Crawl start-crawl [+] control whose accessible name renders stacked on three lines. Threads (a) and (b) are complete and (c) is nearly done, with the multi-domain hosting item the main piece remaining. See the Phase 4.5 arc plan.

Phase 5 — CSS foundation + viewport

- ☐ 26, 27, 28, 29, 30, 23, 25 (viewport meta lands here with the responsive work, not earlier)

Phase 6 — Chat refresh

- ☐ 1, 2, 6, 7, 8, 9

Phase 7 — Wiki UX

- ▮ 17 (page-resource folder UI — most of it landed 2026-06-14/15, logged under Phase 2.5; multi-select + keyboard shortcuts and the CSS-Grid auto-fill rewrite remain)

- ☐ 18, 19, 20, 21 (async secret-ballot witnesses), 37

Phase 8 — Calling

- ☐ 3, 4, 5

Phase 9 — Crawl / index health

- ☐ 15, 22, 43 (deprecate IndexShard read path; full removal may slip post-v10), 45 (decision on crawl mixes)

Phase 10 — Final docs pass + release prep

- ☐ 34, 36, 42 (localization audit + escape-bug fix), 44 (trim VersionFunctions to v9+)

- Item 36 (mail-deliverability docs) moved here 2026-06-19: it documents standards already delivered (DKIM/SPF/DMARC guidance, Banner / PTR), so it belongs in the final v10 documentation push rather than a phase of its own.

Phase 11 — Deliverability (the standards work is done — DKIM, SPF, DMARC, MTA-STS, Banner / `MAIL_HOST_NAME`, PTR / FCrDNS all shipped in Phase 2. Its one remaining piece was the documentation, item 36, which moved 2026-06-19 into the final docs push (Phase 10). This block is kept only as a record of what landed.)

- 36 (document mail deliverability requirements: DKIM, SPF, DMARC, Banner / `MAIL_HOST_NAME`, PTR / FCrDNS — refresh the PDF deliverability docs to match the wiki Mail Services page) — moved to the final docs push, Phase 10.

- Implementation prerequisite tracked under item 12 (RFC 5322 / DKIM signing / List-Unsubscribe headers, in Phase 2); document each piece here as it ships.

- Already landed toward this (groundwork, for reference): item 63 (Banner / `MAIL_HOST_NAME` host identity + suggested PTR row in the Suggested DNS Records panel; the wiki Mail Services page gained a Banner section and PTR note). The HELO/PTR/FCrDNS half of the deliverability story is in place; the RFC-5322/DKIM/List-Unsubscribe half (item 12) and the PDF doc refresh (item 36) remain.

- Log ongoing deliverability work as new bullets here as it lands.

Post-v10

- ☐ 46 (unified search)
- ✓ 54 (cross-account mail move) — landed during Phase 3.5
- ☐ 56 (MailSite spam classifier)
- ☐ 57 (MailSite unread-count caching)
- ☐ 58 (MailSite sort/filter parity with IMAP)

- ☐ mail-backend-search (webmail search UI) — deferred from Phase 2 on 2026-06-19. The backend is done (`MailBackend::search` in both backends, over the storage `searchMessages`); what remains is the webmail search box, a `userMailSearch` handler that turns matched uids into the same list/envelope structure `MailElement` renders, and routing/locale/test (~150 lines). Belongs with the MailSite webmail follow-ups (54/56/57/58) above.

- ☐ 43 follow-up (full IndexShard removal once the deprecation cycle has shipped)

- ☐ Mirror sync latent memory bug. `ResourceController::syncList` builds the whole delta file-list for a mirror request and serves it with `base64_encode(gzcompress(serialize($info)))` echoed whole, holding the entire list (and its base64 copy) in memory at once. On a large cache this is the same unbounded-memory exposure the main resource serve was hardened against (range / offset-limit / large-file streaming), so a mirror puller over a big cache could exhaust the process the way the HTTP/2 scrub did. Latent for now because pollett.org does not run a mirror. Either bound and stream the sync list (page the delta in fixed spans, like the resource serve) or, if the Mirror feature is being retired, remove it; surfaced 2026-06-23 during the HTTP/2 connection-memory work.

## Notes / cross-cutting

- Write unit tests as code lands. Per `Coding.pdf` Code-base Organization rule 10, every new `library/` file should ship with a sibling `*Test.php`. Run via `php executables/CodeTool.php unit` (lists), `unit ClassName` (run one class), or `unit ClassName methodName` (one method). Note: `UnitTest::setUp()` and `tearDown()` are abstract — must be implemented even when empty (skipping silently fatals out with no diagnostic), which masked an earlier test scaffolding attempt.

- MailSite test seam. `MailSiteFactory::setTestOverride($site)` replaces the factory's `build()` return value with a pre-configured `MailSite` instance (typically wired to `RamMailStorage`) so unit tests don't need a WORK_DIRECTORY. Tests must clear the override in `tearDown` to avoid bleeding into other tests. Used by `MailSiteMailBackendTest`.

- Stale-item verification (2026-06-07). A pass over a few "open" housekeeping items found several already done in the code, so the list had drifted. Confirmed already complete: (1) the dead clone locale key `mail_element_clone_errors_toggle` does not exist in the locale files — the live keys (`mail_element_clone_errors_none`, `mail_element_clone_error_resolved_via`, etc.) are all referenced in `SocialComponent` and `mailclone.js`; (2) the `MAIL_CLONE_JOB.LAST_ERROR` column drop (see Step 5 note); (3) mail-account drag-drop reorder — `reorderAccounts` handler in `SocialComponent` (with `userMailReorderAccounts` persisting the order). Lesson: verify "open" items against the code before scheduling work on them.

- Stale-item verification (2026-06-19). Second pass, same lesson. Item 12 (RFC 5322 / DKIM / List-Unsubscribe headers) was still shown `- [ ]` open but is fully done in code — RFC 5322 via `MimeMessage::build`, DKIM via `SmtpClient::dkimSign`, List-Unsubscribe via the 2026-06-17 email subscribe/unsubscribe arc; now checked off above. Re-confirmed already complete and not to be relisted: the dead clone locale key `mail_element_clone_errors_toggle` (absent from `src/`), the `MAIL_CLONE_JOB.LAST_ERROR` drop (`upgradeDatabaseVersion96`; `MAIL_SCHEDULED.LAST_ERROR` and `MAIL_CLONE_ERROR.MESSAGE` are unrelated false positives), and mail-account drag-drop reorder. Rule reinforced: this todo file is the canonical store, not chat memory; an item only "counts" as tracked once it is checked off here, so reconcile the todo against the code rather than re-raising done work.

- Phase-completion checkpoint. When we finish all items in a phase, Chris sends a fresh tarball of the project so Claude can verify its local working tree is in sync with origin/master before starting the next phase.

- `@return` tag style. Keep tags concise: type plus a brief prose phrase. For functions that return a value on some paths and fall off the end on others, write `@return <type> <description of the failure value>; void on the success path`. For activity methods whose only `return` is a `redirectWithMessage(...)` call, the established Yioop convention is to omit `@return` entirely.

- The Public, Help, Search built-in groups + per-user `Personal$<id>` groups underpin chat, clipboard, and context-sensitive help. Touching any chat / wiki-resources / secret-ballot items must preserve this model.

- All end-user-facing strings need to go through `tl()` / `LocaleModel.php`. Code comments and log messages stay English.

- Per `Coding.pdf` general rules: minimise external deps; UTF-8 no BOM; 4-space indent; ≤80 col lines; phpDoc on public surfaces; no `@` error suppression.

- Apache vs WebSite-mode hazards. Yioop runs under both Apache (mod_php / php-fpm — output buffered, `header()` and `exit()` work normally) and atto's WebSite (long-running server, response stream starts early). Handler code that emits non-HTML output must use `$parent->web_site->header()` (delegates to plain `header()` under Apache) and `\seekquarry\atto\webExit()` (falls back to real `exit()` under Apache) rather than the bare PHP calls. Pre-MailSite testing was under Apache; the same code paths under WebSite warn-and-misbehave. Sweep done across `SocialComponent` JSON handlers; other controllers may still have stale patterns.