# Email List Subscribe / Unsubscribe

Arc Post-Mortem Summary. Built the email-list subscribe and unsubscribe flow. It added a per-group mail subscription with its checkbox (Phase A), List-Unsubscribe headers on group and bulk mail plus a web unsubscribe path (Phase B), registration and whole-domain unsubscribe (Phase C), and the bot mailbox with MailSite configured (Phase D). Existing behavior was verified first and the design decisions were resolved with Chris.

Arc started 2026-06-17. This is the List-Unsubscribe piece of v10 ToDo item 12 (Message-ID / MIME-Version / Content-Type / Content-Transfer-Encoding and DKIM signing already landed; the List-Unsubscribe headers plus the web + mailto unsubscribe machinery are what remain). Goal: let a user turn off bulk mail from a group, keep unsubscribed users out of the recipient lists, and attach RFC 8058 List-Unsubscribe headers (a one-click URL and a mailto address) to bulk and registration mail, backed by a web endpoint and — when MailSite is configured — a `bot@<domain>` mailbox that processes mailto unsubscribes.

## What exists today (verified)

1. Membership lives in `USER_GROUP (USER_ID, GROUP_ID, STATUS, JOIN_DATE)`, primary key `(GROUP_ID, USER_ID)` (`ProfileModel` fresh schema). No mail-subscription column yet.
2. `GroupModel::getThreadFollowers($thread_id, $owner_id, $exclude_id)` returns USER_ID / USER_NAME / EMAIL for everyone who posted in a thread plus the owner; it does not currently know the thread's group. `GroupModel::getGroupUsers($group_id, $filter, $sorts, ...)` returns every member.
3. Bulk mail is queued as serialized four-element tuples `[subject, body, to, from]` in per-batch `.txt` files; `BulkEmailJob` reads them and calls `$mail_server->sendImmediate(...)`. The enqueue points with group context are in `SocialComponent` near line 1516 (thread followers) and line 1911 (group users).
4. `MimeMessage::build($from, $to, $subject, $body, $attachments, $extra_headers)` already accepts an `$extra_headers` array — the natural seam for the List-Unsubscribe headers.
5. ✓ DKIM signing is already wired: `SmtpClient::sendImmediate` wraps the built message in `self::dkimSign(...)` (which calls `DkimKey::sign` when a selector is configured, pass-through otherwise), so the List-Unsubscribe headers added via `$extra_headers` end up inside the signed message.

## Phase A — per-group mail subscription + the checkbox

1. ✓ Added a `RECEIVE_MAIL` column to `USER_GROUP` (default 1 = subscribed) in the fresh schema, plus the portable v108 `ADD ... DEFAULT 1` migration that switches it on for every existing membership.
2. ✓ `GroupModel`: `getGroupUsers` gained a `$subscribed_only` flag (filters `RECEIVE_MAIL <> 0`); `getThreadFollowers` gained a `$mail_only` flag that, via a correlated `NOT EXISTS` on thread→group→`USER_GROUP`, drops members with `RECEIVE_MAIL = 0`. The two mail callers in `SocialComponent` (post→thread-followers, owner-post→group-users) now pass the flag; the contact-requests caller of `getThreadFollowers` stays unfiltered.
3. ✓ A "Receive Group Mail" checkbox on the group settings page, shown in both the editable (`editgroup`) and the read-only (`infogroup`) views as a row in the settings table (tied by its HTML `form` attribute to a separate hidden form, so ticking it saves only this choice and the main Save leaves it alone; a new `changemailpref` case checks the viewer is a member, via `checkUserGroup`, then writes their `RECEIVE_MAIL` and returns to the page it came from). Driven by new `GroupModel` `getMailSubscription` / `setMailSubscription`. So a read-only viewer can still unsubscribe themselves.
4. ✓ Thread level: a `THREAD_UNSUBSCRIBE` table (v109) recording per-user opt-outs; `GroupModel` `isThreadUnsubscribed` / `setThreadUnsubscribe`; the second arm of the `getThreadFollowers` exclusion (a `NOT EXISTS` on that table); and a Follow / Unfollow button on the thread view, wired to a `togglethreadmail` feed action that sets the opt-out (idempotent, so a refresh does not re-toggle) and returns to the thread.
5. ✓ Regression fix (DB-error sweep across recent arcs): positional `INSERT`s broke when a column was added to a table they target. (1) `GroupModel::addUserGroup`'s `INSERT INTO USER_GROUP VALUES (?,?,?,?)` now names its columns so `RECEIVE_MAIL` defaults — this is what made joining / creating a group fail. (2) Createdb.php fresh-install seeds: the eight USER_GROUP seeds now supply `RECEIVE_MAIL` (= 1) and the four SOCIAL_GROUPS seeds now supply `RENDER_ENGINE` (the OPTIONS arc added it to the schema at v106 but never to Createdb.php). (3) Swept every table our v100+ migrations added a column to: MAIL_ALIAS (v102, DOMAIN), SOCIAL_GROUPS (v106), USER_GROUP (v108) — MAIL_ALIAS has no positional inserts, so the other two were the only casualties. Verified the fixed seeds insert cleanly at the current schema and the old value-counts still fail. Going-forward rule: when a table is altered, scan every insert into it.
6. ✓ Reworked thread level from opt-out to opt-in (Chris's call): by default you do not follow a thread; posting to or starting a thread follows it, and the state icon toggles it. The `THREAD_UNSUBSCRIBE` table (which recorded "stop mailing me") is replaced by `THREAD_FOLLOW` (which records "mail me about this thread"), via a v110 migration that drops the old table and creates the new one. `GroupModel` gains `isThreadSubscribed` / `setThreadSubscribe` and a `getThreadSubscribers` that lists a thread's followers with their addresses; the post-notify path mails that list, and `getThreadFollowers` reverts to its plain "thread participants" form for the contact-requests caller. The toggle action and the (temporary) Follow / Unfollow button now use a `follow` flag. Open choices: (a) icon will be a bell (following) / bell-with-slash (not following) unless Chris prefers another; (b) thread-follow currently mails replies independent of the group's RECEIVE_MAIL switch — following a thread mails you even with group mail off (alternative would be to treat RECEIVE_MAIL as a master mute).
7. ✓ UI: replaced the big button with a small toggle icon — a small bell (following) or bell-with-slash (not following) that shows the current state and toggles on click. The two bell icons are added to the standard `iconlink` helper. The breadcrumb bell uses `renderButton` (a boxed icon link); the feed-row bell uses `renderFormButton` with an onclick, exactly like the AI-summary button beside it, so they match in size and styling. Also tidied that helper method: fixed the `additional_attibutes` parameter typo (now `additional_attributes`) and switched the feed's AI-summary `echo`s to `e()`. The bell sits in the breadcrumb menu bar right after the thread name on the thread view, and on each thread row in the feed just after the AI-summary control. The feed needs to know the follow state of every thread it lists, so the controller looks them up in one query (`getFollowedThreads`) and the toggle action now takes a `follow_thread` id so a feed row can follow the thread it names and return to the feed.

## Phase B — List-Unsubscribe on group / bulk mail + the web endpoint

Foundation (done): the make / parse primitive that items 3 and 4 both share — `UnsubscribeToken::make($user_id, $group_id)` / `parse($token)` in `src/library/mail/UnsubscribeToken.php`: the two ids plus an `AUTH_KEY`-keyed HMAC, joined by dots, URL safe. Item 3 builds the link with `make`; item 4 validates it with `parse`. Covered by `UnsubscribeTokenTest` (round-trip, tampered ids, forged signature, malformed input). This is preparation, not a numbered deliverable.

1. ✓ Decided the unsubscribe mailto address: `MailSiteFactory::unsubscribeMailtoAddress($root_email)` returns `bot@<first MAIL_DOMAIN>` when this Yioop runs its own mail, otherwise the root account address passed in. Kept it pure (reads config, takes the root e-mail as a parameter, touches no model) so as not to widen the factory's existing model-use exception. (Also fixed two pre-existing >80 lines in GroupfeedElement noticed while here.)
2. ✓ Carried the unsubscribe header through to the send path. Decided to precompute the List-Unsubscribe header at enqueue and carry it (rather than stuff raw group + recipient into the mail tuple): added an `$extra_headers` argument to `SmtpClient::send()` and `sendQueue()` (forwarded to `sendImmediate`, which already supported it), queued records now serialize a five-element tuple, and `BulkEmailJob`'s two readers accept four- or five-element tuples so queued files written before the upgrade still send (with no extra headers). Also added the missing `is_array` guard in `doTasks` so a corrupt queued record no longer trips `count(false)`.
3. ✓ Attach, via `MimeMessage::build`'s `$extra_headers`: `List-Unsubscribe: <mailto:...>, <https://...>` and `List-Unsubscribe-Post: List-Unsubscribe=One-Click` (RFC 8058). Built by a pure `UnsubscribeToken::listUnsubscribeHeaders($user_id, $group_id, $base_url, $mailto_address)` — the web link is the item 4 endpoint with a `make` token, the mailto is item 1's `unsubscribeMailtoAddress` carrying the same token in its subject, plus the one-click marker. Wired into the group-users bulk path in `SocialComponent` (the new-thread-by-owner notification): each recipient's message now carries headers that turn off that user + group's mail on the site it went out from. Covered by a new `listUnsubscribeHeadersTestCase`; full suite green. Open (Chris to decide): the other group mail path, thread-follow notifications, is deliberately left untouched — a group unsubscribe there would be wrong (it would turn group mail off while the follow mail, which is independent of `RECEIVE_MAIL`, kept arriving). That path wants its own thread-level unsubscribe (drop `THREAD_FOLLOW` for that thread), which is a separate token + endpoint action. The owner join-request notice (a single-recipient operational mail) is left as plain mail.
4. ✓ A web unsubscribe endpoint (a route, like the robots / mta-sts routes): the signed token identifies user + group; a one-click POST performs the unsubscribe (a plain GET shows a small confirm page so a link preview fetch does not silently unsubscribe). ✓ Decided (Chris): the endpoint lives in `ApiController`, whose role is being widened from the LLM activities (summarize / transcribe / translate) to general endpoints. As prep, `ApiController` was brought to style conformance this patch (wrapped four long lines, the one `//` comment turned into `/* */`, `$result_obj` renamed off the banned `_obj` suffix to `$decoded`, generic `$model` renamed to `$llm_model`). ✓ Done (this session): added the `unsubscribe` activity to `ApiController`. It cleans the `token`, runs it through `UnsubscribeToken::parse`, and on a bad token shows a short "invalid or expired" page. For a good token it looks up the group name; a GET shows a confirmation page whose button POSTs the token back (so link scanners following the GET cannot unsubscribe anyone), and a POST calls `GroupModel::setMailSubscription(..., 0)` to turn that group's mail off for that user. The page is a small self-contained localized HTML document written straight to the response and ended with `webExit()`, the same emit-and-stop idiom the API JSON path uses, so it renders identically on the atto server and Apache. Five `api_unsubscribe_*` locale strings added. Covered by a new `ApiUnsubscribeTest` (the repo's first controller test): it builds a throwaway membership row in an isolated SQLite database, injects it into the controller, and checks that an invalid link shows no form and changes nothing, a GET shows the confirm form carrying the token without unsubscribing, and a POST actually flips `RECEIVE_MAIL` to off. Full unit suite 40907/40907 green. ✓ Refactored (Chris's review): the activity no longer prints HTML or branch on `$_SERVER['REQUEST_METHOD']`. It now returns view data and lets a new `UnsubscribeView` (web layout) draw the page directly — no separate element, since nothing here is swapped out — and it decides confirm-vs-unsubscribe from a request parameter (`List-Unsubscribe=One-Click`, the RFC 8058 marker the mail client or the confirm button sends) read through `$_REQUEST`, keeping the endpoint agnostic to GET vs POST. Test reworked to assert the returned state and the flipped subscription; full suite 40913/40913 green. ✓ Extended (this patch) for whole-site unsubscribe: the same endpoint now also accepts the whole-site (email-address) token. After the member-and-group parse fails it tries `UnsubscribeToken::parseEmail`; a valid one-click then calls `MailSuppressionModel::suppress` to turn off all of this site's mail to that address, and `UnsubscribeView` gained an all-mail confirm / done message (two new `api_unsubscribe_all_*` locale strings). Routing is by token shape, so the group path is untouched. `ApiUnsubscribeTest` gains whole-site confirm and one-click-suppress cases.

## Phase C — registration / whole-domain unsubscribe

1. ✓ Done (this patch): a suppressed email address is burned — it can no longer be used to register. At registration `dataIntegrityCheck` consults the suppression list and, when the address is on it, refuses the account and shows the user a message that the address can no longer be used; no account is created and no activation mail is sent. The check is gated to registration only (password recovery stays transactional, so an existing user can still recover). Covered by `RegisterSuppressionTest`.
2. ✓ Done: a suppression store keyed by email address, and the send path skips any address on it. Store — a `MAIL_SUPPRESSION` table (one row per address, unique index on EMAIL) added to the fresh schema and via `upgradeDatabaseVersion111` (DATABASE_VERSION 110 → 111), plus a `MailSuppressionModel` with `isSuppressed` / `suppress` / `unsuppress` (case-insensitive, mirrors `MailSenderAllowModel`), covered by `MailSuppressionModelTest`. Skip — the check lives in `BulkEmailJob` (the queue consumer). Before sending each queued message it holds the recipient back only when the message is list mail — it carries a `List-Unsubscribe` header — and the address is on the suppression list. Transactional mail such as a password reset carries no `List-Unsubscribe` and is always sent, so a suppressed address can still recover their account. Covered by `BulkEmailJobTest`.
3. ✓ Done (this patch): the activation email offers the whole-site unsubscribe. `RegisterController::sendActivationMail` now adds the whole-site List-Unsubscribe headers (one-click via the bot mailbox and web links, from `listUnsubscribeHeadersForEmail`) and a plain visible "stop all mail from this site" line at the foot of the message pointing at the web endpoint (`UnsubscribeToken::unsubscribeUrl`, shared with the header builder so the two links never drift). Only the activation mail carries it — password recovery stays plain. Safe because the registration gate already refuses a suppressed address, so this mail never goes to a suppressed recipient. The group and whole-site header builders now share one private helper; covered by `UnsubscribeTokenTest`.

## Phase D — the `bot` mailbox (MailSite configured)

1. ✓ Done (this patch): the username `bot` is reserved — it is added to the registration forbidden-name list, so no account may be created as `bot` and the `bot@<domain>` mailbox identity stays owned by the system. Covered by `RegisterSuppressionTest`.
2. ✓ Done: `BulkEmailJob` reads incoming `bot@<domain>` mail and applies the same unsubscribe a click would. The forgery guard (`UnsubscribeToken::makeEmail` / `parseEmail`, earlier patch) signs the email address itself with the site secret under a distinct all-mail prefix; the bot trusts only this signature, never the spoofable `From`, so an altered signature, a signature reused under a different address, or a plain unsigned address are all rejected (covered by `UnsubscribeTokenTest`). The reader (`processBotMailbox`, this patch) runs at the top of the name-server mail tick: when a local store has a `bot` INBOX it walks each message, unfolds the Subject (the mailto links produce "unsubscribe <token>"), and on a verified token suppresses the address (whole-site token) or turns off the group (member-and-group token) through `applyUnsubscribeToken`, then flags and expunges every examined message so stray mail to the bot address is cleared too. The store is reached through a new shared `MailSiteFactory::storage()` and the mailbox name lives in one `UNSUBSCRIBE_MAILBOX` constant. Covered by `BulkEmailJobTest` (group token, whole-site token, forged token ignored, mailbox process-and-clear, stray-mail clear).

## Other work this phase

1. Found-bug fix folded in this patch: `MailRecordCache` did `new LRUCache(...)` without importing it, so it resolved to a non-existent `...\library\mail\LRUCache` and threw "class not found" the moment `codetool unit` reached `DmarcCheckTest` (which warms the cache). Added the missing `use`; the full unit suite now runs to completion. (The two Index test failures that then surfaced are sandbox-only — PHP 8.3 here vs 8.5 on the dev box — and pass on Chris's box, so left alone.)
2. ✓ Found-bug fix #2 (real portability bug, not sandbox): when the PHP `memory_limit` is unlimited (`-1`, the common CLI default), `metricToInt("-1")` is `-1`, so every "`usage > limit * fill_factor`" memory check goes permanently true (and the matching "<" loop guard permanently false). That made the indexer seal a new partition almost every document — so the on-disk archive's layout depended on `memory_limit`, which it must not (archives are portable). It also one-shotted FetchUrl's curl pump loop and made the fetcher think memory was always low. Fixed with one helper, `memoryCeiling($fill_factor)` in Utility.php (returns `PHP_INT_MAX` when the limit is unlimited), applied at every decision site: PartitionDocumentBundle, FetchUrl (x3), Fetcher, and the legacy IndexArchiveBundle. The log-only site in IndexDocumentBundle keeps printing the raw limit. With the fix the two Index tests pass here under `-1` too; full suite 40893/40894 (the one miss, VersionManager restoreVersion, fails on plain HEAD too and touches none of this code — a separate sandbox-only flake).
3. ✓ Found-bug fix #3 (real defect, surfaced by IconProcessorTest leaving a stray "`..txt`" file in the repo root when OCR is enabled): `ImageProcessor::saveTempFile` built its temp name with `... . " . $file_extension"` — the intended "`.`" separator was swallowed into a string literal, so names came out as "`<hash> . ico`" with embedded spaces (VideoProcessor's twin was already correct). That spaced path was then handed unquoted to `tesseract` in `ComputerVision::recognizeText`, so the shell split it and tesseract saw output base "`.`", writing "`.`"+"`.txt`" = "`..txt`" into the working folder. Fixed the separator (now "`.`" . `$file_extension`) and hardened the exec with `escapeshellarg` on all three path/lang arguments so no future path can spray files into cwd. Added `tempFileNameTestCase` (runs without tesseract) asserting the temp name has no spaces and ends with the given extension; it fails on the old code, passes on the new. Only reproduces where TESSERACT is defined, which is why the PHP 8.3 sandbox never saw it.
4. ✓ Found-bug fix #4 (real flaky defect in VersionManager, ~50% fail rate here, passes more often on the dev box purely by timing luck): `restoreVersionTestCase` failed about half the time because `getActiveVersion` could skip the exact version asked for. A version's folder is named by its timestamp turned into a string at PHP's default float precision, which rounds the last digit; the timestamp searched on is kept at full precision from the serialized record. So restoring a version to its own exact time could compare as just past that version and fall back to an older one (here the 1970 "initial" version, timestamp 1), leaving the expected file missing. Fixed by rounding the incoming search value through the same string form the folder names use, in both `getActiveVersion` and (same latent boundary bug) `getVersionsInRange`. Proven: a 60x repeat loop went from ~half failing to 60/60, the test is now a steady [5/5], and the full unit suite is 40897/40897 all green.
5. Code audit (this patch, no behaviour change): inlined two single-use private helpers in `UnsubscribeToken` (`encode` into `makeEmail`, `decode` into `parseEmail`); merged the two near-identical signing methods (`sign` and `signDomain`, same body differing only by prefix) into one `signPayload($prefix, $payload)` with four callers; and removed the duplicated "one-click → act → done, else confirm" block in `ApiController::unsubscribe` by setting the state once after a single scope branch. Suite unchanged at 40946/40946.

## Out of this arc

1. DKIM signing (RFC 6376) — verified already done (see above), so the only item-12 mail piece left after this arc is documenting SPF / DMARC (DNS-side) in the install guide.

## Design decisions (resolved with Chris)

1. Thread + group level both honored. Group level is the `RECEIVE_MAIL` column. Thread level is supported at least at the header level, with an "unfollow" control on the thread page (Phase A item 4).
2. Whole-domain suppression is a standalone table keyed by email address.
3. Unsubscribe tokens are signed with `crawlAuthHash` so the URL and the mailto suffix cannot be forged.
4. `bot@<domain>` uses the domain the mail was going out on, resolved at enqueue time.
5. ✓ Decided in Phase B: the header is precomputed at enqueue and carried as `$extra_headers` through the send / queue / sweep path (see Phase B item 2).