- Yioop Repo - Seekquarry

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Phase 9 &mdash; Crawl and Index Health, and
Enhanced Testing</title>
<style>
body { font-family: system-ui, sans-serif; max-width: 760px;
    margin: 1.5em auto; padding: 0 1em; line-height: 1.5; color: #1a1a1a; }
h1 { font-size: 1.4em; }
h2 { font-size: 1.1em; margin-top: 1.4em; }
ol { margin: 0.3em 0; } li { margin: 0.25em 0; }
ol.outline { list-style-type: upper-roman; }
ol.outline ol { list-style-type: upper-alpha; }
ol.outline ol ol { list-style-type: decimal; }
ol.outline ol ol ol { list-style-type: lower-alpha; }
.check { color: #1a7f37; font-weight: bold; }
.next { color: #b35900; font-weight: bold; }
.review { color: #0969da; font-weight: bold; }
.working { color: #b35900; font-weight: bold; }
.nope { color: #82071e; font-weight: bold; }
.q { color: #8250df; font-weight: bold; }
code { background: #f2f2f2; padding: 0 0.25em; border-radius: 3px; }
em { color: #555; }
</style>
</head>
<body>
<h1>Phase 9 &mdash; Crawl and Index Health, and Enhanced Testing</h1>
<p><em>Markers: <span class="next">&#9654;</span> being worked on,
<span class="review">&#9873;</span> implemented and waiting to be
reviewed,
<span class="check">&#10003;</span> done,
<span class="check">&#10003;</span><span class="nope">&#10007;</span> closed
by deciding not to do it, <span class="q">?</span> open decision,
unmarked = planned.</em></p>
<ol class="outline">
<li><strong>Crawling using synthetic data.</strong> A crawl of any size
can be run on one machine without fetching anything from the web. The
pages are made up as the crawl asks for them, so a run repeats exactly
and costs nobody bandwidth.
<ol>
    <li><span class="check">&#10003;</span> <strong>A synthetic crawl tab is
added to the crawl options.</strong> Manage Crawl, then Edit Crawl
Options, gains a tab beside the web crawl one. It keeps the settings that
still mean something for a made-up web and replaces the ones that do not.
    <ol>
        <li><span class="check">&#10003;</span>
        <strong>The settings about reaching the live web are
        dropped.</strong>
Repeat type, crawl sleep, robots.txt, restrict sites by url, disallowed
sites and seed sites all describe fetching from real hosts. A made-up web
has no hosts to be polite to and no addresses to restrict.</li>
        <li><span class="check">&#10003;</span>
        <strong>The shape and size of the made-up web are chosen.</strong> A
graph type says how the pages link to one another. A graph size says how
many pages the web holds. A third setting says which pages the crawl
starts from, taking the place of seed sites.</li>
        <li><span class="check">&#10003;</span>
        <strong>What a page is and what it says are chosen by
chance.</strong> One model gives the chance a page is html, pdf or an
image. Another gives the words that appear on it. The same seed gives the
same web every run, so a result can be compared with the one before
it.</li>
        <li><span class="check">&#10003;</span>
        <strong>Each control on the synthetic crawl tab has a help link and
a help page.</strong> The web crawl and archive tabs have a help button
beside each control that opens a wiki page for it; the synthetic tab had
none. Each of its seven controls now has one. The seven pages are Graph
Type, Power Law Exponent, Links Per Page, Domain vs Path Links, Document
Length, Term Frequency Exponent and Image Probability. They are written
as wiki source for Chris to paste through the wiki, since the help pages
file is generated and never patched.</li>
    </ol></li>
    <li><span class="check">&#10003;</span>
    <strong>A synthetic crawl makes its own pages rather than fetching
them.</strong> Where that tab is the one chosen, the fetcher takes its
pages from a function that generates them in place of
<code>FetchUrl::getPages</code>. Everything after the fetch runs
unchanged, so what is measured is the real crawl. A page is a tuple of
numbers: a grid page steps one along in each component, a tree page
appends each child, a power law page draws its targets by hash. By
domain the tuple is one hyphened label, <code>http://i-j.web/</code>,
so each page is its own company level domain; by path it sits under
<code>http://synth.web/i/j/</code>. The queue server and fetcher see a
plain web crawl; only the fetch call differs. Two rules of Yioop's had
to give: a host with no dot has no company level domain and is dropped,
and five consonants in a row read as spam, so <code>y</code> is now a
vowel there. Run as daemons with errors shown: a grid by path reached 66
pages on one robots.txt, by domain 25 pages on one robots.txt per page,
and a search of the finished index found a made-up word.</li>
</ol></li>
<li><strong>Each crawl type is run and what it turns up is
fixed.</strong> A grid, a tree and a power law web are each crawled under
each crawl order, host budgeting and breadth first, with the fetcher and
queue server logs read as the crawl runs. A crawl that slows, stalls or
stops, or an index that comes back wrong for a search, names the stage at
fault, and that is what gets fixed.
<ol>
    <li><span class="check">&#10003;</span>
    <strong>The indexer stores a file of pages in the same time however
full its partition is.</strong> A power law crawl with eight fetchers
ran no faster than with four, at 253 thousand pages an hour. The logs
showed the indexer taking longer per file as each partition filled. It
went from a few seconds to over thirty and fell back at each new
partition. It
was searching the whole packed index once per page, only to count new
keys. It now keeps the partition's keys in a set built once. On one
workload the per-file time had climbed to 1.6 seconds; it now holds at
0.02. The crawl then ran at 830 thousand pages an hour.</li>
    <li><span class="check">&#10003;</span>
    <strong>The scheduler hands out work as fast as the fetchers can
take it.</strong> With the indexer no longer the limit, a crawl of eight
fetchers settled at 611 thousand pages an hour with every process idle.
The scheduler made one fetch
batch a pass and waited five seconds a pass. The rate was the batch size
divided by five.
    <ol>
        <li><span class="check">&#10003;</span>
        <strong>A scheduler pass waits two seconds, an indexer pass still
waits five.</strong> Both jobs shared one minimum pass time. The
scheduler hands out the work and gains from a shorter pass. The indexer
reads a directory of index data and gains nothing. A queue server run
from a terminal does both jobs in one process and keeps five.</li>
        <li><span class="check">&#10003;</span>
        <strong>A batch short of its size is filled from another
tier.</strong> A batch whose tier ran dry went out with room left in it,
and half the batches were short. The rest is now filled from a tier
holding urls. Which tier a batch is made from is still chosen once per
batch. That choice is what sends the most interesting outstanding sites
out first.</li>
        <li><span class="check">&#10003;</span>
        <strong>A dictionary lookup whose file a merge moved finds the
key anyway.</strong> A merge writes the rows of a tier into the tier
below and then takes the tier away. A reader that read the file list a
moment earlier asked for a file that had gone. It reported the key
absent, so a search lost results. The tiers are now walked again, with
nothing locked either way. The lists of files a lookup reads are cached
in the process. A list found stale is dropped before the walk is made
again, since otherwise the second walk failed as the first did.</li>
        <li><span class="check">&#10003;</span>
        <strong>Marking the crawl not stalled no longer warns on every
upload.</strong> Setting a file's time to an explicit value is allowed
only to the file's owner, whatever the permissions say. So the web app
warned about a file the queue server owns. Six calls that mean now drop
the time argument, which needs only write permission.</li>
    </ol></li>
    <li><span class="check">&#10003;</span>
    <strong>A thumbnail is never larger than its picture.</strong> With
the indexer and the scheduler no longer the limit, the fetchers were.
Their logs alternated runs of passes at 1.4 seconds with runs at 5.5, and
the slow runs were batches of images. Each image cost 12 milliseconds,
all of it in the thumbnail. A 240 by 60 picture was scaled up to 414 by
256 and then encoded. A thumbnail now fits inside its picture. A
hundred images went from 1.24 seconds to 0.13 here. The picture also
keeps its proportions inside the thumb, centered on its own average
color. Before, it was stretched to the thumb's shape. And a fetch
batch now spreads its images evenly among its pages. A fetcher's passes
then sit near their two second floor. Before, short passes slept and
long ones did not.</li>
    <li><span class="check">&#10003;</span>
    <strong>A finished synthetic index is checked through the search
API.</strong> A synthetic page is rebuilt from its number and the crawl's
seed. So what the index says of a page can be checked against the page
itself. tests/SyntheticIndexExperiment.php samples pages. It checks that each
carries its own title, is found by its rarest term, and is stored once.
It checks each is listed as a linker of its targets. It reads document
counts by term rank and in-link counts. It reports records stored against pages
visited. Run against an eleven thousand page power law crawl made here,
every check passed.</li>
    <li><span class="check">&#10003;</span>
    <strong>Index and storage faults found by synthetic crawls are
    fixed.</strong> Each came out of running the search experiment over a
    crawl and reading what the index answered. The faults are of one
    shape: a cache or a list keyed by a name that gets reused.
    <ol>
        <li><span class="check">&#10003;</span>
        <strong>The LSM tree reads a tier file again once it has been
        rewritten.</strong> Its cache was keyed by file name alone. A merge
        writes a tier of the same name, so a term's rows came back as its
        first partitions repeated. The cache keeps each file's size and
        time. It checks them at most once a second, since a query makes
        thousands of reads. A test drives the tree as the index does.</li>
        <li><span class="check">&#10003;</span>
        <strong>A dictionary row is no longer lost when a flush meets a file
        or a buffer that already holds its key.</strong> A flush wrote over
        the rows of a file named for the same first key. Two rows for one
        key in one buffer were read as one. Both are merged now, and two
        cases fail on the old code.</li>
        <li><span class="check">&#10003;</span>
        <strong>A dictionary record is read to its own end, and a corrupt
        posting length names its term.</strong> The record read took the end
        position as a length, so every lookup decoded the records after it
        too.</li>
        <li><span class="check">&#10003;</span>
        <strong>Partition postings stay under their own numbers and in
        memory.</strong> The open postings files were renumbered when the
        list was trimmed, so a read took another partition's postings. A run
        with no memory limit held no partition indexes at all. The count
        came from dividing a limit of -1.</li>
    </ol></li>
    <li><span class="check">&#10003;</span>
    <strong>The synthetic index experiment says what it asks and how long
    each answer took.</strong> It is the tool that found the faults above,
    so its report was made to show where an answer went missing.
    <ol>
        <li><span class="check">&#10003;</span>
        <strong>Each question is one query for one result, timed.</strong>
        Two checks asked for two hundred results and looked through them.
        They now ask for the term and the url together, and the report gives
        the seconds each kind of question cost.</li>
        <li><span class="check">&#10003;</span>
        <strong>A link the index does not answer for is asked about four
        more ways.</strong> The row says whether the link record is missing
        or only its url form is.</li>
        <li><span class="check">&#10003;</span>
        <strong>Its usage and its checks say what they ask.</strong> It
        names both arguments and says how far it has gone every ten pages.
        The one-record check counts none and several apart. Term ranks past
        the crawl's vocabulary are left out.</li>
    </ol></li>
    <li><span class="check">&#10003;</span>
    <strong>A day-long synthetic crawl runs without noise or idle
    time.</strong>
    <ol>
        <li><span class="check">&#10003;</span>
        <strong>Four warnings from the day-long crawl log are
        silenced.</strong> They came from a page alias with no group and a
        tier merged away before its read. The others were a missing text
        resource and the SSL: Success notice.</li>
        <li><span class="check">&#10003;</span>
        <strong>A fetcher pass takes at least one second, not two.</strong>
        On the synthetic web the second second was idle time. The three
        curl_multi trace lines in FetchUrl go out only with timing on.</li>
    </ol></li>
    <li><span class="check">&#10003;</span>
    <strong>An index is rebuilt in parallel, and a query reaches all of
    it.</strong> A rebuild of a 336 partition crawl left queries answering
    nothing. The work below found why and changed how indexes are built
    and read.
    <ol>
        <li><span class="check">&#10003;</span>
        <strong>A rebuild runs a pool of builders and one merger that adds
        partitions in order.</strong> Leftover done files are cleared first,
        and a partition left without its four files is built again.</li>
        <li><span class="check">&#10003;</span>
        <strong>A running server sees a dictionary another process
        rebuilt.</strong> It kept its copy of each tier file and its highest
        tier number, so it answered from the dictionary as it was before.</li>
        <li><span class="check">&#10003;</span>
        <strong>A word is read in batches, so a query reaches every
        partition.</strong> A query read only a word's first twenty
        partitions; it now reads the next twenty when it runs out.</li>
        <li><span class="check">&#10003;</span>
        <strong>A guessed query reads one combined key per word.</strong> An
        index built now files each word also under its language and safe
        search values. A guessed query then needs no intersection with the
        lang: and safe: words. An older index gets a one-second bound on its
        batch reads.</li>
        <li><span class="check">&#10003;</span>
        <strong>Fetchers build closed partitions and the Indexer merges them
        in order.</strong> Fetchers on the queue server's machine build
        between batches. The Scheduler holds back batches while more than
        four wait. A stopped crawl finishes its closed partitions.</li>
        <li><span class="check">&#10003;</span>
        <strong>A fetcher builds a partition only when it holds no
        schedule.</strong> A synthetic crawl fell from about a million pages
        an hour to about 300,000. A fetcher had built a partition for three
        minutes while holding schedule 529. The queue server drains results
        in schedule order, so no batch was made from 18:07 to 18:09.</li>
        <li><span class="check">&#10003;</span>
        <strong>A crawl shows it is stopping as soon as stop is
        clicked.</strong> Until a queue server reads the stop, the crawl
        status shows Shutting Down Queue, the first stopping step. The queue
        server's own steps replace it as they are written.</li>
        <li><span class="check">&#10003;</span>
        <strong>Fetchers merge partitions into the dictionary.</strong>
        Between batches a fetcher builds one closed partition and then merges
        the next built one, under a single merge claim. The Indexer merges
        only when stopping or when fetchers fall behind, and no
        DictionaryUpdater runs.
        <ol>
            <li><span class="review">&#9873;</span> Three sandbox crawls ran side by side for 300 seconds each,
            with 400 documents per partition and two fetchers.</li>
            <li><span class="review">&#9873;</span> Indexer merges: 3,262 pages, 61 of 61 partitions in the
            dictionary, stop in 20 seconds.</li>
            <li><span class="review">&#9873;</span> Revert to DictionaryUpdater: 10,529 pages, but only 34 of 230
            partitions in the dictionary, stop in 5 seconds.</li>
            <li><span class="review">&#9873;</span> Fetcher merges: 4,721 pages, 109 of 109 partitions in the
            dictionary, stop in 26 seconds.</li>
        </ol></li>
        <li><span class="check">&#10003;</span>
        <strong>A stopping crawl shows how many partitions are left.</strong>
        Crawl Status reads Closing Crawl Dictionary with a line counting the
        closed partitions still to add. The counts and peak memory the crawl
        reported stay shown while it stops.
        <ol>
            <li><span class="review">&#9873;</span> In a sandbox crawl stopped after 150 seconds, the count fell
            from 25 to 2 over 20 seconds.</li>
            <li><span class="review">&#9873;</span> Peak memory stayed at 37,589,888 and visited URLs at 2,711
            throughout the stop.</li>
            <li><span class="review">&#9873;</span> The new line is translated into all 24 other locales.</li>
        </ol></li>
        <li><span class="check">&#10003;</span>
        <strong>Pictures in news results keep their shape.</strong>
        A news thumbnail fills its wide box without being stretched. The
        box shows the middle of the picture instead.
        <ol>
            <li><span class="review">&#9873;</span> The news result image rule gains object-fit: cover. The
            square thumbnail is cropped to the box, which drops its gray
            bands.</li>
            <li><span class="review">&#9873;</span> A web result with a picture wrote its alternate text from an
            undefined constant. It now uses the result title.</li>
        </ol></li>
        <li><span class="check">&#10003;</span>
        <strong>A stop with a backlog can close with partitions
        unmerged.</strong> The shutdown loop stops waiting after
        BUILD_CLAIM_TIMEOUT, 600 seconds. Builds take about five minutes,
        so a backlog of several can be cut off.
        <ol>
            <li><span class="review">&#9873;</span> The wait now counts from the last partition finished. It
            gives up only after twice BUILD_CLAIM_TIMEOUT with none
            finished.</li>
            <li><span class="review">&#9873;</span> A scripted backlog finishing one partition every two
            seconds ran 12 seconds past a 3-second limit. A stuck one
            stopped at 3 seconds.</li>
        </ol></li>
        <li><span class="check">&#10003;</span>
        <strong>A synthetic tree crawl can seed an empty host.</strong>
        SyntheticWeb::seedUrl gives http://.web/index.html for a tree graph
        when the domain link probability is 0.5 or more.
        <ol>
            <li><span class="review">&#9873;</span> The empty tuple is now the path host's root page. A new test
            case fails on the old code and passes now.</li>
        </ol></li>
    </ol></li>
    <li><span class="check">&#10003;</span>
    <strong>A guessed query reads one combined key per word.</strong> Most
    queries are split into one disjunct per word, each intersected with
    the lang: and safe: words guessSemantics adds. Those two words are on
    nearly every page, so the intersection cost the most time.
    <ol>
        <li><span class="check">&#10003;</span>
        <strong>An index built now files each word under combined
        keys too.</strong> A word gets a key for its page's short language
        code with safe:all, and one with safe:true on a safe page. The key
        is the word's first eight key characters, an underscore and a hash.
        The combined postings point at the word's own positions.</li>
        <li><span class="check">&#10003;</span>
        <strong>A query on such an index reads the combined key.</strong>
        A disjunct with one short lang: word and safe:true or safe:all
        looks each word up by its combined key. The two meta words are
        then dropped, so a one-word disjunct needs no intersection.</li>
        <li><span class="check">&#10003;</span>
        <strong>An older index keeps working, with a one-second
        bound.</strong> A COMBINED_TERMS flag in archive_info marks an index
        built this way, and a rebuild from partition 0 sets it. Without the
        flag, a word stops reading new batches after one second.</li>
        <li><span class="check">&#10003;</span>
        <strong>Rare word queries ran about three times as fast.</strong>
        On the 75 partition test crawl they took 0.027 and 0.031 seconds,
        against 0.075 and 0.103 by the old path. The rebuild took 181
        seconds against 172.</li>
        <li><span class="check">&#10003;</span>
        <strong>A one-word query reports the word's whole count.</strong>
        A word's rows are read twenty partitions at a time, and the count
        came from the first batch alone. getWordInfo now adds the rows past
        the batch to the totals when asked for the remaining total. It
        sums at most EXACT_WORD_COUNT_PARTITIONS rows, 200, and past that
        scales by the completed partitions over the partitions read.</li>
        <li><span class="check">&#10003;</span>
        <strong>The experiment reports held pages by walk length.</strong>
        Its floor of one url in the out-degree held assumed a crawl in
        breadth-first order and was not checked. Step 4 now prints how
        many sampled urls are held at each number of links from the seed.</li>
        <li><span class="check">&#10003;</span>
        <strong>The experiment's checks and summary read in plain
        English.</strong> Checks 2 and 4 say what is searched for and what
        should come back. Each summary line names its check by number and
        says what passing means, and a skipped check says why.</li>
        <li><span class="check">&#10003;</span>
        <strong>Checks 2 and 4 ask queries that stay small as a crawl
        grows.</strong> Check 2 searches a page's five least common words
        together; check 4 searches link: a target with info: the page. Each
        prints the query it asked for the first held page.</li>
        <li><span class="check">&#10003;</span>
        <strong>The experiment samples pages with a site:all
        query.</strong> Walking the synthetic graph found only 16 of 200
        pages held, since the crawl follows its own page ordering. The
        sample is now the first pages site:all returns. Every check then
        runs over the pages asked for, or all the index holds if fewer.
        A check 2 failure now names which of the five words the index does
        not hold for that page.</li>
        <li><span class="check">&#10003;</span>
        <strong>A random walk adds ten deeper pages to the
        sample.</strong> site:all gives only the first pages crawled. The
        walk from before now looks up each page it stops on with info:.
        It walks again until ten held pages are found, and every check
        runs over them too.</li>
        <li><span class="check">&#10003;</span>
        <strong>A join of several words finds pages past the first
        partition.</strong> A word's walk kept its last document index on
        moving to a new partition. An intersection then skipped that
        partition's pages below the stale index. advanceGeneration now
        clears it, and a WordIterator case fails without the change.</li>
        <li><span class="check">&#10003;</span>
        <strong>A queue server listed by the name server's address is this
        machine.</strong> isSingleLocalhost took a machine listed by address
        for another computer, so queries went over HTTP to the site itself.
        UrlParser::isSameMachine now compares hosts and resolved addresses.</li>
        <li><span class="check">&#10003;</span>
        <strong>A word's walk reaches partitions past its first
        batch.</strong> getWordInfo gave the active partition's row with
        every batch, so a walk went from partition 19 to the active one
        and skipped the rest. The row now comes only with the last batch.
        Postings file handles are now kept by file, not partition number.</li>
        <li><span class="check">&#10003;</span>
        <strong>Check 2 asks for all five words with no:rewrite.</strong>
        Without it, rewriteAsDisjunctiveQuery turns the five words into a
        search for any of them. Checks 4 and 7 now look for the page among
        the answers, since a join's reported count is never zero.</li>
        <li><span class="check">&#10003;</span>
        <strong>Check 2 leaves out words the parser joins into one
        term.</strong> hyphenateEntities joins two query words when the
        n-gram Bloom filter answers yes, and it does for some number pairs.
        The joined term is on no page. Check 2 now skips such a word.</li>
        <li><span class="check">&#10003;</span>
        <strong>A partition holds at most MAX_ITEMS_PER_FILE
        documents.</strong> PartitionDocumentBundle started a new partition
        only past the limit, so each full partition held one extra. The
        WordIterator clamp then dropped that one's posting. The limit now
        has one name, MAX_ITEMS_PER_FILE, and the iterators fall back to
        NUM_DOCS_PER_PARTITION rather than 16384.</li>
    </ol></li>
    <li><span class="check">&#10003;</span>
    <strong>A Bloom filter lookup hashes only as far as it needs.</strong>
    Most n-word-gram lookups in hyphenateEntities fail on an early bit, so
    contains now makes its md5 digests one at a time.
    <ol>
        <li><span class="review">&#9873;</span> Output is identical on 200 pages in five languages, which took
        0.44 seconds instead of 0.56. A test checks the answers agree.</li>
    </ol></li>
    <li><span class="check">&#10003;</span>
    <strong>Profile phrase extraction before making builds faster.</strong>
    On the 97 English help pages, canonicalizing took 48 percent of the
    time and stemming about 43 percent of the ticks.
    <ol>
        <li><span class="review">&#9873;</span> A per-term stem cache cut the time by 11 percent. It changed
        term positions in one Spanish page, so it is not in.</li>
        <li><span class="review">&#9873;</span> The Spanish stemmer kept the last word's R2 region, and gave no
        stem for qu&eacute; after words such as una. It and the French
        stemmer now reset it.</li>
        <li><span class="review">&#9873;</span> With that fixed, the stem cache gives identical output on 200
        pages in five languages, 19 percent faster.</li>
    </ol></li>
    <li><span class="check">&#10003;</span>
    <strong>Only a share of fetchers build partitions at once.</strong>
    BUILD_FETCHER_FRACTION, 0.25, times the fetchers running on a channel
    gives the build slots. A fetcher takes a slot file before it builds.
    <ol>
        <li><span class="review">&#9873;</span> With both slots held, a fetcher builds nothing. A slot left
        past BUILD_CLAIM_TIMEOUT is taken back and released.</li>
    </ol></li>
    <li><span class="check">&#10003;</span>
    <strong>A pdf thumbnail draws a magazine cover as a viewer
does.</strong> Two covers came out scrambled or blank, and six faults in
the page reader were found against a pdftoppm render of the same page
and fixed. A large four ink jpeg past four million points is read one
shade per block rather than handed to the drawing library, which dropped
its black ink. A picture paired with a soft mask is laid through the
mask rather than opaque. A composite font is read through its descendant
and its letters reached two bytes at a time. Trimming a run keeps the
zero byte that begins a low numbered letter, and a color set inside a
saved state is restored with it. The character spacing operator sets how
far apart letters are laid. A picture named inside a form the page draws
counts as one of the page's pictures.</li>
        <li><span class="check">&#10003;</span>
        <strong>A rebuild prints how it ended.</strong> A rebuild that
        stopped left no word on the terminal. A shutdown line now names a
        fatal error, the pool's reason and the last progress. A closed
        terminal no longer stops it where PHP has pcntl.
        Its first report named a builder silent for 152 seconds on one
        document. The indexer now gives up on a document after
        DOCUMENT_TIME_LIMIT, 10 seconds, and moves on. A document a builder
        stopped on is skipped by the next builder of that partition.</li>
    <li><span class="review">&#9873;</span>
    <strong>Pages keep more of their text and are stored under the right
    language.</strong> What a crawl keeps of a page can be checked one page
    at a time with ArcTool test-page.
    <ol>
        <li><span class="review">&#9873;</span> Page processors keep entities, structured data, PDF streams and
        the text of subset fonts.</li>
        <li><span class="review">&#9873;</span> A page's declared language wins a tie in the language
        guess.</li>
        <li><span class="review">&#9873;</span> The word filter's Factory Settings drop bot-check pages. Six
        more kinds of site have built-in scrapers.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>The fetcher gets what a browser would and sends it
    promptly.</strong> A crawl's results reach the index within a minute of
    its start.
    <ol>
        <li><span class="review">&#9873;</span> Crawls carry cookies, drop tracking parameters and start from
        2026 seed sites. curl's cookie store reads the null device, so its
        verbose output holds no cookie file warning.</li>
        <li><span class="review">&#9873;</span> A site's root keeps the page of a same-host redirect or of a
        working default-locale subsite.</li>
        <li><span class="review">&#9873;</span> A crawl starts only with plugins that exist, their names trimmed.
        A name ending in a newline stopped the worker starting it.</li>
        <li><span class="review">&#9873;</span> An archive crawl sends its first pages at once, then every
        minute. Uploads go in slices.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A crawl spends its time on new pages.</strong> The crawl went
    from about 60,000 to 90,000 pages an hour.
    <ol>
        <li><span class="review">&#9873;</span> A fetch batch goes to one fetcher. A pass counts the hosts it
        must look up, not urls, and frees its handles in the next
        pass.</li>
        <li><span class="review">&#9873;</span> Summaries stop scoring once full, and a page is parsed once for
        scraper signatures. The Chinese taggers read each weight once.</li>
        <li><span class="review">&#9873;</span> A long PDF guesses its language from its first 8,192 bytes. A
        scanned PDF is read by OCR only until its text fills a
        summary.</li>
        <li><span class="review">&#9873;</span> A PDF picture goes to OCR as a gray map or as its own JPEG
        bytes, not drawn point by point. Scans
        packed by fax or JBIG2 coding, or one bit to a point, are opened
        first, so their words are read.</li>
        <li><span class="review">&#9873;</span> A fetch pass logs how many of its transfers finished by each
        second.</li>
        <li><span class="review">&#9873;</span> Fetchers send how long each kind of document takes to process.
        A kind 100 times slower than a web page is queued two tiers back.
        The first 1,000 kinds are kept, saved and read back on a
        restart.</li>
        <li><span class="review">&#9873;</span> A quick look at squares of a picture skips OCR where none is
        like print. CMYK pictures in a PDF are read.</li>
        <li><span class="review">&#9873;</span> A fetcher packs the pages and links it sends at zlib level 1,
        and takes them off its list without moving the rest.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>PDF thumbnails draw JPEG 2000 photos, bit masks and turned
    text.</strong> A thumbnail is made again by a POST, which the five
    second limit on a GET does not stop.</li>
    <li><span class="review">&#9873;</span>
    <strong>A long dictionary merge no longer stops the crawl.</strong>
    Only closed partitions still to be built hold crawl batches back.
    In a crawl of about 1,550 partitions, one fetcher merged tiers 0 to 8
    from 19:02 to 20:01. The other fetchers got no batch from about 19:25
    to 20:03.</li>
    <li><span class="review">&#9873;</span>
    <strong>Fetchers reach a name server named 0.0.0.0.</strong> A request
    FetchUrl::getPage makes to 0.0.0.0 or :: goes to the loopback address,
    as Linux and macOS do already. Windows refuses 0.0.0.0, which a
    secure-start install had stored as its name server.</li>
    <li><span class="review">&#9873;</span>
    <strong>Scanned PDF pages decode in seconds, and slivers of pictures
    are not read.</strong> A JBIG2 page of 33 million pixels took 60
    seconds and over a gigabyte to decode. It now takes under 5 seconds
    and under 100 megabytes. Pictures under 8 pixels on a side are no
    longer handed to tesseract.</li>
    <li><span class="review">&#9873;</span>
    <strong>A cover drawn without pictures gets its own thumbnail.</strong>
    Such a page was refused, and a photograph from a later page stood in
    for it. Its title letters, colors and single-letter runs now land
    where the page puts them.</li>
    <li><span class="review">&#9873;</span>
    <strong>A magazine cover's thumbnail is made in a tenth of a
    second.</strong> It took 8 seconds, longer than a page request may run.
    Finding the catalog and filling each letter now work on the small part
    they need.</li>
    <li><span class="review">&#9873;</span>
    <strong>A dictionary tier merge writes a fifth of the bytes it did
    and runs three times faster.</strong> Each 32 KB data file made the
    writer list its folder and rewrite the folder's whole index. The writer
    now keeps those lists in memory. A merge writes each index once.</li>
    <li><span class="review">&#9873;</span>
    <strong>A dictionary key is found when its file name sorts out of key
    order.</strong> Tier indexes listed file names sorted as URL-encoded
    text, and = or a byte written as %XX sorts apart from its key. Indexes
    are now sorted by key when written and read. A lookup finds a key
    holding the bytes \xFE or \xFF in its escaped form.</li>
    <li><span class="review">&#9873;</span>
    <strong>Guessing a page's language takes under half the time it
    did.</strong> The Turkish stop word list held an empty word, which
    matched at every word boundary of every page guessed. Chinese
    normalizing now replaces characters with one strtr call.</li>
    <li><span class="review">&#9873;</span>
    <strong>A star in a robots.txt rule can match no characters.</strong>
    A rule starting with a star got a slash put before it, and each star
    had to match at least one character. So a disallowed /search was
    crawled.</li>
    <li><span class="review">&#9873;</span>
    <strong>A word in a page's host, title or path is filed in a first
    tier of the index.</strong> A query for a common word scored only the
    first 200 postings, in partition order. So a page naming the word in
    its title could be missed. Such pages are now filed under a field key,
    and a query reads that key as a disjunct of its own.
    <ol>
    <li><span class="review">&#9873;</span> IndexDocumentBundle adds the field key beside each combined key,
    in new bundles and in ArcTool rebuilds.</li>
    <li><span class="review">&#9873;</span> PhraseModel adds the disjunct where the index has FIELD_TERMS.
    UnionIterator reads its short list first.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A url holding a null byte no longer stops a fetcher.</strong>
    curl refuses such a url, and the error ended the fetcher's run. Nine
    such errors were in yioop.com's error log. getPages now sends the byte
    as %00, in the url and in the referer.</li>
    <li><span class="review">&#9873;</span>
    <strong>A fetcher spends less time on each page it processes.</strong>
    Page processing took 37% of the fetchers' time, and HtmlProcessor 20%.
    Each change below gives the same output as before on the saved pages.
    <ol>
    <li><span class="review">&#9873;</span> Scraper signatures are tested on the document object model (DOM)
    HtmlProcessor already builds. Parsing the page a second time for them took a third of
    HtmlProcessor's time.</li>
    <li><span class="review">&#9873;</span> A matched scraper reuses that DOM for its text path. It parses the
    page for extract fields only when it has some.</li>
    <li><span class="review">&#9873;</span> JSON-LD text goes after the body's opening tag. The body keeps its
    attributes, and a dollar amount such as $250 keeps its digits.</li>
    <li><span class="review">&#9873;</span> English stop words are removed by looking each word up in a set.
    That is about four times faster than the pattern of 676 words.</li>
    <li><span class="review">&#9873;</span> stemCharGramSegment replaces punctuation with Perl Compatible
    Regular Expressions (PCRE), not Oniguruma.
    Phrase extraction for indexing takes 26% less time.</li>
    <li><span class="review">&#9873;</span> tesseract runs with one OpenMP thread. Text recognition took a
    fifth of the time, with the same text read.</li>
    <li><span class="review">&#9873;</span> A tier merge keeps the last key it wrote, so it does not decode the
    last record for each row.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A fetcher downloads its next batch while it processes the
    current one.</strong> In 35% of yioop.com's download rounds, the fetcher
    sat 9 seconds or more waiting on its last few pages.
    <ol>
    <li><span class="review">&#9873;</span> The next batch downloads in a Fiber, and getPages hands control
    back on each pass of its wait. The batch is processed in between.</li>
    <li><span class="review">&#9873;</span> A batch downloading this way is held schedule, as to_crawl is. So a
    fetcher sends its results and asks for a schedule at the same points as
    before.</li>
    <li><span class="review">&#9873;</span> The download is finished before an upload, and a crawl change drops
    it with the rest of the old crawl's urls.</li>
    <li><span class="review">&#9873;</span> A test crawl of slow local sites, with 60 ms per page processed,
    fetched 35% more pages in four minutes.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>The crawler can sign its requests under Web Bot Auth.</strong>
    A site behind Cloudflare that has accepted a signed bot lets it past
    the challenge that stopped www.crunchyroll.com.
    <ol>
    <li><span class="review">&#9873;</span> It is on whenever the name server is reached by https, and needs no
    setting. The name server's https address is the Signature-Agent.</li>
    <li><span class="review">&#9873;</span> The name server keeps the Ed25519 key pair in SECURITY_DIR, beside
    its other keys, and makes it the first time.</li>
    <li><span class="review">&#9873;</span> A fetcher asks for the key pair in its crawl time request until it
    has one. The name server sends it only on a request that came by
    https.</li>
    <li><span class="review">&#9873;</span> getPages signs each https request with Signature-Agent,
    Signature-Input and Signature headers.</li>
    <li><span class="review">&#9873;</span> The web server serves the public key, signed, at
    /.well-known/http-message-signatures-directory.</li>
    <li><span class="review">&#9873;</span> The test checks the code against Cloudflare's example key and
    example signature.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A large sitemap is parsed only as far as the links it
    gives.</strong> In the yioop.com fetcher logs, 1,277 gzipped files took
    about 1,176 seconds. 1,273 of them were istockphoto sitemaps.
    <ol>
    <li><span class="review">&#9873;</span> XmlProcessor parsed each sitemap once for its root element.
    SitemapProcessor then parsed it again for its links.</li>
    <li><span class="review">&#9873;</span> XmlProcessor now reads the root element's name with a regular
    expression. SitemapProcessor::entriesPage cuts the text after the entry
    that gives the last link allowed, and closes the root element.</li>
    <li><span class="review">&#9873;</span> A cut-off sitemap is cut back to its last whole loc element. A
    piece of url before the cut no longer becomes a link.</li>
    <li><span class="review">&#9873;</span> A ten megabyte sitemap like istockphoto's now takes 0.03 seconds
    in CompressedProcessor, where it took 1.0 to 1.2. Seventeen sample
    documents give the same summaries as before.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A gzipped page keeps the crawl's link limit.</strong>
    CompressedProcessor made the processor for the uncompressed page with
    the summarizer option where the link limit goes.
    <ol>
    <li><span class="review">&#9873;</span> That processor took every link, and left 'dk' as every processor's
    limit. It now gets the limit, then the summarizer option.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>PackedTableTools reads and writes seven kinds of row correctly.</strong>
    Each fault has a test case that fails on the code before.
    <ol>
    <li><span class="review">&#9873;</span> findRowFromKeyTableString finds a first row whose key holds a byte
    save writes as two bytes, as a document id may.</li>
    <li><span class="review">&#9873;</span> It finds rows of a table whose keys vary in length.</li>
    <li><span class="review">&#9873;</span> add with ADD_FILE_PATH no longer calls fsize, which PHP does not
    have.</li>
    <li><span class="review">&#9873;</span> An empty table loads as no rows, not as one row with an empty
    key.</li>
    <li><span class="review">&#9873;</span> A negative INT of two or more bytes keeps its sign.</li>
    <li><span class="review">&#9873;</span> An INT size byte equal to the character 0 no longer reads every INT
    column as 0.</li>
    <li><span class="review">&#9873;</span> findEntryAtIndexTableName gives nothing for a row number past the
    last, not the whole table.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A partition builds in less time.</strong>
    Fetchers built partitions for about a third of their time in the logs
    of October 13.
    <ol>
    <li><span class="review">&#9873;</span> The file naming the document being built was opened and closed for
    two writes a document. It is now opened once and written over.</li>
    <li><span class="review">&#9873;</span> addTermPostingLists writes each posting's bytes itself, with
    PackedTableTools::packIntColumn, and makes a word's canonical key once.</li>
    <li><span class="review">&#9873;</span> NWordGrams::ngramsContains keeps up to NGRAM_LOOKUPS_KEPT answers, a
    new Config.php value, so a phrase met again is not hashed again.</li>
    <li><span class="review">&#9873;</span> hyphenateEntities lowercases each word once and joins short
    entities without a slice.</li>
    <li><span class="review">&#9873;</span> A 4,000 page English partition built in 39.8 seconds before and
    18.5 after. A 10,000 page partition in many languages took 140 and
    95.</li>
    <li><span class="review">&#9873;</span> The postings, document map and last entries files came out the same
    byte for byte.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A position list with a long gap reads back as written.</strong>
    encodePositionList wrote some Rice codes wrong, so 3, 6, 2087 read back
    as 3, 67108868, 67108869.
    <ol>
    <li><span class="review">&#9873;</span> It now gathers bits in an integer and writes them a byte at a time,
    which also takes about a fifth less time.</li>
    <li><span class="review">&#9873;</span> Of 390,829 lists from the help page build, 10,789 were written
    differently, and every changed list now reads back as written.</li>
    <li><span class="review">&#9873;</span> A list with a gap of 0 has no gamma code. It is written as
    before.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>Searches and the indexer report where their time goes, and the
    indexer compresses faster.</strong>
    yioop.com's crawl logs showed the indexer idle 0.04% of each hour.
    In 8.5 minutes it indexed 20 uploads while fetchers sent at least 39.
    <ol>
    <li><span class="review">&#9873;</span> AnalyticsManager::addTiming counts and times each kind of step, and
    timingsReport lists them.</li>
    <li><span class="review">&#9873;</span> QueryTool and the query statistics list dictionary, postings,
    positions, document map and latest copy reads.</li>
    <li><span class="review">&#9873;</span> The indexer logs how many index data files wait, and times
    uncompressing, compressing, writing and saving.</li>
    <li><span class="review">&#9873;</span> GzipCompressor uses ARCHIVE_COMPRESSION_LEVEL, 6, in place of 9.
    Indexing a 700 page upload took 24% less time.</li>
    <li><span class="review">&#9873;</span> AnalyticsManager.php's docblocks are brought up to standard.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>Crawl Status has a Crawl Speed History of the crawl's
    speed over time.</strong>
    The crawl status gave only the rate over the last hour since the queue
    server started.
    <ol>
    <li><span class="review">&#9873;</span> A media job, CrawlSpeedJob, runs on the name server each
    CRAWL_SPEED_SAMPLE_INTERVAL, ten minutes, so it needs no queue server
    there.</li>
    <li><span class="review">&#9873;</span> It reads the crawl status from the queue servers and logs a line
    for each running crawl to CrawlSpeed.log in the log folder.</li>
    <li><span class="review">&#9873;</span> CrawlModel::crawlStatus reads the lines into each crawl's
    SPEED_HISTORY.</li>
    <li><span class="review">&#9873;</span> A [Crawl Speed History] link after Most Recent Fetcher shows the
    rows newest first, in a table of fixed height that scrolls.</li>
    <li><span class="review">&#9873;</span> An open table stays open when the status refreshes, through
    crawl_toggle.</li>
    <li><span class="review">&#9873;</span> CrawlstatusView.php's docblocks are brought up to standard.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>The scheduler looks up a fetch batch's etags in bucket
    order.</strong>
    In yioop.com's logs of 15 October, fetchers sat idle because no
    schedule was ready. Writing batch 8490 of 1,301 urls took 17.3 s.
    <ol>
    <li><span class="review">&#9873;</span> Fetcher 7 slept on "No data" 18 times in 22 minutes, 180 s of
    1,318 s, close to its logged 15% idle.</li>
    <li><span class="review">&#9873;</span> Each url of a batch was looked up in the etag table alone. A
    lookup reads its bucket's index unless index_buffer holds it, and the
    buffer is emptied past INDEX_CACHE_SIZE, 100, buckets.</li>
    <li><span class="review">&#9873;</span> LinearHashTable::getMany orders a batch's keys by bucket, so each
    bucket index is read once. produceFetchBatch uses it and logs the
    time the lookups took.</li>
    <li><span class="review">&#9873;</span> For 1,300 urls in a 20,000 url table of 64 record buckets, the
    lookups took 0.021 s against 0.056 s, finding the same 636.</li>
    <li><span class="review">&#9873;</span> A merge of partition 1873 had 9 s with no log line before it began
    and 8 s after "Switch Partition time". Fetchers now log how long
    they look for a partition to build. Merges log the time to open the
    bundle, record the merge and merge tiers.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>Builds keep the stems of words, and fetchers log the load
    average.</strong>
    Building a partition took 75.6 s on yioop.com, 31.4 s of it finding
    phrases. In a profile here, stemming took 13.5% of a build.
    <ol>
    <li><span class="review">&#9873;</span> Tokenizer::stem keeps up to STEMS_KEPT, 50,000, words' stems, and
    stemWord stems a word not seen before.</li>
    <li><span class="review">&#9873;</span> Building partition 1 of a test index took 46.6 s against 51.9 s,
    and extractPhrasesInLists 21.3 s against 27.2 s. The partition's
    files came out the same.</li>
    <li><span class="review">&#9873;</span> HtmlProcessor took about 30% longer per page after fetchers began
    compressing pages at level 6. "Process pages time" now carries the
    load average, to show whether other processes held the cores.</li>
    <li><span class="review">&#9873;</span> A fetcher deflated a page's gzip bytes again inside the message,
    which cannot shrink them. packSeenSite now sends the page apart, after
    PAGE_APART_MARK and the rest of the site.</li>
    <li><span class="review">&#9873;</span> For 600 sites of 60,000 byte pages, packing took 1.21 to 1.41 s
    against 1.24 to 1.59 s. Unpacking took 0.003 s against 0.038 s, and
    the message was 0.8% smaller. The archive's bytes are the same.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>Builds split phrases into words faster.</strong>
    <ol>
    <li><span class="review">&#9873;</span> Tokenizer::stemPhrase splits a phrase with preg_split on runs of
    Unicode space, where mb_split took five times as long. Eleven phrases
    with tabs, no-break and other Unicode spaces split the same both
    ways.</li>
    <li><span class="review">&#9873;</span> With kept stems as well, a build of partition 1 took 49.1 s
    against 51.7 s with kept stems alone. Its postings file came out the
    same.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>The words of an entity are stemmed once.</strong>
    For "cs dept san jose state university", yioop.com ranked other pages
    above San Jose State University's own pages.
    <ol>
    <li><span class="review">&#9873;</span> A page with "San Jose State University" files the entity
    san-jose-state-univers. Its words were found by splitting the stemmed
    entity and stemming each part again. So "university" was filed as
    "univ", but a query for "university" looks up "univers".</li>
    <li><span class="review">&#9873;</span> The words of an entity are now taken from the entity before it is
    stemmed. The local test index has no entity whose last word changes
    when stemmed twice, so its partition 1 came out the same.</li>
    <li><span class="review">&#9873;</span> A word with diacritic marks on Latin letters, such as josé, is
    also filed without them, as jose. foldDiacritics uses PHP's
    Normalizer, or a table of Latin letters where intl is missing.</li>
    <li><span class="review">&#9873;</span> Pages already built keep the old words until they are built
    again.</li>
    <li><span class="review">&#9873;</span> Comments added in the last patch are rewritten so a developer who
    has not read the rest of the class can follow them.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>The scheduler's etag lookups search bucket files as
    strings.</strong>
    On yioop.com, looking up the etags of 226 to 1,664 urls took 2.7 to
    3.6 s. The lookups found 0 to 173 in a table of 4,953,658 rows.
    <ol>
    <li><span class="review">&#9873;</span> At 16,384 rows a bucket, the table has about 300 buckets, and
    index_buffer keeps 100. getMany turned each bucket index it read into
    an array, which took 4.1 ms for a full bucket here. Searching the
    file's string for the key took 0.16 ms.</li>
    <li><span class="review">&#9873;</span> getMany now searches the string. A bucket index already in
    index_buffer is used as it is, since it may hold rows not yet saved.
    get and getMany share recordFromRow, which turns a found row into a
    record.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>Fetchers encode pages while they wait on downloads, not while
    they send.</strong>
    In the yioop.com fetcher logs, sends took about a fifth of each
    fetcher's time. Compressing seen urls took 15.1 s a send.
    <ol>
    <li><span class="review">&#9873;</span> A profile of that step on 300 real pages put 82% of it in
    packSeenSite's gzip encoding of each page.</li>
    <li><span class="review">&#9873;</span> A download round lasted 7.1 s on average, but 90% of its
    transfers were done by 5.6 s.</li>
    <li><span class="review">&#9873;</span> FetchUrl::getPages calls a function while it waits. The fetcher's
    compressWaitingPage encodes one waiting page each call.</li>
    <li><span class="review">&#9873;</span> packSeenSite uses those bytes, and the string sent is the same.
    The send step on the 300 pages took 0.60 s against 4.85 s.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>Dictionary merges read each term's occurrence count straight
    from the last_entries file.</strong>
    Merging the three partitions of a test index took 26.0 s against
    34.4 s. The dictionary's data files came out the same.
    <ol>
    <li><span class="review">&#9873;</span> A profile put about a quarter of a merge in loading last_entries
    as an array, then finding and unpacking each term's row.</li>
    <li><span class="review">&#9873;</span> addPartitionPostingsDictionary now reads NUM_OCCURRENCES from each
    row in one pass over the file. A row of another shape is unpacked in
    full.</li>
    <li><span class="review">&#9873;</span> The progress line of the merge loop is formatted only when it is
    written.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>Builds split words with preg_split and join entities with one
    pass less.</strong>
    Building partition 1 of a test index took 43.4 and 41.8 s against
    44.6 and 44.7 s. Its postings and positions came out the same.
    <ol>
    <li><span class="review">&#9873;</span> stemTermsK split text with mb_split. preg_split on the same space
    characters gave the same terms in a quarter of the time.</li>
    <li><span class="review">&#9873;</span> hyphenateEntities lower cases the text once, and goes on to the
    next word after an entity without an empty step between.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A rebuild stops a stuck builder and goes on, and mb_ereg calls
    become preg or string calls.</strong>
    A rebuild of yioop.com stopped at partition 698 when one builder sat
    151 s on one document.
    <ol>
    <li><span class="review">&#9873;</span> ArcTool now stops a stuck builder and hands its partition out
    again. The next builder skips the document the stopped one was on.</li>
    <li><span class="review">&#9873;</span> A stuck merger still stops the rebuild, as a merge run twice could
    add rows twice.</li>
    <li><span class="review">&#9873;</span> mb_ereg_replace and mb_split calls became preg_replace,
    str_replace or explode. They were in PhraseParser, PhraseModel,
    SearchController, UrlParser, the RSS and SVG processors, the word
    filter, the Greek tokenizer and TokenTool.</li>
    <li><span class="review">&#9873;</span> On 2,028 texts, including every Unicode space character, each new
    call gave the same result, four to eleven times faster.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>The Chinese word splitter joins at most 64 characters into one
    unknown term, so its time grows in step with the text.</strong>
    A page of yjt.xizang.gov.cn held 8,564 characters of base64 text, and
    building its words took minutes.
    StochasticTermSegmenter::segmentSentence joins runs of non-Chinese
    characters, numbers and punctuation into terms.
    It tested the growing string at each step from every start, so a run
    of m characters cost m cubed.
    The walk from each start now stops after MAX_JOINED_LENGTH characters.
    A run of 8,000 dashes took 0.24 s, against 33 s for 4,000 before.</li>
    <li><span class="review">&#9873;</span>
    <strong>Merges sync the files they write with one file system sync on
    Linux.</strong>
    A tier merge and a finished merge step synced each data file on its
    own, waiting for one journal commit per file.
    UndoLog::syncPaths runs sync -f once on Linux and syncs file by file
    elsewhere or when that fails.
    A local tier merge of 3,430 files went from a median of 5.67 s to
    4.84 s with the same files.</li>
    <li><span class="review">&#9873;</span>
    <strong>A page whose download fails in a way that may pass is fetched
    again later.</strong>
    A page with no answer on its second try in a batch was dropped
    silently.
    A server error ended a page for the crawl.
    Fetchers now send such pages back in RETRY_URLS, and the Scheduler
    queues each again up to FETCH_TRIES_MAX times.
    A 403, a 404 and other final codes are not retried.</li>
    <li><span class="review">&#9873;</span>
    <strong>Each https connection sends a different TLS ClientHello, and
    a refused landing or seed page may be tried again.</strong>
    Every https fetch sent the same ClientHello, so a site could block
    Yioop by its JA3 or JA4 fingerprint.
    FetchUrl::varyTlsHello now offers a random choice and order of cipher
    suites and groups, and sometimes asks for HTTP/1.1.
    With robots.txt set to Allow Landing Page or Ignore, a 403 on a seed
    or landing page is queued again like a server error.</li>
    <li><span class="review">&#9873;</span>
    <strong>A site whose name leads to the fetcher's own machine is not
    indexed with that machine's pages.</strong>
    www.tovinothomas.com had the address 127.0.0.1.
    A fetcher sent it to its own web server and indexed yioop.com's home
    page under that name.
    getPages now refuses such a url before connecting when the queue
    server gave the address.
    Otherwise it drops the page once curl shows a loopback or 0.0.0.0
    address.
    The page gets code 421, so a robots.txt with it stops the host.
    A url for one of localHosts(), or for an address, is still fetched.</li>
    <li><span class="review">&#9873;</span>
    <strong>Each crawl's speed history has its own Crawl Speed Data
    page.</strong>
    A running crawl's [Crawl Speed History] link opens it.
    So does the Visited/Extracted Urls count of a crawl in the Crawls
    table.
    The page heading names the crawl, and a close button returns to Manage
    Crawls.
    The manageCrawls argument speedhistory reads the rows through
    CrawlModel::crawlSpeedHistory, and CrawlspeedElement draws them.
    The table that opened inside Crawl Status is gone.</li>
    <li><span class="review">&#9873;</span>
    <strong>The Crawl Speed Data table is taller, and the feed crawl's
    counts are not a link.</strong>
    The table now scrolls past 36rem rather than 24rem.
    The media updater makes the feed crawl and logs no speed for it.</li>
    <li><span class="review">&#9873;</span>
    <strong>A fetcher that has held a schedule for two minutes sends what
    it has, so the other fetchers keep getting schedules.</strong>
    The queue server takes schedules back in the order it sent them.
    In the logs of 18 October one fetcher held schedule 415 for 683
    seconds on pages of www.olx.ua.
    The send window stayed full nearly that long, and fetchers slept
    2,030 seconds in it.
    Fetcher::ANSWER_SCHEDULE_AFTER now makes the fetcher send once the
    schedule is that old.
    Later sends of the same schedule are taken as before.</li>
    <li><span class="review">&#9873;</span>
    <strong>A slow HTML page's log line names the steps it spent its time
    in.</strong>
    HtmlProcessor::process keeps the seconds of each step in
    $step_times.
    The Fetcher adds them to its Slow page line.
    The olx.ua pages took about four seconds each, and the cause is not
    yet known.</li>
    <li><span class="review">&#9873;</span>
    <strong>A fetcher that stops on a page goes on with its schedule
    without that page when it restarts.</strong>
    On 18 October fetchers stopped on two lobbying spreadsheets at 16:08,
    16:25, 16:42 and 17:16.
    Each restart dropped the schedule, so the window waited 17 minutes and
    the next fetcher stopped on the same page.
    The Fetcher now keeps the url being processed and the schedule's
    unsent urls in SCHEDULES_DIR.
    On restart it skips that url and goes on, unless the schedule has
    been sent elsewhere.
    A fatal PHP error is now written to the fetcher's log with the
    page.</li>
    <li><span class="review">&#9873;</span>
    <strong>A PDF gets at most five seconds of picture reading.</strong>
    In the 18 October logs one batch spent 266 of its 294 PDF seconds in
    tesseract, on two nissan-global.com PDFs of 112 and 136 seconds.
    getText read every picture while the words found stayed shorter than
    a summary.
    It now starts no further reading once a document has had
    MOST_READING_SECONDS of it.
    A made-up PDF of 25 charts each holding one word went from 14.9 to 5.8
    seconds, and a slide PDF read the same 17,314 characters as before.</li>
    <li><span class="review">&#9873;</span>
    <strong>The five second limit on reading a PDF's pictures is taken
    out.</strong>
    It did not change the crawl's pages an hour.
    PDFs past five seconds took at most 454 of 7,723 fetcher seconds in
    the 18 October logs.
    That is under 6 percent, within the speed table's own swings.
    The limit also lost words of picture-heavy PDFs, so it went.</li>
    <li><span class="review">&#9873;</span>
    <strong>Pages with no text are no longer grouped as duplicates, and
    hyphenated hosts and titles give the hyphenated term.</strong>
    www.pepsi.com's info: entry led to www.cyon.ch's page, and a pepsi
    search could not find it.
    Both pages had no text, so the same content hash, and the partition
    kept only the shorter url's words.
    computePageHash now hashes such a page by its url.
    A coca-cola search gave www.coca-cola.com neither its host nor its
    first title bonus.
    Host words now keep coca-cola whole, Unicode's hyphens become the plain
    hyphen, and ownership signs such as the registered sign leave terms.
    Pages indexed before the patch keep their old terms.</li>
    <li><span class="review">&#9873;</span>
    <strong>A fetch opens no file, so a restarted fetcher no longer stops,
    and challenge pages and Drupal menus stay out of the index.</strong>
    A restarted fetcher held every connection the web server had open.
    Under a limit of 1024, its first batch could not open curl's record
    files.
    getPages then handed curl false and the fetcher stopped, 226 times on
    18 and 19 October.
    getPages now reads the address and request from curl, so it opens no
    file, and each fetcher has its own temporary folder.
    A page holding Imperva's challenge script and no text is fetched again
    later, not indexed.
    The Drupal scraper also reads main tags, through database version
    131.
    Page processors now write temporary files in their fetcher's folder.
    A redirect's link adds the redirecting host's words, such as coke, to
    the page it leads to.
    A home page redirect also links the target's home page, which counts
    those words as host words.</li>
    <li><span class="review">&#9873;</span>
    <strong>A document whose words are not all UTF-8 gets its thumbnail
    with no deprecation notice.</strong>
    TextProcessor::drawText read the words with a UTF-8 pattern, which
    gives null on a stray byte.
    wordwrap was then handed null, a notice on PHP 8.1 and later.
    The words are now scrubbed to UTF-8 first.</li>
    <li><span class="review">&#9873;</span>
    <strong>Sentence splitting reads a long run of letters once, and a slow
    page line names the slow part of a summary.</strong>
    Three of ten fetchers spent about 75% of their time summarizing
    charter.com newsroom pages, at 26 to 49 seconds a page.
    The sentence end pattern reread a run of letters from each of its
    letters, so its time grew as the square of the run.
    From about 7,000 letters on it gave up, and the block's text was lost.
    The blank line pattern repeated a group once per newline and gave up
    the same way.
    Both now take time in step with the text.
    ScrapeSummarizer times its parts, and HtmlProcessor adds them to the
    slow page line with block, sentence and byte counts.</li>
    <li><span class="review">&#9873;</span>
    <strong>A home page with no text of its own is indexed by the words of
    its url when landing pages are allowed.</strong>
    https://www.pepsi.com/ answers with an Incapsula bot challenge, which
    the fetcher dropped, so the site never reached the index.
    In a crawl whose robots.txt mode is Allow Landing Page, such a home page
    is now indexed by its host words on the Scheduler's last try.
    The schedule's first line lists those urls as LAST_TRY_URLS, so an
    earlier real copy is not replaced while tries remain.
    A home page that answers with no text is described the same way and
    keeps its title.
    A host that never answers is not indexed.</li>
    <li><span class="review">&#9873;</span>
    <strong>A redirect's record keeps the urls it went through.</strong>
    The fetcher set the LOCATION field to array_push's count, not the
    list, so the redirect's urls were lost.
    PhraseParser now gives those urls location: words only.
    An info: word would let a redirect stand in for its target's
    page.</li>
    <li><span class="review">&#9873;</span>
    <strong>A page with a megabyte of inline style keeps its text.</strong>
    The whole corporate.charter.com newsroom page has 1.8 MB, mostly
    inline style.
    The lazy pattern removing style elements hit pcre.backtrack_limit and
    returned null, so no text was found.
    L\removeTagSpans now removes script, noscript and style elements by
    string search.
    HtmlProcessor, TextProcessor, Summarizer, FetchUrl and AddressesPlugin
    use it.
    On 100,039 test pages it gave what the patterns gave wherever they did
    not give up.</li>
    <li><span class="review">&#9873;</span>
    <strong>Tokenizers find their shipped lexicons and character map
    again.</strong>
    The English and Hindi lexicons and the Chinese character map were read
    only from the site's locale folder.
    A site with no such folder got no part of speech tags and no simplified
    characters, so three unit test cases failed.
    They are now found with localeResource, which falls back to
    src/locale.
    The Hindi lexicon's lines end in \r\n, and the \r is no longer kept in
    a word's tag.
    The English stemmer case no longer ends the whole test run at its first
    wrong stem.</li>
    <li><span class="review">&#9873;</span>
    <strong>A fetcher's processing time line names more of its
    rest.</strong>
    The next batch's download passes and the link checks are timed apart
    from the rest.
    In the logs of 21 October the rest was 719 of about 2,900 seconds of
    processing.</li>
    <li><span class="review">&#9873;</span>
    <strong>A url goes into a fetch batch once.</strong>
    The queue can hold a url more than once, a landing page above all.
    The Scheduler put every copy in the batch.
    In the logs of 23 October about one page in ten a fetcher processed
    was a copy; fetcher 4 processed http://www.jyb.cn/ 39 times in two and
    a half minutes.
    produceFetchBatch now keeps the urls a batch holds and drops a
    copy.</li>
    <li><span class="review">&#9873;</span>
    <strong>A fetcher's send queues each url once.</strong>
    A fetcher sends one copy of a url for each page it found the url on.
    The url filter removes only urls queued by earlier sends.
    So every copy of a url new to the filter went into the queue, and
    later into a fetch batch.
    processToCrawlArchive now skips a url it already queued from the same
    send.</li>
    <li><span class="review">&#9873;</span>
    <strong>Chinese text is segmented in about half the time.</strong>
    Since the tokenizers found their shipped files, normalize maps
    traditional characters to simplified ones.
    It used strtr, which builds a table of 3,964 characters on each call.
    The named entity tagger calls it once a word, so it took 40 percent of
    the time of reading a Chinese description.
    A short text is now mapped a character at a time.
    The segmenter also keeps the words of the runs it segmented lately.
    A description is segmented to be indexed and again to find its
    questions and answers.
    A 2,000 character description takes 61 to 72 milliseconds, from 117 to
    152.</li>
    <li><span class="review">&#9873;</span>
    <strong>The centroid weighted summarizer times its parts.</strong>
    The crawl uses it, and its slow page lines said only
    &ldquo;summary&rdquo;.
    They now name the sentences, the term frequencies, the average, the
    scoring and the putting together.</li>
    <li><span class="review">&#9873;</span>
    <strong>A fetcher spends less of its time summarizing pages and
    packing them to send.</strong> In the crawl logs of 24 October the
    summarizer took 369 of 410 seconds on 130 slow pages. Packing a send
    took 815 seconds over 60 sends, mostly working out unused terms.
    <ol>
        <li><span class="review">&#9873;</span>
        <strong>The summarizer counts a sentence's terms from the list it
        split the sentence into.</strong> It searched the sentence for each
        term, time that grew with length times terms. That search counted
        no Chinese word and no term such as #tag.</li>
        <li><span class="review">&#9873;</span>
        <strong>A send works out only a page's question answers.</strong>
        extractPhrasesInLists takes a question answers only parameter. A
        language that cannot answer questions skips the work, and a link no
        longer gets the answers of the page packed before it. Packing 2,000
        pages of eight languages went from 8.8 to 2.3 seconds.</li>
        <li><span class="review">&#9873;</span>
        <strong>Pages tagged fr, vi or el get their names joined as
        entities.</strong> NWordGrams::filter finds the list in the fr_FR,
        vi_VN or el_GR folder. A language with no list no longer looks for
        its file on every phrase.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A dictionary merge no longer checks every tier file.</strong>
    Each tier merge synced the merged tier a second time. It then added
    every file of it to the dictionary's checkpoint, and that checkpoint
    came to list every tier file. Each open for writing read the size of
    each one, which took 18 to 28 seconds before each partition merge on
    pollett.org. Tier files are written once and synced before they take
    their place, so they are no longer checkpointed. An old checkpoint is
    dropped when the dictionary is opened. Tier files now hold up to
    256KB in place of 32KB, so a merge syncs and deletes an eighth as many
    files. A merged tier is moved aside and deleted later, a second's worth
    after each file the indexer takes. Merging two tiers of 800,000 rows
    here took 1.15 seconds in place of 1.77. A tier merge now copies
    rows as the tier files store them and decodes only keys. Where both
    tiers hold a key, the two rows are joined without decoding their
    records. Merging 20,000 keys of 200 records each took 0.4 seconds in
    place of 0.55. The bundle's checkpoint no longer lists tier files, so
    fetchers stop reporting them gone. Nor does it list a built
    partition's files, which never grow, so a build's end no longer reads
    every file the checkpoint names. After a partition merge, tier 0 and
    at most one higher tier now merge. On pollett.org tiers 0, 1 and 2
    merged at once in 28 seconds, 4, 8 and 15 each. That work is now
    spread over several partitions. No more than two tiers are left full, since a
    lookup reads both slots of a full tier. A lookup walks the tiers again
    when a merge's two renames fall inside its walk. A merge count kept in
    the tree's folder is odd between them. So a walk never gets a word's
    rows twice or not at all. A fetcher's slow page line now gives the
    page's language and length. hyphenateEntities no longer trims words
    split on white space. A picture more than 1,600 points on a side is
    scaled down before tesseract reads it. A 2,400 point wide picture
    read the same at half size, in 0.25 seconds against 0.33 to 0.40.
    In the logs of 26 October, compressing summaries was still 3.7 of
    the indexer's 6.6 seconds per file. So fetchers now compress each
    summary, as they do pages, and the indexer stores those bytes as
    sent. A summary still gets its url hash added before it is
    compressed. A sentence of more than 512 characters, as on pages
    with no periods, is now cut into pieces at spaces. A Bloom filter
    lookup that unpacked one word at a time gave no gain above noise,
    so it was not kept. The crawl then fell from 200,000 to 240,000 pages
    an hour to 160,000 to 180,000. Each fetcher still sent the whole site
    beside its encoded summary, so every summary was compressed and sent
    twice. A send carried 1,700 to 6,600 bytes a site, against 740 to
    920 in that morning's logs. The site part now holds only the six fields
    the indexer reads. Packing 2,000 test sites took 0.30 seconds and 7.8
    MB, against 0.55 seconds and 15.0 MB. With that, the crawl still ran
    under 180,000 pages an hour, so both changes are taken back. Fetchers
    again send the site whole and the indexer compresses summaries. The
    cap on sentence length stays. The rate had not fallen. The 200,000 to
    240,000 came from the indexer working off a backlog of 1,726 fetcher
    files on 25 October. It held 592 at 07:41 on 26 October and none by
    17:56. Each file raised the count by about 650 pages, about 70,000 an
    hour while the backlog lasted.</li>
    <li><span class="review">&#9873;</span>
    <strong>The indexer stores a fetcher's file in less time.</strong>
    On pollett.org it spent half of each file compressing summaries at
    gzip level 6, and 14% saving the partition index after each slice.
    Summaries are now stored at SUMMARY_COMPRESSION_LEVEL, 3, and pages
    stay at 6. The partition index is kept in memory over a file's slices
    and saved once at its end. So searches see a file's pages seconds
    after it arrives. Storing 60,000 pages here took 3.3 seconds in place
    of 5.0. A writable open of the bundle now logs its checkpoint check's
    time. Opening it to merge a partition took 18 to 28 seconds. A slice
    here is a group of up to 5,000 sites of one file. The constant is now
    SITES_STORED_AT_ONCE, and the comments about it say so in full.</li>
    <li><span class="review">&#9873;</span>
    <strong>Crawl Speed Data shows the indexer backlog and the fetchers'
    downloads per hour.</strong> Visited Urls/Hour counts pages as the
    indexer stores them. So it read 240,000 while the indexer worked off a
    backlog, with fetchers at 175,000. Two columns now show both.
    <ol>
        <li><span class="review">&#9873;</span> Each queue server puts its
        count of fetcher files waiting in its crawl status as
        INDEXER_BACKLOG. aggregateStatuses sums it over the machines.</li>
        <li><span class="review">&#9873;</span> A fetcher sends its
        downloads per hour with each crawl time check. FetchController keeps
        it in the robot table, and fetcherDownloadsPerHour sums the
        fetchers heard from in the last ten minutes.</li>
        <li><span class="review">&#9873;</span> CrawlSpeedJob logs both
        in two more columns, and an older line shows them empty. On a phone
        the numbers stay whole and the table scrolls sideways under a fixed
        Time column.</li>
        <li><span class="review">&#9873;</span> The columns follow a page
        through the crawl: Time, Downloaded/Hour, Index File Backlog,
        Indexed Pages/Hour, Indexed Page Count, Total Urls Seen. The four
        renamed labels are new strings in all 25 locales.</li>
        <li><span class="review">&#9873;</span> Downloaded/Hour read 0 on
        Chris's crawl, so the fetchers' rate no longer comes from
        MediaUpdater reading the name server's robot table. A fetcher sends
        its rate with each upload of data. Each queue server keeps the rates
        of the fetchers heard from in the last ten minutes in its crawl
        status as FETCHER_RATES. aggregateStatuses keeps one rate per
        fetcher, the newest, and sums them as FETCHER_DOWNLOADS_PER_HOUR,
        which CrawlSpeedJob logs as it does the backlog.</li>
        <li><span class="review">&#9873;</span> A run on the local test
        system found two causes of the 0. A web server that loaded
        FetchController before the update keeps the old class until it
        restarts. That class writes no rate in the robot table. And a
        fetcher's first downloads per hour divided by the hours since 1970.
        Fetcher::ratePerHour now keeps the window of each per-hour figure
        the fetcher logs, and the first window starts when the fetcher was
        made. On the local system the first figure read 360 and the speed
        log read 327.</li>
        <li><span class="review">&#9873;</span> CrawlSpeed.log lines no
        longer hold the crawl's channel, which was 0 on every line of
        Chris's log. crawlSpeedHistory reads lines with and without
        it.</li>
        <li><span class="review">&#9873;</span> In Chris's fetcher logs of
        27 October, 848 pages of vx-3.com took 3,033 of the 4,470 seconds
        of HTML processing, 3.6 s each, mostly the summary's scoring step.
        scoreSentencesVersusAverage now scores each sentence once and lowers
        only the scores a pick changes. The slow page line also gives how
        many terms the sentences hold, to show why such pages are slow.</li>
        <li><span class="review">&#9873;</span> The vx-3.com page Chris
        sent explains the slow summaries. The fetcher keeps the first
        996,326 characters of a 1.7 MB page, which ends inside a script.
        removeTagSpans now takes out an element with no close tag, up to
        the end of the page. That script's JSON is no longer read as 1,189
        sentences, and the page's summary takes 0.02 s, not 0.76 s.</li>
        <li><span class="review">&#9873;</span> A host may give a fetcher
        SLOW_PAGES_BEFORE_FAST slow pages in an hour. Its pages then get
        the fast variant of their processor until the hour ends. For HTML
        that is HtmlProcessor with the ScrapeSummarizer. Each fetcher keeps its
        own counts.</li>
        <li><span class="review">&#9873;</span> The logs of 27 October show
        uploads taking about 27 percent of fetcher time. Sending was 3,154
        parts of 2 MB, 12 percent. Each fetcher now keeps a part size for
        each queue server. It starts at DEFAULT_POST_MAX_SIZE and grows to
        the limit the server reports. When a size fails twice in a row, the
        fetcher restarts the upload. It then searches between the largest
        size that went through and the smallest that failed.</li>
        <li><span class="review">&#9873;</span> Under Yioop's own web server,
        FetchController reports MAX_REQUEST_LEN less POST_HEADER_ROOM as
        its largest post, not PHP's post_max_size. That server does not
        use post_max_size. The first part of an upload now starts its file
        again, and the file name keeps only hash characters.</li>
        <li><span class="review">&#9873;</span> Under Yioop's own web server,
        index.php raises MAX_REQUEST_LEN to the Git push size limit. That
        limit is none by default, so fetchers were told they could post
        any size. FetchController now gives post_max_size, raised to the
        web server's default and kept under MAX_REQUEST_LEN.</li>
        <li><span class="review">&#9873;</span> ARCHIVE_COMPRESSION_LEVEL is
        now 4, as Chris chose. On sample pages it takes 0.293 ms a page
        against 0.532 ms at level 6, with output 2.4% larger.</li>
        <li><span class="review">&#9873;</span> index.php no longer raises
        the web server's request limit to the Git push size limit, as Chris
        asked. The limit follows post_max_size only, raised to the web
        server's default. GIT_MAX_PUSH_SIZE stays a cap of its own on git
        pushes under that limit.</li>
        <li><span class="review">&#9873;</span> GIT_MAX_PUSH_SIZE is gone, as
        Chris asked; a git push is held to post_max_size. Its field on the
        users, roles and groups page and its locale string are gone too.</li>
        <li><span class="review">&#9873;</span> Profile.php holds only fields a
        page lets an administrator change. GROUP_ITEM and the four
        PRIVATE_DB fields left the list and come from Config.php. Database
        version 137 writes Profile.php again without them.</li>
        <li><span class="review">&#9873;</span> Chris's crawl kept reading 0
        after every machine restarted. His error log of 15 October shows the
        robot table write failing with no space left on the device. That
        write emptied the table, and FetchController never wrote an
        unreadable table again. The queue server then read nothing on every
        pass. FetchController::recordRobot now starts such a table again. It
        changes the table under an exclusive lock and makes it readable by
        the crawl processes. The queue server reads it under a shared
        lock.</li>
        <li><span class="review">&#9873;</span> SmtpClient::smtpCommand
        notes in its log a command written to a closed connection. The
        notice it raised before showed up in a suite run under load.</li>
    </ol></li>
</ol></li>
<li><span class="next">&#9654;</span> <strong>An index is never left corrupt
    by a crawl that stops part way through.</strong>
Phase 4 added guards that keep a corrupt index
from taking the server down. Those are guards rather than prevention.
This item is the prevention.
<ol>
    <li><span class="review">&#9873;</span> <strong>Every point that
        could leave a half-written entry is named.</strong>
The walk covers the partition append, the dictionary
merge and the postings write. What is looked for is a write that a
later read takes as a garbage length or offset. Nine were found.
        <ol>
            <li><span class="review">&#9873;</span> PartitionDocumentBundle::put and
                advanceSavePartition save the partition index in
                place. A kill part way leaves offsets and
                lengths that point at garbage.</li>
            <li><span class="review">&#9873;</span> PartitionDocumentBundle::addArchive appends each
                document to the partition file. A kill part way
                leaves a partial tail that no index entry names
                yet.</li>
            <li><span class="review">&#9873;</span> IndexDocumentBundle::buildInvertedIndexPartition
                writes doc_map, postings, last_entries and
                positions in place, one after another. A kill
                part way leaves a mix.</li>
            <li><span class="review">&#9873;</span> mergeNextPartition checks only that those four
                files exist, so it can merge a truncated
                postings file into the dictionary.</li>
            <li><span class="review">&#9873;</span> IndexDocumentBundle::updateDictionary writes
                next_partition.txt in place. A kill part way
                leaves it empty, and it reads as partition
                0.</li>
            <li><span class="review">&#9873;</span> LSMTree::flush and writeRecords write tier data
                files and file lists in place. A kill part way
                leaves a record list with a cut last line.</li>
            <li><span class="review">&#9873;</span> LSMTree::mergeTier writes the merged tier while
                both source tiers still exist. A kill part way
                leaves a partial target beside complete
                sources.</li>
            <li><span class="review">&#9873;</span> LSMTree::getMaxTier and
                IndexDocumentBundle::setArchiveInfo write small
                files in place; the parameter files alone go
                through a rename.</li>
            <li><span class="review">&#9873;</span> The build, merge and slot claims are opened with
                x, so a kill leaves only a stale claim, which
                the timeout already clears.</li>
        </ol></li>
    <li><span class="review">&#9873;</span> <strong>An undo log guards
        each write point named in A.</strong>
Before a step changes any index file, it records how to undo the change.
When the index next opens, any step left unfinished is undone.
        <ol>
            <li><span class="review">&#9873;</span> The log is one file in the bundle folder. A step appends a
            record naming itself and each file it will change, then syncs
            the log.</li>
            <li><span class="review">&#9873;</span> For a file the step appends to, as the partition file and
            positions are, the record keeps the file's length before the
            step.</li>
            <li><span class="review">&#9873;</span> For a file the step rewrites in place, the record keeps a copy
            of the old file, taken before the first change.</li>
            <li><span class="review">&#9873;</span> When the step has written and synced all its files, it
            appends a record saying it finished.</li>
            <li><span class="review">&#9873;</span> On opening a bundle, each step with no finished record is
            undone. Appended files are cut back to their old length, and
            rewritten files are copied back.</li>
            <li><span class="review">&#9873;</span> An undone partition build or merge is then run again, as it
            would be if it had never started. A log line says what was
            undone.</li>
            <li><span class="review">&#9873;</span> Small files that are written whole, such as next_partition.txt
            and max_tier, are instead written to a new name and renamed.</li>
            <li><span class="review">&#9873;</span> Once every step in the log has finished, the log and its
            copies of old files are removed.</li>
            <li><span class="review">&#9873;</span> UndoLog does this. A partition build, a partition's dictionary
            merge with the partition count, and each tier merge are steps.</li>
            <li><span class="review">&#9873;</span> A step whose process still runs is never undone, so a search
            opening the index during a crawl leaves a merge alone.</li>
            <li><span class="review">&#9873;</span> A tier merge writes under a name searches never read, renames
            it in, and commits. A later stop then completes the removal.</li>
            <li><span class="review">&#9873;</span> A dictionary read now also reads each tier's B slot, and finds
            the highest tier again when the one it held is gone.</li>
        </ol></li>
    <li><span class="review">&#9873;</span> <strong>The recovery is
        shown by stopping a crawl on purpose.</strong> A synthetic crawl
        is driven and the queue server is
stopped at chosen points in the write path. The index is read back
afterward and must come back clean every time.
        <ol>
            <li><span class="review">&#9873;</span> UndoLogTest kills a build, a dictionary merge and a tier
            merge at 0, 2 and 10 milliseconds into each step.</li>
            <li><span class="review">&#9873;</span> Each time the bundle opens whole, and holds every merged
            partition exactly once after the step runs again.</li>
            <li><span class="review">&#9873;</span> Each stop case now removes its bundle when done, so the suite
            leaves no folder under tests/test_files.</li>
        </ol></li>
    <li><span class="review">&#9873;</span> <strong>Two processes never
build the same partition at once.</strong> A claim now holds its maker's
process id and stands while that process runs.
        <ol>
            <li><span class="review">&#9873;</span> On a synthetic crawl a partition of about 147,000 documents
            took 12 to 13 minutes to build, past the 10 minute timeout.</li>
            <li><span class="review">&#9873;</span> So claims and build slots looked abandoned mid build. Six of
            seven fetchers built at once, two of them partition 32.</li>
            <li><span class="review">&#9873;</span> IndexDocumentBundle::claimIsAbandoned now decides, for the
            merge claim, the build slots and the partition claims.</li>
            <li><span class="review">&#9873;</span> Builds slowed about 29% with combined terms, not ticks. Last
            rows are now plain arrays until saved, 13% faster, same files.</li>
            <li><span class="review">&#9873;</span> A word's term ids are made once per word and page combination
            and reused, about 6% faster, with the same files.</li>
            <li><span class="review">&#9873;</span> A term's postings are appended in place and counted once at
            the end. Merging rows had cost time in the square of a row.</li>
        </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>Crawls survive stops, power cuts and races between
    processes.</strong> A partition's files count as built only when they
    hold its documents.
    <ol>
        <li><span class="review">&#9873;</span> The undo log syncs each write and keeps a checkpoint, so a power
        cut rolls a file back to it.</li>
        <li><span class="review">&#9873;</span> A crawl stopped before saving a document still stops. A
        partition read as empty is not built as empty.</li>
        <li><span class="review">&#9873;</span> Each queue server process reads a start message once, under a
        lock.</li>
        <li><span class="review">&#9873;</span> A partition built while it was still being filled is built
        again before it is merged. A built_from file holds the hash of the
        document index the build read. In a local crawl with 100 documents
        a partition, partition 0 had gone into the dictionary with 1 of its
        99 documents.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>The news feed index stays whole when MediaUpdater is
    stopped.</strong> A feed partition's files are saved as one undo log
    step, and every item is indexed. A saved second seen-item filter no
    longer stops the bundle opening. A feed partition's terms are saved in
    order, so the dictionary finds them; ArcTool rebuild IndexDataFeed mends
    a bundle saved before. An update that adds a partition still saves the
    seen-item filters, so the next update does not add its items
    again.</li>
    <li><span class="review">&#9873;</span>
    <strong>A partition not added to the dictionary is not recorded as
    merged.</strong>
    On yioop.com, info:https://www.sjsu.edu/ and
    info:https://www.sjsu.edu/robots.txt found no dictionary row. The
    site was a seed, and robots.txt pages are indexed with their meta
    words.
    <ol>
    <li><span class="review">&#9873;</span> Here, the query's key for each url is the key a build files the
    url's info: word under. On five local indexes, info: queries found
    their page.</li>
    <li><span class="review">&#9873;</span> updateDictionary ignored a false return from
    addPartitionPostingsDictionary, which comes when the partition's
    postings or counts file is missing. It recorded the partition as
    merged anyway, so none of its words could be found again.</li>
    <li><span class="review">&#9873;</span> The partition now stays the next to merge, and the log says so.
    Whether this is what dropped the San Jose State University rows on
    yioop.com is not shown.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A rebuild says where it can safely continue from.</strong>
    A rebuild printed every DictionaryUpdater log line and a progress line
    every two seconds.
    It now keeps those logs in files and prints its state only when it
    changes, naming the safe partition to continue from.
    The new rebuild-status command prints the same for a stopped
    rebuild.</li>
</ol></li>
<li><strong>Other work, grouped by what it touched.</strong> Each is a
deliverable of its own that came up during the arc. They are grouped by
the part of Yioop they concern rather than listed one by one. <ol>
    <li><span class="check">&#10003;</span>
    <strong>The unit tests pass with no warnings or log lines.</strong>
    Each case cleans up after itself and needs nothing installed.
    <ol>
        <li><span class="review">&#9873;</span> cullByDomainFilterTestCase makes its own filter instead of
        needing a top ten million domain file.</li>
        <li><span class="review">&#9873;</span> WebSiteTest puts back the request globals, so the runner's
        NO_LOGGING setting survives. WebSite reads a missing SCRIPT_NAME as
        the site's root.</li>
        <li><span class="review">&#9873;</span> PartitionDocumentBundleTest removes the folders it makes, and
        LSMTree checks for a renamed file instead of reading it.</li>
        <li><span class="review">&#9873;</span> TurnServer.php leaves NO_LOGGING alone when a test only reads
        its class, so later tests no longer print indexer log lines.</li>
        <li><span class="review">&#9873;</span> yioop_error_handler leaves an error silenced with @ to PHP, so
        it is no longer printed.</li>
        <li><span class="review">&#9873;</span> A mergeNextPartition case covers a partition not yet built, a
        fresh and an abandoned merge claim, and the merged dictionary.</li>
    </ol></li>
    <li><span class="check">&#10003;</span>
    <strong>The mail library no longer translates.</strong> Backends throw
    a plain Exception starting with an ImapClient or MailBackend ERROR_
    constant; MailComponent::mailErrorText matches the constant.
    <ol>
        <li><span class="review">&#9873;</span> MailBackendException is removed. MailCloneJob asks
        ImapMailBackend::isTransientError whether to try again.</li>
        <li><span class="review">&#9873;</span> Every string MailComponent uses begins with mail_component_,
        including eight that had never been added to any locale.</li>
        <li><span class="review">&#9873;</span> The api_unsubscribe strings are unsubscribe_view strings; the
        other api_ strings, used nowhere, are removed.</li>
    </ol></li>
    <li><span class="check">&#10003;</span>
    <strong>Serve with several web server processes.</strong> The web
    server keeps every connection, and runs requests in worker processes
    when Web Server
    Workers is above zero.
    <ol>
        <li><span class="review">&#9873;</span> WebSite starts the workers with proc_open once it runs as its
        run-as user, and links them by loopback sockets.</li>
        <li><span class="review">&#9873;</span> Each request goes to the least busy worker with its globals and
        session records, and the reply comes back with any changed
        sessions.</li>
        <li><span class="review">&#9873;</span> Requests whose replies stream, such as resources and byte
        ranges, stay in the web server, as WebSocket connections do.</li>
        <li><span class="review">&#9873;</span> Workers share the file write journal, and exit when the web
        server stops. With one process nothing changes.</li>
        <li><span class="review">&#9873;</span> A WebSiteTest case starts a site with two workers and checks
        replies, sessions, a streamed reply and worker exit.</li>
        <li><span class="review">&#9873;</span> The server's select also watches the worker links, so a
        reply wakes it. Yioop keeps /wd/ paths in the server, as
        resources.</li>
        <li><span class="review">&#9873;</span> Each access line for a relayed request ends with the number of
        the worker that ran it.</li>
        <li><span class="review">&#9873;</span> A restart no longer signs users out. The supervisor hands the new
        server the sessions saved in the last five minutes.</li>
        <li><span class="review">&#9873;</span> WebSite::loadSavedSessions takes them back; before, nothing read
        them, and the restart page's first status poll got a new
        session.</li>
        <li><span class="review">&#9873;</span> A worker takes the front's server settings with each request.
        Before, a request it made to its own site, as Manage Machines does
        to start a queue server, went to the error page.</li>
        <li><span class="review">&#9873;</span> A worker link is accepted only after the worker shows a hash of a
        secret the front gave it. The front answers with its own hash.</li>
        <li><span class="review">&#9873;</span> Connections to the worker link are read side by side, so silent
        ones cannot hold up a worker's start. Each is closed after half a
        second.</li>
        <li><span class="review">&#9873;</span> Refused connections are noted in the worker log, five a minute
        and then a count. A worker that fails to start waits 30 seconds.</li>
        <li><span class="review">&#9873;</span> A worker that stops is replaced on the next request. The
        worker log records each start, failure to start, and stop.</li>
        <li><span class="review">&#9873;</span> A worker that stops is replaced at the next request. Each start,
        failed start and stop is written to the worker log.</li>
    </ol></li>
    <li><span class="check">&#10003;</span>
    <strong>WebSite can share a file write journal between
    processes.</strong> Each process forgets a cached file another has
    written, so it neither serves the old copy nor skips a needed write.
    <ol>
        <li><span class="review">&#9873;</span> Two server objects on one file without it: a stale read and a
        lost write. With it: neither.</li>
    </ol></li>
    <li><span class="check">&#10003;</span>
    <strong>Yioop no longer uses pcntl, except to start programs.</strong>
    Its signal and alarm uses become tick functions, with ticks declared
    every 1,000 statements in the files they must reach.
    <ol>
        <li><span class="review">&#9873;</span> runWithTimeLimit stops a document's slow parsing through ticks
        in PhraseParser and the 25 tokenizers. It adds 6 percent to phrase
        extraction.</li>
        <li><span class="review">&#9873;</span> MediaUpdater checks for a stop once a second, even inside a
        job. MailSite's watchdog reports a stuck event loop once.</li>
        <li><span class="review">&#9873;</span> ArcTool's rebuild and remerge run in a process of their own,
        so closing the terminal does not stop them.</li>
        <li><span class="review">&#9873;</span> MailServer's signal log is dropped. CrawlDaemon still forks to
        start a program, since otherwise it keeps the web server's
        port.</li>
    </ol></li>
    <li><span class="check">&#10003;</span>
    <strong>The Web Server legend has a Log link.</strong> It shows beside
    the help button only when Yioop is its own web server, and reads Log
    in each locale.</li>
    <li><span class="check">&#10003;</span>
    <strong>The web server reports how busy it is, and its log opens from
    the Web Server settings.</strong> Each memory sample line now gives the
    server process's processor use since the last line. A value near 100
    means one core is full.
    <ol>
        <li><span class="review">&#9873;</span> Sampled directly, the figure read 0.0 after two idle seconds
        and 99.9 after two seconds of busy work.</li>
        <li><span class="review">&#9873;</span> The Web Server fieldset links to index.log through the
        machines log view. The link returned the file's lines.</li>
        <li><span class="review">&#9873;</span> The link has its own string,
        servers_element_web_server_log_view, in all 25 locales.</li>
    </ol></li>
    <li><span class="check">&#10003;</span>
    <strong>Form labels draw their markup, and the keyword captcha
    previews.</strong> Bold in a text field's label showed its quote
    marks in both parsers. The editor's preview did not draw a keyword
    captcha at all.
    <ol>
        <li><span class="review">&#9873;</span> Both parsers now draw markup in the label of text fields,
        text areas, checkboxes, radio buttons and the keyword
        captcha.</li>
        <li><span class="review">&#9873;</span> The preview draws the captcha's label and box without
        its answer. New cases fail on the old code and pass now.</li>
    </ol></li>
    <li><span class="check">&#10003;</span>
    <strong>A large file downloaded from the web server does not stall
other requests and arrives whole.</strong> One reader downloading a large
video held up every other request for the length of the download, and a
three hundred megabyte video ended the web server process. A whole file
request is now handed to the event loop, which sends the file a block at
a time between other requests, and a ten megabyte pdf that had come back
as headers with no body now arrives complete over both HTTP/1.1 and
HTTP/2. Measured: a page answered in twenty seconds during a two hundred
megabyte download before and under five milliseconds after.</li>
    <li><span class="check">&#10003;</span>
    <strong>A document thumb on a wiki page shows a picture, not a
frame.</strong> A resource-thumb mark for a pdf drew an iframe of the
document where no thumbnail was there. The mark is now always a link with
a picture, the thumbnail is made from the document on the first read that
asks for it, and an image thumb draws the thumbnail address rather than
the full picture.</li>
    <li><span class="check">&#10003;</span>
    <strong>A reader who may not edit, or see a private history or
source, is answered 403.</strong> A stranger's arg=edit with a stale
token was quietly downgraded to a page read and served 200; a private
page's source was served at 200 too. The group controller now answers 403
for a writing arg on a request whose token is bad; the wiki component
refuses edit, history and source by the group's Access setting, where only
Read Write Wiki lets a member edit. Walked as a stranger, a member of a
Read group, a member of a Read Write Wiki group, and root.</li>
    <li><span class="check">&#10003;</span>
    <strong>The server log names the signed in user, marks each reply's
speed, and names a timer that held the loop.</strong> Each finished-reply
line carries the username after the address, and its time ends in -slow,
-slower or -slowest by band so one grep finds a band. A timer callback
that runs past a tenth of a second writes a line with its length and the
file and line it was set at, since a request arriving during one waits
the whole of it and two cheap 403s that took 2.6 seconds in the same
second had waited on one such callback.</li>
    <li><span class="check">&#10003;</span>
    <strong>Every request's completion line in the server log says how
long it took.</strong> The line that records a reply's status and size
ran only for an immediate HTTP/1 reply, so no HTTP/2 page and no deferred
request, which is every search, ever logged one. It now runs once on every
reply path, a deferred reply logs after its own environment is applied
with a file reply's true size, a redirect logs as 302, and the line ends
with the milliseconds since the request arrived. Measured here: a search
over h2 22ms, the same page over h1 3ms, a ten megabyte download 2ms to
hand off.</li>
    <li><span class="check">&#10003;</span>
    <strong>A web server that has given up root can still start, stop and
restart what needs a privileged port.</strong> The server binds 80 and
443 as root, then becomes the user WWW_USER names. After that it could not
start the mail server, nor a replacement for itself. A restart could
leave the site down with nothing in any log. The process that
started the server stays as root and starts what is missing.
    <ol>
        <li><span class="check">&#10003;</span>
        <strong>One process keeps root and starts what is
missing.</strong> It is index.php in a supervisor mode, begun by the
start once the server is on its way. It binds nothing and listens to
nobody, so it takes no request and needs no channel. Every few seconds
it compares what should be running against what is. What should be
running it reads where that was already written: the rows Manage
Machines keeps and the address the server records. It adds no file of
its own. It lives through a restart and ends on a stop.</li>
        <li><span class="check">&#10003;</span>
        <strong>A stop applies only to services that were there when it
was asked for.</strong> The old server and the launcher acted on the same
services with no ordering. A late stop could then end a mail server
already brought back. A stop now carries the moment it was asked for
and leaves a younger service alone. A lock file carries the time its
service began, so the two can be told apart.</li>
        <li><span class="check">&#10003;</span>
        <strong>Every silent death now writes down its reason.</strong>
The mail server's bind results, and a failure before it can listen, go to
its log rather than to output that is thrown away. Every place that
removes a service's lock names the process, the reason and the calls that
led there. A service says how long it ran when it stops.</li>
        <li><span class="check">&#10003;</span>
        <strong>A restart from the command line reports for a reader who
barely knows sudo.</strong> It ends with which services are running again and
which are still starting. For one that did not come back it gives the
command to check and the path of its log. It waits for nothing. A restart
from the Servers screen brings the site and its services back the same
way.</li>
    </ol></li>
    <li><span class="check">&#10003;</span>
    <strong>A scraper that rotates its address is turned away by what it
    asks for.</strong> A count kept per address never slows a scraper whose
    every request comes from a new address. The count is kept on the thing
    it wants instead.
    <ol>
        <li><span class="check">&#10003;</span>
        <strong>A wiki page whose history or source draws many refusals is
        refused before any work.</strong> Twenty refusals in five minutes
        make a page hot for an hour. A reader with no session then gets a
        bare 403. The web server of index.php checks before the session.
        Apache checks a marker file under work_directory/hot.</li>
        <li><span class="check">&#10003;</span>
        <strong>The cached copies of an index are guarded the same
        way.</strong> In a five minute log from yioop.com, 1,058 cache
        requests came from 1,040 addresses. 1,046 were misses, each walking
        every index, at 878 ms on average. A miss by a reader with no
        session is now a strike against the index the request names. The
        busiest single page drew 19 requests, so the key is the index, not
        the page. The walk ran on a server begun with secure-start. The
        twentieth miss made the index hot. Then came 403 without a cookie,
        302 with one, and 200 for an icon.</li>
        <li><span class="check">&#10003;</span>
        <strong>With Cache unchecked, a cached copy answers 404.</strong>
        The Page Options Cache box only hid the link, and the copies were
        still served. The cache activity now answers 404 when it is off. A
        search result's icon names a field and is still served. Checked
        after a secure-restart with Cache off, then ended with
        secure-stop.</li>
        <li><span class="check">&#10003;</span>
        <strong>A cache miss no longer prints stray words before the
        page.</strong> ParallelModel echoed "very null" ahead of the body on
        every miss. The echo is removed.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A search for a site's name finds the site first.</strong>
    Relevance is one divergence score over weighted fields, with url signals
    as pseudo words.
    <ol>
        <li><span class="review">&#9873;</span> The shortest url of pages with the same content keeps their
        words.</li>
        <li><span class="review">&#9873;</span> A site's name typed as words also looks up the joined host
        word, and keeps the words together. A host or path holding the
        joined words gets 0.9 of its keyword bonus.</li>
        <li><span class="review">&#9873;</span> A union of query parts is term at a time with accumulator
        pruning, after Lester et al. A part with a joined word may open ten
        accumulators, so tik tok finds tiktok.com.</li>
        <li><span class="review">&#9873;</span> A search no longer runs the index's undo, which held a worker
        on every file of the checkpoint.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>Searches use the index as it is now, and the web server's
    workers keep up under load.</strong> A worker never holds a request for
    good, and the crawl keeps a worker of its own.
    <ol>
        <li><span class="review">&#9873;</span> A web server process reopens an index another process changed.
        It checks the index folder at most once a second.</li>
        <li><span class="review">&#9873;</span> A new default index and new debug settings apply without a
        restart.</li>
        <li><span class="review">&#9873;</span> A search over queue servers stops once every document is asked
        for, and the offset a server takes is capped.</li>
        <li><span class="review">&#9873;</span> A worker treats the name server's host as its own. A search then
        runs its queue server request inside the worker.</li>
        <li><span class="review">&#9873;</span> The front ends a worker whose GET has run five seconds from when
        it took it up. A silent worker is ended after thirty.</li>
        <li><span class="review">&#9873;</span> The crawl's requests have the last worker, which takes other
        requests while it is idle.</li>
        <li><span class="review">&#9873;</span> A request for a hot target with no session is refused by the
        front. Strikes are counted under the hot record's lock.</li>
        <li><span class="review">&#9873;</span> Requests reach a worker from a list written without waiting. A
        depth-one git clone through two workers fell from 51 to 5.5
        seconds.</li>
        <li><span class="review">&#9873;</span> A full git clone of many versions of large locale files builds
        its pack ten times faster. A delta copies the shared start and end
        of two versions whole and compares the rest in pieces. A chain runs
        up to 50 deltas.</li>
        <li><span class="review">&#9873;</span> A delta is built only on an object of its own type. A commit
        built on a tag reached git as a tag, which stopped the clone with
        "invalid tag".</li>
        <li><span class="review">&#9873;</span> Workers stop through CrawlDaemon::stopProcess on Linux, macOS
        and Windows, and log the calls they were in.</li>
        <li><span class="review">&#9873;</span> A fetcher sends what it found after its first hundred
        downloads. The downloads between sends then double, up to
        1,000.</li>
        <li><span class="review">&#9873;</span> The summarizer and term frequencies count a page's words once.
        Image processors shrink a sample of a large picture.</li>
        <li><span class="review">&#9873;</span> Partition builds and the crawl queue sync less. signalShares
        reads only next_partition.txt.</li>
        <li><span class="review">&#9873;</span> A secure-terminal also starts the root supervisor, which stops
        when the terminal does.</li>
        <li><span class="review">&#9873;</span> A company domain's root page counts the host and domain boost
        words, so www.yahoo.com ranks above sports.yahoo.com.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>Tests start helper processes the same way on every
    system.</strong> UnitTest::startScript and CrawlDaemon::endProcess
    replace proc_open in the test cases. A case runs a helper method of
    its own class with UnitTest::startMethod, so tests/test_files holds no
    PHP script. A profile written to another folder no longer changes the running
    process's settings, so the full run completes. CodeTool prints the
    error and test that end a run. The undo cases wait for a killed
    process to end, then open the bundle for writing, which runs the
    undo.</li>
    <li><span class="review">&#9873;</span>
    <strong>The web server keeps its workers running.</strong> A request
    starts any stopped worker again, and the front sleeps on a worker's
    link while it waits.
    <ol>
        <li><span class="review">&#9873;</span> The supervisor, the workers and the mail sender start as php
        through CrawlDaemon. Workers hold none of the front's listening
        sockets.</li>
        <li><span class="review">&#9873;</span> A stop of the whole site also stops what Manage Machines
        started. The supervisor names any process holding a port it
        needs.</li>
        <li><span class="review">&#9873;</span> A stopped worker is logged with the request it was running, and
        the memory report names each worker's request.</li>
        <li><span class="review">&#9873;</span> A request waiting behind a worker that is ended goes to another
        worker. Only the running request gets a 502.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>Mail one server cannot take yet is tried again for days, and
    reaches a domain's hosted server.</strong> Other recipients still get
    it. A domain with no MX record is tried at its Microsoft 365 or Google
    Workspace server.</li>
    <li><span class="review">&#9873;</span>
    <strong>Yioop runs under Apache and PHP-FPM without notices or
    clashes.</strong> A request no longer writes Profile.php, and one
    request at a time runs an upgrade. Web logging goes to index.log, and
    style sheets keep their own headers.</li>
    <li><span class="review">&#9873;</span>
    <strong>The private database always has a name, and a person using
    the site is not refused.</strong> A signed-in person with an old token
    is sent back with a fresh one.</li>
    <li><span class="review">&#9873;</span>
    <strong>Outside programs are used once installed, with no setting
    needed.</strong> Yioop asks the system where ffmpeg or tesseract is.
    It uses where on Windows and command -v elsewhere, and skips the work
    when neither finds the program.
    <ol>
        <li><span class="review">&#9873;</span> A setting such as FFMPEG in LocalConfig.php still wins. A
        setting of false turns the work off.</li>
        <li><span class="review">&#9873;</span> WHISPER and the three transcription settings, which no code
        read, are removed.</li>
        <li><span class="review">&#9873;</span> Temporary files for git packs, zip downloads, certificate
        requests and node tests go under C\TEMP_DIR.</li>
        <li><span class="review">&#9873;</span> On Windows the memory size comes from PowerShell's
        Get-CimInstance, since wmic is not installed on newer
        versions.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A saved form no longer sends the browser to 0.0.0.0.</strong>
    A redirect to the site's base gives only its path, so the browser keeps
    its host. baseUrl does not use 0.0.0.0 or :: as a host name.</li>
    <li><span class="review">&#9873;</span>
    <strong>A program ended with Ctrl-C on Windows no longer blocks its next
    start.</strong> isProcessAlive looks the id up in what tasklist lists,
    read at most once a second. Before, Windows had no check, so every lock
    looked held.</li>
    <li><span class="review">&#9873;</span>
    <strong>A connection that closes before its TLS handshake no longer
    keeps the web server busy.</strong> The front's loop spun on such a
    socket for up to 30 seconds. The [website-front] line now counts them
    as closed_before_first_byte.</li>
    <li><span class="review">&#9873;</span>
    <strong>Each bundle of packed objects is cut by its own
    dictionary.</strong> A small bundle right after another was cut by the
    earlier one's numbers. A magazine's page tree then came apart, and a
    later page was drawn as its cover.</li>
    <li><span class="review">&#9873;</span>
    <strong>A request over HTTP/3 keeps the server's script name.</strong>
    Each HTTP/3 request named an empty script name, so Yioop sent every
    path to the error page. The web server now also logs a client's
    stream resets and closing errors on HTTP/2.</li>
    <li><span class="review">&#9873;</span>
    <strong>An HTTP/2 frame of an unknown type is skipped.</strong> A
    PRIORITY_UPDATE frame ended the whole connection with an exception, and
    every response still owed on it was lost. Such a frame is now read by
    its length and passed over, as RFC 9113 asks.</li>
    <li><span class="review">&#9873;</span>
    <strong>A remade thumbnail is served whole.</strong> The web server
    kept the old picture in its cache of small files. It sent those bytes
    under the new picture's length until it was restarted. The remade
    picture is now written through the web server's journal, and the
    length sent is that of the bytes sent.</li>
    <li><span class="review">&#9873;</span>
    <strong>An exact match of a typed word doubles its share of a
    result's relevance.</strong> The index holds stems, so anime also finds
    animal. Once a result's title and summary are in hand, each typed word
    found in them multiplies its share by EXACT_MATCH_FACTOR.</li>
    <li><span class="review">&#9873;</span>
    <strong>The guards from the tenth-round review are in the process file
    and its scripts.</strong> A withdraw now counts the round, and a cut
    prints the git add line of each new file. The word check warns about
    an acronym not spelled out and a missing To check line.</li>
    <li><span class="review">&#9873;</span>
    <strong>Manage Users shows when each user joined.</strong> The list
    of users gains a Join Date column, and the edit form a Join Date row
    that cannot be changed. Both read the account's CREATION_TIME. The new
    locale key takes each locale's wording of the group join date.</li>
    <li><span class="review">&#9873;</span>
    <strong>Errors in yioop.com's message log are fixed.</strong> The log
    held a Postgres error from the wiki page list and thousands of false
    checkpoint notes. It also showed keys reported absent during tier
    merges.
    <ol>
    <li><span class="review">&#9873;</span> WikiModel::getPageList no longer uses SELECT DISTINCT, which
    Postgres refuses with ORDER BY LOWER(GP.TITLE). A category is matched
    with IN, so each page still comes back once.</li>
    <li><span class="review">&#9873;</span> A bundle's undo checkpoint drops the tier files a dictionary merge
    removes, and the queue server's removed partition marker. They were
    reported as lost each time the bundle opened.</li>
    <li><span class="review">&#9873;</span> LSMTree::emptyTier moves a tier aside in one rename before deleting
    it. A lookup during the deletion then sees the tier gone.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A web server worker writes its log lines to index.log.</strong>
    A worker never named its log, so lines it wrote during requests went to
    message.log.</li>
    <li><span class="review">&#9873;</span>
    <strong>A word's walk no longer skips the last posting of a
    partition.</strong> In a one-word query, the field key's score was lost
    for each partition's last page with the word in host, title or path.
    That page then ranked as if the word were only in its body.
    <ol>
    <li><span class="review">&#9873;</span> WordIterator::advanceSeenDocs counted the postings left from
    next_offset, which findDocsWithWord had already moved. It now counts
    from current_offset, where the block starts.</li>
    <li><span class="review">&#9873;</span> getPostingsSliceResults took a descending block of at least
    results_per_block. It now takes at most the postings left, so a walk
    from last to first reads no page twice.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A recrawled page is counted once in a word's union of
    keys.</strong> A page crawled twice has a posting for each copy in a
    word's own key and in its field key.
    <ol>
    <li><span class="review">&#9873;</span> WordIterator gives both postings the latest copy's key, and
    UnionIterator added the copies' rank and relevance. The page then
    scored about twice its relevance.</li>
    <li><span class="review">&#9873;</span> UnionIterator now keeps the higher scoring copy when two documents
    share a key. A link's score is still added to the page it points
    to.</li>
    <li><span class="review">&#9873;</span> In a scratch index with a song page crawled twice, the song page's
    relevance for yahoo went from 24.07 to 12.03. yahoo.com, at 12.44, now
    ranks first.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A one-letter query is looked up as typed.</strong>
    guessSemantics rewrote a query of one letter, such as x, as letter
    x|x.
    <ol>
    <li><span class="review">&#9873;</span> The added part matched pages holding both letter and x, so
    dictionary pages about the letter x led the results for x.</li>
    <li><span class="review">&#9873;</span> The rewrite is removed. In a scratch index, x.com went from second
    to first for x.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>Results much like one above them move down.</strong>
    PhraseModel::diversify reorders the top three pages of results by
    maximal marginal relevance, after Carbonell and Goldstein (1998).
    <ol>
    <li><span class="review">&#9873;</span> Each next result has the highest lambda times its share of the top
    score, less a penalty. The penalty is 1 - lambda times its similarity
    to a result placed.</li>
    <li><span class="review">&#9873;</span> Similarity is the cosine of the title word sets, without the
    query's words. Shared words are counted once from per-word lists of
    results. Nothing in the index changes.</li>
    <li><span class="review">&#9873;</span> Every page inside the window is cut from one reordered list, since
    getSummariesByHash fetches the same summaries for each of them.</li>
    <li><span class="review">&#9873;</span> Lambda is the profile value MMR_LAMBDA, 0.7 by default, set on Page
    Options. 1 keeps the order by score.</li>
    <li><span class="review">&#9873;</span> On yioop.com's results for x, x.com went from 5th to 3rd. The second
    and third dictionary.com pages went to 8th and 9th.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>Searches no longer stop on a missing MMR_LAMBDA.</strong>
    MMR_LAMBDA was a profile setting defined only for a site with no
    profile.
    <ol>
    <li><span class="review">&#9873;</span> On a site whose profile predates it, every search threw "Profile
    constant MMR_LAMBDA not defined".</li>
    <li><span class="review">&#9873;</span> The weight is now MMR_RELEVANCE_WEIGHT, a plain Config.php setting.
    The profile field, Page Options field and locale string are gone.</li>
    <li><span class="review">&#9873;</span> The new name keeps a leftover MMR_LAMBDA line in a saved profile
    from being read.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A term in the active partition is found by binary search.</strong>
    getWordInfo reads the postings of the partition still being filled as one
    string, and searched the whole string for each term.
    <ol>
    <li><span class="review">&#9873;</span> A query looks up each page it scores this way, to find the page's
    latest copy. Each lookup read the whole string.</li>
    <li><span class="review">&#9873;</span> PackedTableTools::findRowInSortedTableString does a binary search on
    byte positions. The rows are saved sorted by term.</li>
    <li><span class="review">&#9873;</span> On a scratch index with 39,000 pages in its active partition, y
    combinator went from 5.7 seconds to 0.4 seconds.</li>
    <li><span class="review">&#9873;</span> Of 2,242 searches from the site's logs replayed on that index, 100
    timed out at 5 seconds before and 2 after.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A search with a negated word gives every page without it, and
    quickly.</strong>
    Two searches from the site's logs with -eliminate -abuses took over 5
    seconds on the scratch index.
    <ol>
    <li><span class="review">&#9873;</span> DocIterator read a partition's whole document map again on every
    reset. One such search made 52 walks, each reading it three times.</li>
    <li><span class="review">&#9873;</span> A walk now reads rows through the bundle's doc_map_tools, which
    keeps a map and its row positions for every walk of a process.</li>
    <li><span class="review">&#9873;</span> A place in the walk is a document map row number, the DOC_MAP_INDEX
    a WordIterator gives for the same page.</li>
    <li><span class="review">&#9873;</span> A walk stopped before each partition's last row, so ten pages in
    partitions of four gave seven. Walks in both directions now give ten.</li>
    <li><span class="review">&#9873;</span> NegationIterator stopped once the negated word had no page left.
    Without rare, on one page of ten, it gave two pages; it now gives
    nine.</li>
    <li><span class="review">&#9873;</span> The slower search went from 5.2 seconds to 0.23 seconds. In a
    replay of 2,242 logged searches, none timed out, against one before.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>Search suggestions sit under the search box and list the most
    frequent completions.</strong>
    On a page of results, the dropdown was fixed 50 pixels from the top and
    began 9 pixels inside the search box.
    <ol>
    <li><span class="review">&#9873;</span> placeDropdown sets its top, left and width from the box when the
    page fixes it on the screen. The landing page is unchanged.</li>
    <li><span class="review">&#9873;</span> It lists the six completions of highest frequency, after matching
    history. For la it made 382 rows before.</li>
    <li><span class="review">&#9873;</span> For en-US, completions under the capitalized prefix are found too,
    so eng gives England.</li>
    <li><span class="review">&#9873;</span> spellCheck read its history count from a variable out of scope, so a
    query sent six times still got a correction.</li>
    <li><span class="review">&#9873;</span> suggest.js's docblocks are brought up to standard.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>yioopbar.xml searches yioop.com over https.</strong>
    Its search URL template began http://www.yioop.com.</li>
    <li><span class="review">&#9873;</span>
    <strong>A search reads each dictionary data file once.</strong>
    Each page a search scores looks up its newest copy under an info: key.
    That walk read a whole data file in every tier, though most pages share
    the files of the small tiers.
    <ol>
    <li><span class="review">&#9873;</span> Tier::get keeps a data file's entries in the cache that holds the
    index lists, checked against the file on each read.</li>
    <li><span class="review">&#9873;</span> It finds the key by binary search over the sorted entries. The 8
    million entries of two scratch indexes held no key twice in a
    file.</li>
    <li><span class="review">&#9873;</span> For mary pollett over 300 partitions, a search read 927 data files
    totalling 30.3 megabytes, but only 116 different ones. It now
    reads 121.</li>
    <li><span class="review">&#9873;</span> lookupLatestVersionPage asks the dictionary itself rather than
    making a model, which connected to both databases for each page.</li>
    <li><span class="review">&#9873;</span> IndexManager.php's docblocks are brought up to standard.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>QueryTool times three parts of opening a partition for scoring
    apart.</strong>
    yioop.com's mary pollett run spent 1.06 s opening partitions for
    scoring. Positions file opens, document map loads and signal share
    reads now print as lines of their own.</li>
    <li><span class="review">&#9873;</span>
    <strong>A search scores the newest copy of a page wherever it
    is.</strong>
    A page's newest copy was looked for only in the active partition, so a
    newer copy merged into the dictionary was never found.
    <ol>
    <li><span class="review">&#9873;</span> A lookup read the postings of the newest row only when that row was
    the active partition's. A dictionary row holds only where its postings
    are.</li>
    <li><span class="review">&#9873;</span> IndexDocumentBundle::newestRow asks the active partition first, then
    walks the dictionary from its lowest, newest tier and stops at the first
    row.</li>
    <li><span class="review">&#9873;</span> LSMTree::get takes $newest_first for this walk.</li>
    <li><span class="review">&#9873;</span> A newer copy without the query's word drops the page. One with it
    is scored from its own partition's document map and positions.</li>
    <li><span class="review">&#9873;</span> checkTermExists tested bit $hash % 7 where storeTerms set
    $hash & 7, so every newer copy found was taken to lack the
    word.</li>
    <li><span class="review">&#9873;</span> The terms filter holds plain words, so it is used only for a plain
    word's key.</li>
    <li><span class="review">&#9873;</span> Lookups found are remembered, so mary pollett on a 300 partition
    test index made 200 lookups, not 601.</li>
    <li><span class="review">&#9873;</span> A word iterator asks for the word's whole count only with its
    first batch of rows, the only batch whose totals are read.</li>
    <li><span class="review">&#9873;</span> The Crawl Speed History strings are translated into every
    locale.</li>
    <li><span class="review">&#9873;</span> IndexDocumentBundle.php's docblocks are brought up to
    standard.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A search reads one document map entry without reading the whole
    map.</strong>
    Scoring a posting loaded its partition's whole document map, about 3 MB
    on yioop.com, to read one entry.
    <ol>
    <li><span class="review">&#9873;</span> Each partition gets a doc_map_marks file: the map's size, its entry
    count, and where every 32nd entry starts.</li>
    <li><span class="review">&#9873;</span> IndexDocumentBundle::docMapEntry seeks to the mark before an entry
    and reads 4 KB at a time past at most 31 entries.</li>
    <li><span class="review">&#9873;</span> A build writes the file; an older partition gets it on its first
    search, and again if the map's size changed.</li>
    <li><span class="review">&#9873;</span> For 30 entries of a 3.2 MB map, the whole map read took 0.064 s and
    the marked reads 0.0011 s.</li>
    <li><span class="review">&#9873;</span> DOC_MAP_MARK_EVERY and DOC_MAP_READ are new in Config.php.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A search looks up the newest copy only for pages that reach the
    results.</strong>
    yioop.com's mary pollett run made 2,277 latest copy lookups, 2.224 s,
    for 197 result rows.
    <ol>
    <li><span class="review">&#9873;</span> WordIterator no longer looks up a newest copy for each posting; it
    marks a text page's posting with TERM_KEY, the word found.</li>
    <li><span class="review">&#9873;</span> GroupIterator::newestCopy checks each copy of a grouped page once,
    with one lookup.</li>
    <li><span class="review">&#9873;</span> A newest copy without the word drops the copy; one with it is shown,
    keeping the scored copy's score.</li>
    <li><span class="review">&#9873;</span> IndexManager::termPostingInPartition, moved from WordIterator, checks
    the newest copy for the word.</li>
    <li><span class="review">&#9873;</span> Only the top LATEST_COPY_FRACTION, a tenth, of a block's pages by
    doc rank are checked, since a newer copy does not change doc rank. On
    the test index, mary pollett made 20 lookups, not 200.</li>
    <li><span class="review">&#9873;</span> QueryTool also times making the query iterators, scoring and
    grouping the results, and reading their summaries.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A search keeps each document map and positions file open while
    it scores a partition.</strong>
    docMapEntry opened and closed a partition's document map for every
    entry; yioop.com's mary pollett read 2,919 entries.
    <ol>
    <li><span class="review">&#9873;</span> IndexDocumentBundle::$doc_map_handles keeps a handle per map, with
    the size it had, and reopens a map saved again since.</li>
    <li><span class="review">&#9873;</span> 30 entries of a 3.2 MB map took 0.0007 s, against 0.0011 s.</li>
    <li><span class="review">&#9873;</span> WordIterator keeps the scored partition's positions file and
    document count across calls, and reads the signal shares once a
    query.</li>
    <li><span class="review">&#9873;</span> On the test index, mary pollett read the signal shares 4 times, not
    601, in 0.004 s against 0.075 s.</li>
    <li><span class="review">&#9873;</span> docMapEntry keeps each entry it reads for the query, so the parts
    of a union scoring the same page read its entry once.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>Fetchers compress the pages they send, so the indexer stores
    them as they come.</strong>
    In yioop.com's indexer log, compressing archive items took 101 of
    163 s, and the indexer was 1,172 files behind.
    <ol>
    <li><span class="review">&#9873;</span> Fetcher::compressAndUnsetSeenUrls packs a text page as a
    PRECOMPRESSED_PAGE array holding the page serialized and compressed
    with gzencode at ARCHIVE_COMPRESSION_LEVEL.</li>
    <li><span class="review">&#9873;</span> PartitionDocumentBundle::put stores those bytes as they are when
    the bundle compresses with GzipCompressor, as an index bundle does.
    For any other compressor it unpacks the page first.</li>
    <li><span class="review">&#9873;</span> A page sent plain, from a fetcher not yet updated, is compressed
    by the indexer as before.</li>
    <li><span class="review">&#9873;</span> On 400 pages, the indexer's put took 0.016 s against 0.75 s, and
    the archive file was 4,621,377 bytes both ways.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A search keeps at most DOC_MAP_HANDLES_KEPT document maps
    open.</strong>
    San jose state university ran past the web server's time limit on
    yioop.com. Keeping up to 1,000 maps open in each process had raised
    the files a test index search held from 111 to 308.
    <ol>
    <li><span class="review">&#9873;</span> docMapEntry keeps at most DOC_MAP_HANDLES_KEPT, 32, maps open and
    closes the one used longest ago.</li>
    <li><span class="review">&#9873;</span> WordIterator closes the positions file of the partition it leaves.
    The same searches now hold 143 files and give the same rows.</li>
    <li><span class="review">&#9873;</span> The document map caches drop their oldest entry with unset rather
    than array_shift, 4 against 22 microseconds an entry at 10,000
    entries.</li>
    <li><span class="review">&#9873;</span> GroupIterator::getDirection asks the iterator below it, and
    currentGenDocOffsetWithWord returns its answer. GroupIterator's
    docblocks meet the standard.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A search checks filtered hosts in memory and looks up the index
    once per partition.</strong>
    San jose state university spent 1.910 s making its query iterators on
    yioop.com. A profile of the state university on a test index put 23%
    of word scoring in SearchverticalsModel::isFiltered.
    <ol>
    <li><span class="review">&#9873;</span> isFiltered queried GROUP_ITEM for each new host, and forgot all
    hosts past 400. filteredPairs reads all filtered hosts in one query,
    up to MAX_SEARCH_FILTERS_LOADED.</li>
    <li><span class="review">&#9873;</span> The pairs are read again after an edit, or when lastChange gives a
    newer time.</li>
    <li><span class="review">&#9873;</span> WordIterator keeps the index with the scored partition, rather than
    calling IndexManager::getIndex for each posting.</li>
    <li><span class="review">&#9873;</span> On that query, making the iterators took 0.074 s against 0.097 s,
    the middle of five runs, with the same 56 rows.</li>
    <li><span class="review">&#9873;</span> SearchverticalsModel called addGroupItem, which is FeedModel's, so
    filtering or editing a result stopped with a fatal error. It now
    adds them through FeedModel.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A word iterator reads up to 500 dictionary rows in one
    lookup.</strong>
    San jose state university made 466 term dictionary lookups taking
    2.2 s of the 2.9 s spent making its query iterators on yioop.com.
    <ol>
    <li><span class="review">&#9873;</span> LSMTree::get reads a word's rows from the start of its list to the
    last row asked for. Read 20 rows at a time, a word's early rows were
    read again for every batch.</li>
    <li><span class="review">&#9873;</span> NUM_DISTINCT_GENERATIONS goes from 20 to 500, kept in each
    iterator's generation_batch_size.</li>
    <li><span class="review">&#9873;</span> A later batch that fills up no longer looks up the active
    partition's row, which it neither adds nor counts.</li>
    <li><span class="review">&#9873;</span> On a 300 partition test index, a ten word query made 20 lookups
    against 38, with the same rows. batchesReachEveryPartitionTestCase
    still walks batches of 20 rows.</li>
    <li><span class="review">&#9873;</span> A batch holding the active
    partition's row is the word's last. Where that row filled the batch,
    the walk loaded one more batch, got the row again, and read the active
    partition's pages twice. batchesReachEveryPartitionTestCase now walks
    batches of 2 rows over 4 partitions of 2 pages.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>QueryTool says where a stopped query was, and a union part that
    does not advance steps on.</strong>
    On yioop.com, querytool "cs dept san jose state university" never
    finished, even for its plan. Here it took 0.08 s on three indexes.
    <ol>
    <li><span class="review">&#9873;</span> Ctrl-C now prints the calls the query was in and its timings so
    far. Windows has no such signal, so there the tool ends as before.</li>
    <li><span class="review">&#9873;</span> A union part whose document comes before an accumulator's is
    advanced to it. If the advance left it where it was, the union asked
    again forever.</li>
    <li><span class="review">&#9873;</span> Such a part now steps on one document, and QueryTool counts it as
    "union parts that did not advance".</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>The suggested query keeps numbers and short words, and shows
    the query as text.</strong>
    On yioop.com, "pandemic legacy rule vaccinate 6 cities" was offered as
    "pandemic legacy rule vaccinate p cities".
    <ol>
    <li><span class="review">&#9873;</span> correctSpelling in suggest.js changed any word not in the
    dictionary to its most frequent word one edit away. Every one letter
    string is one edit from 6, so 6 became p.</li>
    <li><span class="review">&#9873;</span> A word with a digit, or shorter than MIN_CORRECTED_LENGTH, three
    characters, is now kept as typed. A candidate not in the dictionary no
    longer takes the frequency of the candidate before it.</li>
    <li><span class="review">&#9873;</span> spellCheck put the query into the page as markup and into its
    link without encoding. It now adds the query as text, and encodes it
    in the link.</li>
    <li><span class="review">&#9873;</span> edits1 drops repeated candidates with a Set. Array.unique, which
    it replaces, skipped the element after each one it removed.</li>
    <li><span class="review">&#9873;</span> SuggestJavascriptTest checks the corrections under node. A walk
    of three queries shows the same on a results page.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A page that writes a name with accents ranks as one that writes
    it without them.</strong>
    www.sjsu.edu, titled San José State University, fell out of the top
    results for san jose state university.
    <ol>
    <li><span class="review">&#9873;</span> hyphenateEntities looks a name up again with its diacritics
    folded when it is not found as written. The list of names holds san
    jose state university without the accent.</li>
    <li><span class="review">&#9873;</span> addExactMatchScore compares the typed word and the result's text
    with diacritics folded, so jose counts as an exact match for José.</li>
    <li><span class="review">&#9873;</span> On a test index of the four pages from the reported results, the
    page went from fourth, relevance 14, to first, relevance 21.17.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>A search's first look at the url signal shares starts from
    totals saved under the temporary folder.</strong>
    signalShares read every partition's signals file in each new process.
    It now saves its totals under C\TEMP_DIR and later processes start
    from them.
    Saved totals are dropped when the last counted signals file was
    written again.</li>
    <li><span class="review">&#9873;</span>
    <strong>A full first block scores a word's field key from the word's
    other key.</strong>
    Once the first block of a union is full, a field key part read its
    postings in every partition only to add relevance.
    It now moves past the block, and the other key's walk adds its
    relevance from the same position list.
    On the local 300-partition index, postings reads fell from 782 to 576
    over ten queries with identical results.</li>
    <li><span class="review">&#9873;</span>
    <strong>A stop reaches a daemon process that misses its lock being
    gone.</strong>
    An Indexer in a long merge missed a four second stop.
    It then found the restarted queue server's lock and ran on with the
    old crawl.
    Both Indexers wrote the crawl status, so fetchers switched between
    the two crawls.
    CrawlDaemon::stop now records when it was asked, and processHandler
    stops a process that began before then.</li>
    <li><span class="review">&#9873;</span>
    <strong>Sites get the locale resources changed since version 15, and
    the unit tests run without a deprecation notice.</strong>
    Ten patches changed Tokenizer files without raising
    RESOURCES_WIKI_VERSION, so a site's own locale copy kept the old ones.
    The version is now 16, and cut_patch.sh refuses a resources change
    without a version change.
    EnTokenizerTest's case checking Tokenizer::$stems tested only the
    cache against itself, so it is gone.
    CrawlDaemon's str_getcsv now names its escape character.
    check_deprecated.sh now finds a csv call that does not.</li>
    <li><span class="review">&#9873;</span>
    <strong>The locale copy that failed EnTokenizerTest was made by code
    from 28 to 30 September.</strong>
    From commit ad520bf until commit 56dd699, Config.php copied src/locale
    into a work directory with no app/locale whenever Yioop loaded.
    A copy made then has a Tokenizer without $stems, at version 15.
    Gold, still at 15, never refreshed it, and the test stopped at line 73.
    Rebuilt that way in a scratch tree, the copy failed the gold test the
    same way.
    One page served at version 16 brought the copy up to date, and the test
    then passed.
    Today's code made no copy over a full unit run, createdb and a page
    walk.</li>
    <li><span class="review">&#9873;</span>
    <strong>QueryTool says where the site home page named by a query word
    stands in its results.</strong>
    Searches for yahoo and pepsi do not list www.yahoo.com or
    www.pepsi.com near the top, though both are in the index.
    After its statistics QueryTool runs the query again for every result it
    scores.
    For each word it prints the place, rank, relevance and score of
    https://www.word.com/ and https://word.com/.
    This gives the numbers a ranking change has to be measured against on
    the index that shows the fault.</li>
    <li><span class="review">&#9873;</span>
    <strong>A site's app/locale holds only its own changes, and programs
    the web server starts no longer hold its connections.</strong>
    Strings and resources are read from src/locale, with app/locale's
    changes in their place.
    A saved configure.ini keeps only strings that differ from the shipped
    ones.
    Database version 132 deletes resources...old folders and resources
    folders holding only copies.
    Nothing writes to src/locale; TokenTool and Manage Locales write to
    app/locale.
    A Launcher, started before the server takes connections, starts the
    programs the server is asked to start.
    A fetcher it starts held 8 files and connections, against 967.</li>
    <li><span class="review">&#9873;</span>
    <strong>php GroupWikiTool.php clean deletes unused folders from
    app/resources.</strong>
    It looks only at folders named by a hash, and their thumbnail
    folders.
    A hash is in use if it belongs to a page or post of an existing
    group, a user, or the feed.
    Folders reached from one in use by a redirect.txt or a link are kept
    too.
    Unused folders, empty hashed folders and empty prefix folders are
    deleted; a link is deleted without what it points to.
    If no folder is in use, AUTH_KEY has likely changed, and nothing is
    deleted.</li>
    <li><span class="review">&#9873;</span>
    <strong>clean deletes prefix folders that look empty.</strong>
    A prefix folder holding only .DS_Store, as Finder leaves, was kept,
    since it was not strictly empty.
    .DS_Store, Thumbs.db, desktop.ini and empty dot folders such as
    .archive no longer count.
    A page's empty sub-folder, made by a person, still keeps its page
    folder and prefix folder.
    A thumbnail folder holding only empty folders goes once its resource
    folder is gone.</li>
    <li><span class="review">&#9873;</span>
    <strong>A git page in an Invite Only group can be cloned with a user
    name and Git application code.</strong>
    A clone of a group the public may not read got 404 Not Found.
    git sends the url's name and code only after a 401 asking for them.
    GitComponent::checkReader now answers 401 with WWW-Authenticate when
    no code is accepted.
    A member of the group, or root, is then served; anyone else gets
    404.</li>
    <li><span class="review">&#9873;</span>
    <strong>Every docblock in GitComponent.php names what it documents and
    explains WebDAV.</strong>
    The class and 26 methods opened with a bare verb, such as Sends or
    Reports.
    Each was rewritten by hand to open with its name and say who calls it.
    WebDAV is explained as the protocol for writing files on a web server
    wherever it is used.</li>
    <li><span class="review">&#9873;</span>
    <strong>A git page takes pushes over the smart protocol, so a pushed
    user name and code can come from a prompt.</strong>
    git-http-push, the older dumb push, never asks for a password.
    It failed with "Cannot access URL ..., return code 22" unless the url
    held the code.
    GitReceivePack answers service=git-receive-pack and stores each
    pushed object as a loose object.
    It moves a ref only when its old value matches, and reports each
    command.
    GitComponent::serveReceivePack asks for an editor's code with a 401
    first.
    settle_gold.sh now moves work's confirmed-head to the work commit
    matching gold, not to work's HEAD.
    On HEAD, a settle made mid-round left that round's commits out of
    the next cut.</li>
    <li><span class="review">&#9873;</span>
    <strong>Each Git repository page has its own Git application
    code.</strong>
    A code was kept per person, so a fresh code on one page broke every
    other repository's clone urls.
    GIT_APP_CODE is now keyed by USER_ID, GROUP_ID and PAGE_ID, the page's
    id in the default locale.
    Database version 133 rebuilds the table and keeps each old code with
    group and page 0.
    GitModel::getAppCode gives that old code for any page with no code of
    its own.</li>
    <li><span class="review">&#9873;</span>
    <strong>A git push of tens of megabytes goes through over HTTP/2 and
    HTTP/1.1.</strong>
    Over HTTP/2 the server copied a request body whole on every frame, so
    a 40 MB push never finished.
    Over HTTP/1.1 a chunked body, which git sends past one megabyte, was
    not read at all.
    The HTTP/2 body now grows in place, and WebSite::decodeChunkedBody
    joins chunked bodies.
    index.php raises the request size limit to the Git push size limit,
    or lifts it when that is Unlimited.
    GitReceivePack writes each object as it reads it, keeping only names
    in memory.</li>
    <li><span class="review">&#9873;</span>
    <strong>A clone or pull copies stored pack entries instead of
    rebuilding every object.</strong>
    A clone decompressed, rebuilt and recompressed every object, so a
    large history took minutes before git got a reply.
    GitPackReader::storedEntry gives an entry still compressed; packChunks
    copies it when the client can read it.
    A delta stored by position is sent by its base's name.
    On a 6,967-commit test repository a clone's reply took 7 s, against
    54 s before.</li>
    <li><span class="review">&#9873;</span>
    <strong>An old site upgrades without warnings, and imports a large
    export of wiki pages.</strong>
    Upgrade 65 made the Search group through GroupModel, which writes to
    SOCIAL_GROUPS; before upgrade 70 that table is GROUPS.
    It now inserts into GROUPS, and addGroup gives -1 when a group cannot
    be stored.
    updateTranslationForStringId skips a locale the site does not have.
    WikiPagesFile::eachPage reads a pages file a page at a time, and
    upgradePublicHelpWiki writes each page as it comes.
    The shipped pages now import within 24 MB of memory.</li>
    <li><span class="review">&#9873;</span>
    <strong>A clone answered by a web server worker no longer runs the
    server out of memory.</strong>
    A worker sent its whole reply to the front process as one message.
    A 334 MB pack then overran a 128 MB limit, and the front stopped.
    git then saw the connection end with no reply.
    A worker now hands back the name of the reply's file, and the front
    sends the file from disk.
    The HTTP/2 path also deletes such a file once it is sent, as HTTP/1.1
    already did.</li>
    <li><span class="review">&#9873;</span>
    <strong>A clone's reply starts reaching git while its pack is still
    being built.</strong>
    serveUploadPack names its reply file to the server as a growing file
    once the first piece is written.
    The server sends what is written, in chunks under HTTP/1.1 and in DATA
    frames under HTTP/2 and HTTP/3.
    A deferred file reply over HTTP/3 had been handed to the HTTP/2 sender
    and sent nothing; it now has its own path.
    A worker names the file to the front at once, and its last reply ends
    the send.
    An HTTP/1.1 connection waiting on the file leaves the write set, so the
    loop does not spin.
    On the 6,967-commit test repository git had headers 1.2 to 1.6 s after
    asking, with and without workers.</li>
    <li><span class="review">&#9873;</span>
    <strong>git fetch --depth deepens a shallow clone, and git fetch
    --deepen works.</strong>
    Held history was marked past the client's shallow commits, so a deeper
    fetch found nothing to send.
    That marking now stops at them, and shallowCutoff works out the new
    end of history apart from what the client holds.
    The advertisement offers deepen-relative, which --deepen counts from
    the client's shallow commits.
    A depth round that offers held commits now gets its acknowledgments,
    which --unshallow waited for.
    Every docblock in GitUploadPack, GitUploadPackTest and WebSiteTest now
    opens with the name of what it documents.
    The one-line isObjectName and two one-line test helpers are written
    where they were used, and an unused test helper is gone.</li>
    <li><span class="review">&#9873;</span>
    <strong>The process file asks for whole-request patches and enforces the
    nine-minute round.</strong>
    A patch carries every part of the request, including items the work
    finds the request needs.
    settle_gold.sh records when a round began, and cut_patch.sh refuses an
    early cut without SOLVED=1.
    A reproduction takes the reporting site's worker count and memory
    limit, and no command reaches an outside host.</li>
    <li><span class="review">&#9873;</span>
    <strong>Resource folders are named from group, page and user names, so
    they are the same on every instance.</strong>
    A page's files are in &lt;G&gt;/wiki/&lt;locale&gt;/&lt;P&gt;/page and
    its thumbnails in the thumbs folder beside it.
    A post's are under &lt;G&gt;/posts/&lt;YYYY-MM&gt;, named from its
    author and time; feed pictures are in the Public group's feeds folder.
    Each fixed name is seven readable characters, padded with _, then the
    raw crawlHash in hexadecimal.
    Icons are in users/&lt;U&gt;, site files in site, and a host's files in
    domains/&lt;host&gt;.
    Personal groups are named &lt;user name&gt;$Personal, and createGroup
    refuses any name holding a $.
    A request with s reads only user_icon.webp, and one with neither g nor s
    only the site or a host folder.
    Upgrade 134 and ConfigureTool migrate-pre-v10-resources move the
    pre-version 10 folders and rename personal groups.
    Running either again does nothing.
    The migration is the class V10ResourceFolderMigration, and no
    day-to-day class knows the pre-version 10 layout.
    Message links are relative, and the clean command follows the version
    10 layout.</li>
    <li><span class="review">&#9873;</span>
    <strong>The resource migration deletes prefix folders holding only
    Finder or Explorer files.</strong>
    A prefix folder holding only .DS_Store, Thumbs.db or desktop.ini was
    kept, as a UuB folder was on a Mac test site.
    Such folders now go, and a prefix folder holding anything else is named
    in the report with what it holds.</li>
    <li><span class="review">&#9873;</span>
    <strong>ConfigureTool's menu has a step that migrates pre-version 10
    resource folders.</strong>
    Migrate Pre-Version 10 Resource Folders runs what the
    migrate-pre-v10-resources command runs.
    Its report shows as the menu's message.</li>
    <li><span class="review">&#9873;</span>
    <strong>An empty size setting no longer stops the server's session
    settings.</strong>
    metricToInt read the last letter of an empty GIT_MAX_PUSH_SIZE and
    printed a warning.
    That output came before index.php set session.gc_maxlifetime, so PHP
    refused the setting with a second warning.
    An empty size now reads as 0, and an empty name server field gets its
    slash the same way.</li>
    <li><span class="review">&#9873;</span>
    <strong>A slow request no longer leaves the requests queued behind it
    unanswered.</strong>
    On yioop.com a worker was ended after 6 seconds on a search, with 11
    requests queued to it.
    Those requests then waited on the worker started under the same
    number, which never got them, for 30 to 359 seconds.
    A request now waits on its worker's link, not its number.
    The front holds a request until a worker is free, so none queues behind
    a slow one.
    The worker's word that it took a request up is kept apart, so an ended
    request is answered with a 502, not run again.</li>
    <li><span class="review">&#9873;</span>
    <strong>The web server keeps serving while a worker starts.</strong>
    startWorker waited for a new worker to connect, and the whole front
    waited with it.
    A favicon on yioop.com took 3.96 seconds during one such start.
    finishWorkerStarts now takes a worker in once it connects, on each
    pass of the event loop.
    A worker whose front never answers its handshake gives up after
    WORKER_START_SECONDS, not waiting for good.</li>
    <li><span class="review">&#9873;</span>
    <strong>The supervisor starts the mail server as root.</strong>
    The root supervisor handed MailServer to the Launcher, which runs as
    WWW_USER.
    The mail server then could not bind its ports below 1024 and stopped,
    and was started again every 20 seconds.
    The Launcher now writes the user it runs as, and only a caller running
    as that user hands it a program.</li>
    <li><span class="review">&#9873;</span>
    <strong>A start that fails says why.</strong>
    On pollett.org, after a move from Intel to Apple silicon Homebrew,
    secure-start started nothing and printed nothing.
    The programs a start launches sent their output to /dev/null, so a
    php that failed to load said so to no one.
    Their output now goes to the end of index.log.
    They are run with the php running the start, not whichever php the
    path finds.
    A start of the web server waits up to 15 seconds for it to answer,
    and otherwise prints the end of index.log.</li>
    <li><span class="review">&#9873;</span>
    <strong>The Launcher writes no log of its own.</strong>
    Its lines go to index.log.</li>
    <li><span class="review">&#9873;</span>
    <strong>No file is kept for the web server's port or the Launcher's
    user.</strong>
    IndexServerPort.txt and LauncherUser.txt are gone.
    A start connects to the server's address to see that it came up.
    It refuses at once when another program answers there.
    A caller that is not root shares the Launcher's user when it may
    signal the Launcher's process.
    Root hands a program over only where there is no WWW_USER.</li>
    <li><span class="review">&#9873;</span>
    <strong>The site's logos, favicon and background are found by the
    names of their roles.</strong>
    Profile.php no longer holds LOGO_SMALL, LOGO_MEDIUM, LOGO_LARGE,
    FAVICON or BACKGROUND_IMAGE.
    An upload is saved in app/resources/site as logo-small, logo-medium,
    logo-large, favicon or background, with its type's extension.
    ResourceFolders::siteImageUrl gives that file, or else the image
    Yioop ships.
    Database version 135 renames images uploaded under their own names
    and writes Profile.php again without the five fields.</li>
        <li><span class="check">&#10003;</span>
        <strong>A ballot witness under LDAP gives a directory login
        again.</strong> The witness form asked for a login only when USE_LDAP
        was set, and nothing defines it now. It checks AUTH_METHOD instead,
        and the emailed-link page asks for the login too. The login must map
        to that witness's own account, which a new case checks.</li>
        <li><span class="check">&#10003;</span>
        <strong>A saved page resource is served as saved.</strong> An
        edited HTML resource kept being served in its old form. The save
        went through VersionManager, which wrote with file_put_contents.
        WikiModel now gives VersionManager its filePutContents, and
        WebSite::filePutContents drops the gzip copy of a file it
        writes.</li>
    <li><span class="review">&#9873;</span>
    <strong>The unit tests write only inside the tests folder.</strong>
    A run under strace found 53 test files writing the site's work
    directory, its databases, /tmp or /root.
    <ol>
        <li><span class="review">&#9873;</span>
        <strong>The tests run in a work directory of their own.</strong>
        CodeTool's unit command uses tests/test_files/test_work_directory.
        Its profile names two SQLite databases there, whatever database the
        site uses, and createdb fills them on the first run. Config.php takes
        that folder from YIOOP_TEST_WORK_DIRECTORY on the command line.</li>
        <li><span class="review">&#9873;</span>
        <strong>Tests keep their own files in tests/test_files.</strong>
        Files made in /tmp or the work directory now go there. openssl's
        random seed goes there through RANDFILE, and the test web servers
        keep sessions in C\TEMP_DIR.</li>
        <li><span class="review">&#9873;</span>
        <strong>check_test_writes.sh names each write outside
        tests.</strong> It runs each test file under strace and prints full
        paths. CrawlDaemonTest no longer runs php -r, and MessagesBundleTest
        no longer runs rm -rf.</li>
        <li><span class="review">&#9873;</span>
        <strong>The tests' profile parses when the site's database address
        holds a semicolon.</strong> UnitTest::testWorkProfile replaces each
        database setting's whole line. Replacing up to the first semicolon
        broke an address such as mysql:host=localhost;dbname=yioop.</li>
        <li><span class="review">&#9873;</span>
        <strong>The tests' databases are copied again when they are empty
        or old.</strong> They are copied from the two databases Yioop ships
        in src/data and upgraded to DATABASE_VERSION. That takes about two
        seconds, where createdb took 43. A run only checked that the file was there, so a
        run that stopped left empty databases every later run used.
        ProfileModel::testDatabasesMade checks the version and tables
        first. The unit command upgrades databases only in the tests' work
        directory. Listing the tests leaves the site's databases as they
        are.</li>
        <li><span class="review">&#9873;</span>
        <strong>A request waiting while every worker is replaced waits for
        the new one.</strong> waitForFreeWorker answered 503 when all workers
        had been asked to stop. It now waits until one is started in their
        place.</li>
    </ol></li>
    <li><span class="review">&#9873;</span>
    <strong>The unit tests start no processes and use the site's work
    directory.</strong>
    <ol>
        <li><span class="review">&#9873;</span>
        <strong>The tests' own work directory is gone.</strong> The tests
        use C\WORK_DIRECTORY, C\TEMP_DIR and C\TEST_DIR again. With the
        second process that set it up gone, results are colored again.</li>
        <li><span class="review">&#9873;</span>
        <strong>No test starts a process.</strong> UnitTest::startMethod and
        the unit command's child mode are gone. CsrBuilderTest, ArcToolTest
        and AVToolTest now call the code in their own process.</li>
        <li><span class="review">&#9873;</span>
        <strong>Cases that need a second process are removed.</strong> They
        are three UndoLog stop cases, the feed stop case and the queue
        server message case. Also four CrawlDaemon cases, seven FetchUrl
        server cases and five WebSite worker cases.</li>
        <li><span class="review">&#9873;</span>
        <strong>Tests write only in the tests folder.</strong> Code a test
        reaches takes the folder it writes in as a property, such as
        CrawlDaemon::$schedules_folder and ProfileModel::$resource_path.
        Tests that used the site's databases make their own in
        tests/test_files.</li>
        <li><span class="review">&#9873;</span>
        <strong>UndoLog syncs with fsync only.</strong> It ran sync -f on
        Linux. The tesseract case of PdfProcessorTest is removed.</li>
        <li><span class="review">&#9873;</span>
        <strong>Folder lookups that can give false are not taken
        apart.</strong> PHP 8.5 warns when list() takes apart false.
        deleteGroupPage and two other callers now give an array in its
        place, and check_deprecated.sh names such statements.</li>
    <li><span class="review">&#9873;</span>
        <strong>The unit tests run faster and print as they go.</strong>
        The suite took 38 seconds, with 66 cases over a tenth of a second.
        It now takes about 12 seconds, and the cases still over that limit are
        listed for Chris to decide on.
        <ol>
            <li><span class="review">&#9873;</span> CodeTool.php unit prints
            each test class's results as soon as the class has run.</li>
            <li><span class="review">&#9873;</span> Key and password cases use
            smaller RSA keys and the cheapest bcrypt cost. Production sizes and
            costs are unchanged.</li>
            <li><span class="review">&#9873;</span> The storage cases use
            fewer rows and trees, and each still checks the same behavior.
            Two waits on a clock take a twentieth of a second.</li>
            <li><span class="review">&#9873;</span> namesOfferedExtend
            WhatWasTypedTestCase uses its own test database. It failed on a
            site with five users whose names start with "ro" before
            "root".</li>
            <li><span class="review">&#9873;</span> AVToolTest no longer
            checks that a still picture is the default. Chris removed it as
            checking one line of option handling.</li>
            <li><span class="review">&#9873;</span> indexServerUpWaitsFor
            AnAnswerTestCase checks that the wait writes no file of ports.
            It failed on a site still holding IndexServerPort.txt from
            before.</li>
        </ol></li>
    </ol></li>
</ol></li>
</ol>
</body>
</html>