Add a conformance suite for HermitCrab, with its adequacy argument by johnml1135 · Pull Request #480 · sillsdev/machine

johnml1135 · 2026-08-19T19:22:23Z

Deferred until after the Hermit Crab Optimizations release. Nothing stacks on this PR any more: #534, #540, #522 and #519 were retargeted to master, and #543 was closed. Their suite-side work is held back and will be added here when this resumes, each once its own PR has merged:

Trace HermitCrab failure reasons and template slot outcomes #534: FailureRuleAttributor credits through Trace.Allomorph; FieldWorks-producibility rows for the four new failure reasons.

Fix narrowing-rule unapplication across an interposed optional node #540: the edge-cases/interposed-optional-multi-rhs fixture.

Remove HermitCrab's analysis-cascade memo #522: removing the harness's --no-memoization flag along with the memo.

Add conformance fixtures for the #504 and #507 evidence gaps #543: the syntactic-feature-inversion-controls (HermitCrab: track exact analysis feature inversion and chained override parse loss #504) and final-template-gap-coverage (HermitCrab: complete cross-engine correctness controls for final-template pruning #507) fixtures.

Quick summary

Every HermitCrab change now runs against a conformance suite of synthetic grammars, and CI fails when a parse, a coverage claim or a generated ledger drifts.
It is large because most of it is fixture data and generated ledgers; the harness is one new project and no engine code changes.
Engine fixes that the suite surfaced live in their own PRs, stacked on this one.

Where to look

.github/workflows/conformance.yml -- the gate. Runs on any change under src/SIL.Machine.Morphology.HermitCrab*, src/SIL.Machine, the HermitCrab tests or conformance/: self-check of every fixture, the coverage report, and a freshness check on the generated CSVs.
conformance/dead-rules-allowed.tsv -- the coverage report now exits 1 on a rule no word exercises unless it is listed here with a reason, and on a listed rule that is exercised again. Today: 15 entries, all deliberate (inactive decoys, refused controls, crash pins).
conformance/PROTOCOL.md -- the adapter contract a second engine implements.
src/SIL.Machine.Morphology.HermitCrab.Conformance/Runner.cs -- self-check: fixtures run in-process against HermitCrab as the oracle.
global.json -- new: pins the SDK to 10.0.100 with latestFeature roll-forward, for the whole repository.

Reading order

About 25k lines are harness code (src/SIL.Machine.Morphology.HermitCrab.Conformance) and 16k are its tests. The rest is data: fixtures (conformance/edge-cases, conformance/languages, about 21k), generated ledgers and evidence cards (about 28k), and docs. Generated files are rewritten by hc-conformance --write-* and pinned by tests, so they need checking for freshness, not reading line by line.

Stacked on this PR

Trace HermitCrab failure reasons and template slot outcomes #534 -- restores failure-allomorph attribution through its own Trace.Allomorph, and classifies its four new FailureReason values in the FieldWorks-producibility ledger.
Fix narrowing-rule unapplication across an interposed optional node #540 -- adds interposed-optional-multi-rhs, which fails on master and passes with the fix.
Remove HermitCrab's analysis-cascade memo #522 -- removes the analysis memo, and with it this harness's --no-memoization flag.
Prune reduplication matches whose copies cannot agree #519 -- rebased only.
Add conformance fixtures for the #504 and #507 evidence gaps #543 -- fixtures for the HermitCrab: track exact analysis feature inversion and chained override parse loss #504 and HermitCrab: complete cross-engine correctness controls for final-template pruning #507 evidence gaps.

Validation

./local_check.sh --agent-strict steps at this head: comment hygiene clean; dotnet csharpier check . clean (870 files); Release build clean.
dotnet test -- HermitCrab 608 passed, 0 failed at this head. SIL.Machine 887 passed, 3 skipped; Thot 93 passed; SentencePiece 4 passed, at cc947db; nothing outside the HermitCrab conformance code and tests has changed since.
hc-conformance --fixtures conformance --include-pathological -- 58 passed, 0 failed.
hc-conformance --fixtures conformance --coverage-report -- exit 0; 30/30 constructs; 15 dead rules, all allowed; CSVs unchanged.
hc-conformance --check-manifest -- current (58 fixtures).
ANonAsciiSurfaceFormSurvivesTheMutantChildProcess fails without the encoding fix under code page 437 and passes with it.

What changed in this revision

Rebased onto master (be5407d).
Dropped Trace.FailureAllomorph: an engine API change does not belong in a test-suite PR, and Trace HermitCrab failure reasons and template slot outcomes #534 adds the same fact as Trace.Allomorph. Generated CSVs regenerate byte-identical without it.
The coverage report could not fail; it now gates dead rules and uncovered constructs.
ci.yml is back to master. Its added fixture run duplicated ConformanceFixtureGateTests under dotnet test, and parity-check.py's remaining check is now enforced by the coverage report.
The counterfactual sweep read mutant output in the console code page, so "katibɯd" came back as "katib?d" and a mutation that changed nothing looked like evidence. It now reads UTF-8 (CounterfactualGate, plus a regression test). Regenerating the ledger demoted two surfaces that had only that false evidence: Gloss to Unobservable and HeadFeatures to RequiredByDtd. The gap ratchet moves from 25 to 26 for Gloss.
local_check.sh ran --semantic-coverage, which exits 1 by design while the catalog is incomplete, so it could never pass. It now runs the same gates as conformance.yml.
chained-output-feature-override-loss said it fails on master on purpose. It has passed since Change Add to PriorityUnion #494; checked by running it at 52d069f (red) and 7109531 (green). The header now says so.

Earlier decisions, and why

PCRE.NET (now on master) ships an IIncrementalGenerator. The compilation-graph loader admits it only by content hash, and a probe runs it and fails closed on any generated output.
An expect_crash fixture passes only if the same engine completes an ordinary fixture in the same run, so a broken adapter command cannot pass as a crash (conformance/PROTOCOL.md).
*.fwdata is marked binary in .gitattributes; its pinned hash depended on core.autocrlf.
Hand-maintained fixture totals were removed from the README and docs; the manifest is the count.
Four fixture grammars keep trailing whitespace: their hashes are pinned in the manifest.

🤖 Generated with Claude Code

This change is

codecov-commenter · 2026-08-20T05:05:21Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.17%. Comparing base (a2f80e2) to head (6646050).

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #480      +/-   ##
==========================================
- Coverage   74.36%   74.17%   -0.19%     
==========================================
  Files         456      556     +100     
  Lines       38325    56087   +17762     
  Branches     5263     7637    +2374     
==========================================
+ Hits        28499    41603   +13104     
- Misses       8669    12625    +3956     
- Partials     1157     1859     +702

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Introduces an engine-agnostic conformance suite for HermitCrab-formalism morphological parsers: a `conformance/` tree of language grammars and edge-case micro-grammars, each an HC-XML grammar.xml paired with a words.yaml ground truth (oracle-provenance, gloss, and per-word parse signatures). A new SIL.Machine.Morphology.HermitCrab.Conformance project and hc-conformance CLI tool run those fixtures against an engine adapter, producing coverage and rules CSVs, an interface inventory, dataflow- obligation ledgers, evidence cards, a FieldWorks-producibility table, and a JSON manifest for external consumers. PROTOCOL.md is the adapter contract a second engine (e.g. the in-progress Rust port) implements to plug into the same suite. Includes the FieldWorks-producibility classification campaign: for every HermitCrab construct the suite covers, a verdict on whether FieldWorks' own HC-XML export path can actually produce a grammar exercising it, distinguishing genuine data-model limits from mere loader gaps, with five previously-unresolved verdicts now classified and corrected write-ups for two more cases initially misread as exporter silence. Also lands one new edge-case fixture, edge-cases/chained-output-feature-override-loss, isolating an analysis- side OutputHeadFeatures retraction bug: a well-formed four-morph parse of `zudiua` that the founding C# oracle currently loses. The fixture is deliberately red against hc.dll today (expected, not a broken test) and documents the mechanism, provenance, and the HC-Rust divergence that motivated it. The suite is deliberately stopped at this point rather than finished: scope, known gaps, and next steps are documented in the suite's own tracking docs rather than left implicit. This commit squashes the full conformance-suite branch stack (integrate-conformance-framework through conformance/fieldworks- witnesses, 49 commits) into one commit on top of current master, for PR #480. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Records a break-attempt run of PR #494 (AnalysisAffixProcessRule fix) against the conformance suite: 36/36 fixtures pass (including pathological cases), and a generated-word fuzz sweep (every root x every 0..3-subset of affix rules, up to 6 nonhead roots where the grammar compounds) produced ~1,300 words across 35 grammars, parsed by both master and the PR branch with 0 divergences attributable to PR #494. The one observed diff (edge-cases/truncate-morphotactic, words `a`/`ga`) is pre-existing oracle nondeterminism in analysis multiplicity ordering, reproducible on master alone, unrelated to the syntactic FS change. This was previously recorded only at tag archive/pr494-conformance; folding it into the conformance suite's own PR (#480) so the evidence travels with the suite it was run against. Fixed two references to branches deleted since the doc was written (perf/pr494-conformance, conformance/fieldworks-witnesses) to point at their preserved archive/* tags instead; no other content changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

johnml1135 · 2026-09-15T17:40:35Z

Added two independently justified conformance fixtures in a20bce1: final-template-partial-discriminators (10 words; #507) and template-category-sharing (6 words; #505). Fresh Release build: zero warnings/errors. Both fixtures pass with memoization on and off, zero skipped; 22 focused manifest/coverage tests pass. Four grammar counterfactuals produce the independently predicted mismatches, documented under conformance/docs. These are grammar-sensitivity controls, not proof that pruning, cache hits, or same-key feature collisions execute. No parser code changed. Generated inventory also restores previously omitted rows for the already-existing Exact-inverse fixture; its valid zudiua loss remains tracked in #504, not relabeled as correct.

johnml1135 · 2026-09-15T18:49:09Z

Added the shared-template unconstrained-suffix conformance fixture in f150e2a. The grammar has a slot-only feature-unconstrained suffix shared by mandatory TV/IV templates, an ordinary N-to-IV verbalizer, four positive rows, and six rejection controls. The companion rationale explains complete forward derivations and loader reachability (not a FieldWorks UI round trip).

Fresh C# harness: all 10 rows pass in both template orders, memoization on/off (40 comparisons; no skips). Focused template/manifest tests: 22 passed, no skips. Regenerated manifest is current at 40 fixtures; coverage/rule reports updated. No Machine parser code changed.

PanGloss main now contains the narrow gate-only alignment (fc357daf), with this fixture mirrored unchanged until its old Machine submodule pin is upgraded. Direct state assertions provide red/green evidence; full-parse fixture checks pass both before and after the change. Issue #505 remains open for collision/correlation and broader merge-equivalence evidence.

Introduces an engine-agnostic conformance suite for HermitCrab-formalism morphological parsers: a `conformance/` tree of language grammars and edge-case micro-grammars, each an HC-XML grammar.xml paired with a words.yaml ground truth (oracle-provenance, gloss, and per-word parse signatures). A new SIL.Machine.Morphology.HermitCrab.Conformance project and hc-conformance CLI tool run those fixtures against an engine adapter, producing coverage and rules CSVs, an interface inventory, dataflow- obligation ledgers, evidence cards, a FieldWorks-producibility table, and a JSON manifest for external consumers. PROTOCOL.md is the adapter contract a second engine (e.g. the in-progress Rust port) implements to plug into the same suite. Includes the FieldWorks-producibility classification campaign: for every HermitCrab construct the suite covers, a verdict on whether FieldWorks' own HC-XML export path can actually produce a grammar exercising it, distinguishing genuine data-model limits from mere loader gaps, with five previously-unresolved verdicts now classified and corrected write-ups for two more cases initially misread as exporter silence. Also lands one new edge-case fixture, edge-cases/chained-output-feature-override-loss, isolating an analysis- side OutputHeadFeatures retraction bug: a well-formed four-morph parse of `zudiua` that the founding C# oracle currently loses. The fixture is deliberately red against hc.dll today (expected, not a broken test) and documents the mechanism, provenance, and the HC-Rust divergence that motivated it. The suite is deliberately stopped at this point rather than finished: scope, known gaps, and next steps are documented in the suite's own tracking docs rather than left implicit. This commit squashes the full conformance-suite branch stack (integrate-conformance-framework through conformance/fieldworks- witnesses, 49 commits) into one commit on top of current master, for PR #480. Folds in three review fixes: AdapterEngine's contract in MaterializedRunner now credits an expect_crash fixture only once the same engine also completes an ordinary fixture from the same run, so a misconfigured adapter (bad path, rejected arguments) can no longer register as reproducing the pinned crash; GuesserSignatureTests' batch-path test is renamed for the overload it actually calls; and .gitattributes freezes project.fwdata as binary so its pinned SHA-256 is reproducible across platforms and CI runners, not just this one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Records a break-attempt run of PR #494 (AnalysisAffixProcessRule fix) against the conformance suite: 36/36 fixtures pass (including pathological cases), and a generated-word fuzz sweep (every root x every 0..3-subset of affix rules, up to 6 nonhead roots where the grammar compounds) produced ~1,300 words across 35 grammars, parsed by both master and the PR branch with 0 divergences attributable to PR #494. The one observed diff (edge-cases/truncate-morphotactic, words `a`/`ga`) is pre-existing oracle nondeterminism in analysis multiplicity ordering, reproducible on master alone, unrelated to the syntactic FS change. This was previously recorded only at tag archive/pr494-conformance; folding it into the conformance suite's own PR (#480) so the evidence travels with the suite it was run against. Fixed two references to branches deleted since the doc was written (perf/pr494-conformance, conformance/fieldworks-witnesses) to point at their preserved archive/* tags instead; no other content changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Introduces an engine-agnostic conformance suite for HermitCrab-formalism morphological parsers: a `conformance/` tree of language grammars and edge-case micro-grammars, each an HC-XML grammar.xml paired with a words.yaml ground truth (oracle-provenance, gloss, and per-word parse signatures). A new SIL.Machine.Morphology.HermitCrab.Conformance project and hc-conformance CLI tool run those fixtures against an engine adapter, producing coverage and rules CSVs, an interface inventory, dataflow- obligation ledgers, evidence cards, a FieldWorks-producibility table, and a JSON manifest for external consumers. PROTOCOL.md is the adapter contract a second engine (e.g. the in-progress Rust port) implements to plug into the same suite. Includes the FieldWorks-producibility classification campaign: for every HermitCrab construct the suite covers, a verdict on whether FieldWorks' own HC-XML export path can actually produce a grammar exercising it, distinguishing genuine data-model limits from mere loader gaps, with five previously-unresolved verdicts now classified and corrected write-ups for two more cases initially misread as exporter silence. Also lands one new edge-case fixture, edge-cases/chained-output-feature-override-loss, isolating an analysis- side OutputHeadFeatures retraction bug: a well-formed four-morph parse of `zudiua` that the founding C# oracle currently loses. The fixture is deliberately red against hc.dll today (expected, not a broken test) and documents the mechanism, provenance, and the HC-Rust divergence that motivated it. The suite is deliberately stopped at this point rather than finished: scope, known gaps, and next steps are documented in the suite's own tracking docs rather than left implicit. This commit squashes the full conformance-suite branch stack (integrate-conformance-framework through conformance/fieldworks- witnesses, 49 commits) into one commit on top of current master, for PR #480. Folds in three review fixes: AdapterEngine's contract in MaterializedRunner now credits an expect_crash fixture only once the same engine also completes an ordinary fixture from the same run, so a misconfigured adapter (bad path, rejected arguments) can no longer register as reproducing the pinned crash; GuesserSignatureTests' batch-path test is renamed for the overload it actually calls; and .gitattributes freezes project.fwdata as binary so its pinned SHA-256 is reproducible across platforms and CI runners, not just this one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Records a break-attempt run of PR #494 (AnalysisAffixProcessRule fix) against the conformance suite: 36/36 fixtures pass (including pathological cases), and a generated-word fuzz sweep (every root x every 0..3-subset of affix rules, up to 6 nonhead roots where the grammar compounds) produced ~1,300 words across 35 grammars, parsed by both master and the PR branch with 0 divergences attributable to PR #494. The one observed diff (edge-cases/truncate-morphotactic, words `a`/`ga`) is pre-existing oracle nondeterminism in analysis multiplicity ordering, reproducible on master alone, unrelated to the syntactic FS change. This was previously recorded only at tag archive/pr494-conformance; folding it into the conformance suite's own PR (#480) so the evidence travels with the suite it was run against. Fixed two references to branches deleted since the doc was written (perf/pr494-conformance, conformance/fieldworks-witnesses) to point at their preserved archive/* tags instead; no other content changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Introduces an engine-agnostic conformance suite for HermitCrab-formalism morphological parsers: a `conformance/` tree of language grammars and edge-case micro-grammars, each an HC-XML grammar.xml paired with a words.yaml ground truth (oracle-provenance, gloss, and per-word parse signatures). A new SIL.Machine.Morphology.HermitCrab.Conformance project and hc-conformance CLI tool run those fixtures against an engine adapter, producing coverage and rules CSVs, an interface inventory, dataflow- obligation ledgers, evidence cards, a FieldWorks-producibility table, and a JSON manifest for external consumers. PROTOCOL.md is the adapter contract a second engine (e.g. the in-progress Rust port) implements to plug into the same suite. Includes the FieldWorks-producibility classification campaign: for every HermitCrab construct the suite covers, a verdict on whether FieldWorks' own HC-XML export path can actually produce a grammar exercising it, distinguishing genuine data-model limits from mere loader gaps, with five previously-unresolved verdicts now classified and corrected write-ups for two more cases initially misread as exporter silence. Also lands one new edge-case fixture, edge-cases/chained-output-feature-override-loss, isolating an analysis- side OutputHeadFeatures retraction bug: a well-formed four-morph parse of `zudiua` that the founding C# oracle currently loses. The fixture is deliberately red against hc.dll today (expected, not a broken test) and documents the mechanism, provenance, and the HC-Rust divergence that motivated it. The suite is deliberately stopped at this point rather than finished: scope, known gaps, and next steps are documented in the suite's own tracking docs rather than left implicit. This commit squashes the full conformance-suite branch stack (integrate-conformance-framework through conformance/fieldworks- witnesses, 49 commits) into one commit on top of current master, for PR #480. Folds in three review fixes: AdapterEngine's contract in MaterializedRunner now credits an expect_crash fixture only once the same engine also completes an ordinary fixture from the same run, so a misconfigured adapter (bad path, rejected arguments) can no longer register as reproducing the pinned crash; GuesserSignatureTests' batch-path test is renamed for the overload it actually calls; and .gitattributes freezes project.fwdata as binary so its pinned SHA-256 is reproducible across platforms and CI runners, not just this one. Also folds in the follow-up PR commits: fuzz/break-attempt evidence; admitting PCRE.NET's source generator via a vetted content-hash allowlist; final-template and template-sharing controls; oracle-verified fixtures for per-piece environments, simultaneous mode and iterative epenthesis; reduplication x phonology cases; and a regenerated manifest, ledgers and evidence cards. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude · 2026-10-10T00:49:58Z

+            MaterializedFixture materialized = FixtureMaterializer.Materialize(fixture);
+            try
+            {
+                MaterializedRunReport oneReport = MaterializedRunner.Run(


Minor: F2. The expect_crash control guard is dead in --adapter mode: RunAdapter hands MaterializedRunner.Run a one-element list, so fixtures.FirstOrDefault(!ExpectCrash) is always null and adapterIsFunctional is always true (MaterializedRunner.cs:170-171). An adapter that exits nonzero on everything passes simultaneous-epenthesis-cascade, which is the case the PR description says the guard prevents. Pass all fixtures in one call, or pick the control from fixtures here.

claude · 2026-10-10T00:49:59Z

+        {
+            sw.Stop();
+            result.ElapsedMs = sw.ElapsedMilliseconds;
+            if (fixture.Words.ExpectCrash)


Minor: F3. In self-check mode an expect_crash fixture passes on any exception, including this file's own watchdog TimeoutException (line 190) and harness bugs. The fixture header in simultaneous-epenthesis-cascade/words.yaml and PROTOCOL.md say a timeout is never a crash match; adapter mode honors that by catching only EngineCrashException. Filter out TimeoutException here at least.

claude · 2026-10-10T00:50:00Z

+<Project>
+  <ItemGroup>
+    <Reference Include="Microsoft.CodeAnalysis">
+      <HintPath>$(MSBuildToolsPath)/Roslyn/bincore/Microsoft.CodeAnalysis.dll</HintPath>


Minor: F4. Unverified: $(MSBuildToolsPath)/Roslyn/bincore/ exists only in the .NET SDK layout; Visual Studio MSBuild keeps Roslyn in MSBuild\Current\Bin\Roslyn with no bincore. The existing SIL.Machine.Morphology.HermitCrab.Tests project now imports this, so a VS build of Machine.sln would likely fail to resolve Microsoft.CodeAnalysis, and the compiler version follows whichever SDK is installed (hence the 10.0.303 pin in ci.yml). A Microsoft.CodeAnalysis.CSharp PackageReference avoids both.

F4 still applies at cc947db: ci.yml dropped its 10.0.303 pin, but conformance.yml keeps it. dotnet test in the main CI job therefore builds SIL.Machine.Morphology.HermitCrab.Tests against whichever 10.0.x SDK Roslyn is installed. The two workflows can now compile the semantic-coverage tests against different Microsoft.CodeAnalysis versions. Unverified that any pinned hash depends on it.

claude · 2026-10-14T20:37:02Z

+
+    internal static readonly TimeSpan DefaultTimeout = TimeSpan.FromSeconds(180);
+
+    private const int TimeoutConfirmationAttempts = 1;


FYI: F12. TimeoutConfirmationAttempts = 1 contradicts its callers' docs. OutcomesWithTimeout (line 154) and RunOnceWithTimeout (line 182) say a single timeout is never accepted as final and point to "that constant's own doc comment", which does not exist. With 1, the first timeout is final. Either raise it or fix both comments.

claude · 2026-10-14T20:37:04Z

+
+        // Absolute construct-coverage check against constructs.txt: every construct except Tracing
+        // must be covered.
+        if (File.Exists(constructsPath))


FYI: F13. The construct gate that --coverage-report now enforces is silently skipped when constructsPath is missing. A mistyped --constructs or a renamed conformance/constructs.txt prints nothing and exits 0 if dead rules are clean. Fail, or at least warn, when the checklist is absent.

claude · 2026-10-14T20:37:07Z

+        foreach (string rawLine in File.ReadAllLines(path))
+        {
+            string line = rawLine.Trim();
+            if (line.Length == 0 || line.StartsWith("#"))


FYI: F14. Ordinal comparison missing: line.StartsWith("#") uses the current culture for a comment marker, against the AGENTS.md ordinal rule. LoadDeadRuleAllowlist below already passes StringComparison.Ordinal; do the same here.

claude · 2026-10-14T20:37:10Z

+                continue;
+            }
+
+            // Read once, before the try: the old code re-read expected.tsv from inside the catch


FYI: F15. History comment: "the old code re-read expected.tsv" describes a previous version, which code-comments excludes; so does SIL.Machine.Morphology.HermitCrab.Tool/Program.cs:149 ("the Conformance harness's own copy did not ..."). State why the code is shaped this way now. Unverified whether --agent-strict flags them; I could not run it.

claude · 2026-10-15T00:56:08Z

Approve with fixes. Third review round, covering cc947db..3a63132: two commits that pin UTF-8 on the counterfactual mutant pipe and regenerate the ledgers. The delta has no new findings, and no open finding's code changed.

Verdict: approve with fixes. Nothing open blocks, but four Important findings still need an answer from the author.
Most important: F10, CounterfactualGate.cs:438. An enum surface whose siblings all time out is still recorded as Unobservable, which is a false "no delta" claim. This round's fix stops wrong Evidenced verdicts; F10 is the opposite error, a wrong Unobservable.
Counts: 0 Critical, 4 Important open (F2, F3, F4, F10), 10 Low open (F5 to F9, F11 to F15). F1 was addressed earlier. Nothing new.
Ran: dotnet build -c Release succeeded with 0 warnings and 0 errors. dotnet test tests/SIL.Machine.Morphology.HermitCrab.Tests --no-build -c Release passed 608 and failed 0, including the new ANonAsciiSurfaceFormSurvivesTheMutantChildProcess. I also ran CounterfactualGateTests under LC_ALL=en_US.ISO-8859-1, a non-UTF-8 console encoding, and all 14 passed. I reviewed the 8-line source delta and the regenerated rows by hand, not with /code-review. The rows match the commit message: Gloss is dropped from the evidence ledger, HeadFeatures moves to RequiredByDtd, and the ratchet goes from 25 to 26.
Could not verify: the sandbox still needs approval for ./local_check.sh --agent-strict and dotnet run on the conformance CLI. So the strict comment scan (F15), the local_check.sh conformance gates, and a fresh --counterfactual recompute against the regenerated ledgers are all unrun. I also did not show that the new test fails without the fix. On Linux the default console encoding is already UTF-8, so the test can only catch a regression on ci.yml's windows-latest leg. The adapter pipe (AdapterEngine.cs:44) does not pin an encoding either. That code is unchanged here, and I have not checked whether its output can carry non-ASCII.

Contracts: this round changes no public API, target framework or package. sillsdev/machine.py parity: none verified.

Finding	Severity	Status
F1	Critical	addressed (`cc947db`)
F2	Important	open
F3	Important	open
F4	Important	open, unverified
F5	Low	open, unverified
F6	Low	open
F7	Low	open
F8	Low	open
F9	Low	open
F10	Important	open
F11	Low	open, unverified
F12	Low	open
F13	Low	open
F14	Low	open
F15	Low	open, unverified

Reviewed at 3a63132

Introduces an engine-agnostic conformance suite for HermitCrab-formalism morphological parsers: a `conformance/` tree of language grammars and edge-case micro-grammars, each an HC-XML grammar.xml paired with a words.yaml ground truth (oracle-provenance, gloss, and per-word parse signatures). A new SIL.Machine.Morphology.HermitCrab.Conformance project and hc-conformance CLI tool run those fixtures against an engine adapter, producing coverage and rules CSVs, an interface inventory, dataflow- obligation ledgers, evidence cards, a FieldWorks-producibility table, and a JSON manifest for external consumers. PROTOCOL.md is the adapter contract a second engine (e.g. the in-progress Rust port) implements to plug into the same suite. Includes the FieldWorks-producibility classification campaign: for every HermitCrab construct the suite covers, a verdict on whether FieldWorks' own HC-XML export path can actually produce a grammar exercising it, distinguishing genuine data-model limits from mere loader gaps, with five previously-unresolved verdicts now classified and corrected write-ups for two more cases initially misread as exporter silence. Also lands one new edge-case fixture, edge-cases/chained-output-feature-override-loss, isolating an analysis- side OutputHeadFeatures retraction bug: a well-formed four-morph parse of `zudiua` that the founding C# oracle currently loses. The fixture is deliberately red against hc.dll today (expected, not a broken test) and documents the mechanism, provenance, and the HC-Rust divergence that motivated it. The suite is deliberately stopped at this point rather than finished: scope, known gaps, and next steps are documented in the suite's own tracking docs rather than left implicit. This commit squashes the full conformance-suite branch stack (integrate-conformance-framework through conformance/fieldworks- witnesses, 49 commits) into one commit on top of current master, for PR #480. Folds in three review fixes: AdapterEngine's contract in MaterializedRunner now credits an expect_crash fixture only once the same engine also completes an ordinary fixture from the same run, so a misconfigured adapter (bad path, rejected arguments) can no longer register as reproducing the pinned crash; GuesserSignatureTests' batch-path test is renamed for the overload it actually calls; and .gitattributes freezes project.fwdata as binary so its pinned SHA-256 is reproducible across platforms and CI runners, not just this one. Also folds in the follow-up PR commits: fuzz/break-attempt evidence; admitting PCRE.NET's source generator via a vetted content-hash allowlist; final-template and template-sharing controls; oracle-verified fixtures for per-piece environments, simultaneous mode and iterative epenthesis; reduplication x phonology cases; and a regenerated manifest, ledgers and evidence cards. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ce suite The suite at the integrate-conformance-framework head was blind to #539: base, the PR head and a fixed variant gave identical results on every fixture and test, while on real grammars the PR head hangs on a simultaneous expansion rule and loses parses behind narrowing (merge) rules. Add fourteen synthetic edge-case fixtures that close that gap, each verified against the C# founding oracle at this commit and shown to fail (or time out) on the #539 head: - simultaneous-expansion-terminates (#539 head hangs), feature-rule-before-merge, epenthesis-fed-merge, geminate-under-merge and default-symbol-after-merge (engine-only: SymbolicFeature.defaultSymbol), ported from the PanGloss conformance-staging set. - boundary-first-in-narrowing-lhs, boundary-first-in-narrowing-lhs-strrep, boundary-leftover-in-narrowing-lhs-strrep, boundary-mid-in-narrowing-lhs-strrep, deletion-reinserted-feature-unapply, expansion-overlapping-targets, merge-subrules-pos-rtl-template, merge-then-metathesis and two-merges-cascade-env, from the #539 regression hunt. The two zero-feature boundary fixtures also fail on the no-dirty-marks variant of the PR. Regenerate every ledger the gates check (manifest, coverage.csv, rules.csv, interface inventory, interaction chains, rule-interaction pairs, counterfactuals, evidence, the traceability bundle and evidence cards) and move the measured-count pins in the ledger tests to the new corpus. Add the fourteen rows to fixtures.csv and correct the README fixture count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The suite added Trace.FailureAllomorph so FailureRuleAttributor could credit an affix rule whose allomorph failed an environment or co-occurrence check. That is an engine API change riding in a test-suite PR, and #534 adds the same fact as Trace.Allomorph. Drop it here; the attribution returns with #534. No generated ledger changes: coverage.csv and rules.csv regenerate byte-identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

--coverage-report printed dead rules and uncovered constructs but always exited 0, so the CI step that ran it could not fail. It now exits 1 when an in-scope construct has no coverage, when a rule no word exercises is missing from conformance/dead-rules-allowed.tsv, or when an allowed rule is exercised again. The allowlist holds today's 15 dead rules, each with its reason: inactive decoys, refused controls, and rules in crash-pin fixtures. The Conformance workflow also runs when SIL.Machine core or the HermitCrab tests change, since both can change parse results. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The suite added two steps to ci.yml: a fixture run, which ConformanceFixtureGateTests already performs under dotnet test, and parity-check.py, whose one remaining check the coverage report now enforces in the Conformance workflow. It also pinned SDK 10.0.303, which global.json's latestFeature roll-forward overrides. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

chained-output-feature-override-loss was written red: its header says the oracle loses zudiua today and that the loss survives PriorityUnion. Both are stale. Running the fixture against master shows zudiua red at 52d069f and green from 7109531 (#494) onward; the earlier note measured the pre-merge research branch. The header now describes the fixture as the regression pin it has become. Manifest regenerated for the new file hash. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

local_check.sh ran `--semantic-coverage`, whose CLI exits 1 while the semantic catalog is an incomplete backlog -- the state the suite expects, and gates under dotnet test instead -- so the script always failed after the tests passed. It now runs what conformance.yml runs: self-check with pathological fixtures, the coverage report, and the CSV freshness check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A counterfactual mutant is parsed in a child process and its outcomes come back through stdout, while the baseline it is compared with is computed in-process. Neither end pinned an encoding, so on a non-UTF-8 console code page a surface form such as "katibɯd" came back as "katib?d", and "∅" as "φ". Every mutant of a fixture with non-ASCII output then looked like it changed a parse, and its surface counted as Evidenced. The checked-in ledger has two such verdicts, Gloss and HeadFeatures; Gloss is the surface the suite's own tests describe as never read by the parser. The child now writes UTF-8 and the parent reads UTF-8. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

With the encoding fixed, the counterfactual sweep agrees with the checked-in ledger on every verdict but two: Gloss is Unobservable (no word's parse reads it) and HeadFeatures is RequiredByDtd. Both had been recorded as Evidenced from mis-encoded output. Every other changed row keeps its verdict and only loses the garbled delta text. The ratchet's pinned gap count goes from 25 to 26 for Gloss, a gap that was always there and is now counted. The ledger-shape pins in FoldInCandidateLedgerTests and GrammarCoverageLedgerTests move with the regenerated files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

#529 added isCliticRule to MorphologicalRule and RealizationalRule in the library DTD, and let a vacuous final template in a later stratum complete a non-final template from an earlier one. - Sync conformance/HermitCrabInput.dtd with the library copy, and add ConformanceDtdTests so the two cannot drift again: the coverage inventory is read from the conformance copy, so a missed sync hides new surfaces. - Add edge-cases/clitic-rule-designation: isCliticRule true and false on MorphologicalRule, the "Clitics" stratum default, stacked clitics, and the cross-stratum completion of a non-final template (kolpu). - Keep morphotactic-attribute-breadth's kulpu a same-stratum negative control: the Second stratum's material moves to its own part of speech, so no later final template applies to kul. - Map the ten new DTD surfaces in semantic-catalog.yaml. The two RealizationalRule/isCliticRule values stay todo in the baseline: the engine never reads them, and FieldWorks does not use realizational rules. - Point conformance/docs/README.md at PanGloss's xample-projector, which produces the .fwdata witnesses and runs the XAMPLE comparison. - Regenerate the ledgers and re-pin the measured counts. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

claude · 2026-10-15T21:38:46Z

+    private static string NormalizeLineEndings(string text) => text.Replace("\r\n", "\n", StringComparison.Ordinal);
+
+    [Test]
+    public void ConformanceDtdMatchesTheDtdTheLoaderEmbeds()


FYI: F16. This test duplicates ConformanceManifestTests.PublishedDtdMatchesTheLibraryResource (ConformanceManifestTests.cs:81), which already compares the same two DTDs after line-ending normalization and has been in the PR since 2306cbe. If they drift, both fail, with different fix instructions. Drop this one, or move its "regenerate the coverage ledgers" hint into the existing assertion message.

A counterfactual sweep child reached 6 GB, and the parallel ledger runs exhausted the machine's memory. A heap limit makes the collector work instead of the heap growing. The cap is set on each mutant child only: the parent sweep stays near 75 MB, and the Roslyn-based catalog audit needs more than 2 GB of live data. An inherited DOTNET_GCHeapHardLimit wins. Regenerating the counterfactual ledger under the cap reproduces it byte for byte. A cap must not create evidence. CounterfactualGate.Outcome recorded any exception as a word's outcome, so a mutant that ran out of memory differed from its baseline and counted as Evidenced. An OutOfMemoryException now ends the child with its own exit code, which the gate reports as a Timeout: a resource limit, like a wall-clock one, is not a semantic delta. AMutantThatRunsOutOfMemoryIsATimeoutNotEvidence fails without this (Evidenced) and passes with it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

claude · 2026-10-16T01:48:40Z

+                        continue;
+                    index.AllRuleIds.Add(id);
+                    if (name != null)
+                        index._ruleIdByRuleName[name] = id;


FYI: F17. A duplicate rule key is silently overwritten here, as it is at lines 72 and 78. Two PhonologicalRules named "Epenthesis", or two MorphologicalRules with the same MorphemeId, both resolve to the last id. So rules: checks and the dead-rule report credit the wrong rule, with no diagnostic. Throw on a duplicate key instead. I could not scan the fixtures for an existing collision because the sandbox blocked it.

claude · 2026-10-16T01:48:41Z

+if [ $? -ne 0 ]; then
+  exit 1
+fi
+git diff --exit-code -- conformance/coverage.csv conformance/rules.csv


FYI: F18. Unverified: this freshness gate depends on line endings. CoverageReport.WriteCsv writes with AppendLine (CRLF on Windows), and .gitattributes has no rule for *.csv. On Windows with core.autocrlf=false, an unmodified tree would therefore show every line as changed. Writing \n explicitly, or adding conformance/*.csv eol=lf, would make it hold on any platform.

johnml1135 requested a review from ddaspit August 19, 2026 19:27

johnml1135 force-pushed the integrate-conformance-framework branch from d826869 to 8f9dcea Compare August 19, 2026 19:46

johnml1135 force-pushed the integrate-conformance-framework branch 2 times, most recently from d0b5448 to 9dfb2f6 Compare August 20, 2026 09:10

johnml1135 mentioned this pull request Aug 20, 2026

Add HermitCrab grammar health diagnostics #475

Closed

johnml1135 force-pushed the integrate-conformance-framework branch 2 times, most recently from de95458 to 4823a05 Compare September 4, 2026 00:15

johnml1135 force-pushed the integrate-conformance-framework branch from 4823a05 to 3128301 Compare September 15, 2026 12:29

johnml1135 mentioned this pull request Sep 15, 2026

HermitCrab: track exact analysis feature inversion and chained override parse loss #504

Open

This was referenced Sep 15, 2026

HermitCrab: establish discriminating conformance coverage for analysis-state and template/slot merges #505

Open

HermitCrab: complete cross-engine correctness controls for final-template pruning #507

Open

This was referenced Sep 24, 2026

Prune reduplication matches whose copies cannot agree #519

Open

POS-scoped deletion blocks metathesis analysis of other words (ParseWord incomplete) #520

Open

johnml1135 force-pushed the integrate-conformance-framework branch from 93f470c to 2933fcc Compare September 26, 2026 02:55

johnml1135 force-pushed the integrate-conformance-framework branch from 2933fcc to f412c25 Compare September 26, 2026 04:15

johnml1135 force-pushed the integrate-conformance-framework branch from f412c25 to d8ff628 Compare September 30, 2026 20:29

claude Bot reviewed Oct 10, 2026

View reviewed changes

Comment thread local_check.sh

claude Bot reviewed Oct 10, 2026

View reviewed changes

claude Bot reviewed Oct 14, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

This was referenced Oct 14, 2026

Trace HermitCrab failure reasons and template slot outcomes #534

Open

Remove HermitCrab's analysis-cascade memo #522

Open

jtmaxwell3 added the Hermit Crab label Oct 14, 2026

This was referenced Oct 14, 2026

Use IsCliticRule to allow clitics to apply to templates #529

Merged

Pin that a required template slot with no rules is skipped (question) #542

Draft

This was referenced Oct 15, 2026

Fix narrowing-rule unapplication across an interposed optional node #540

Open

Add conformance fixtures for the #504 and #507 evidence gaps #543

Closed

johnml1135 and others added 11 commits October 15, 2026 15:46

fix(hc): retain template trace failure reason

6e3d585

johnml1135 force-pushed the integrate-conformance-framework branch from 3a63132 to df2cf21 Compare October 15, 2026 21:21

claude Bot reviewed Oct 15, 2026

View reviewed changes

claude Bot reviewed Oct 16, 2026

View reviewed changes

johnml1135 removed this from the Hermit Crab Optimizations - Oct 2026 milestone Oct 16, 2026


		internal static readonly TimeSpan DefaultTimeout = TimeSpan.FromSeconds(180);

		private const int TimeoutConfirmationAttempts = 1;

Conversation

johnml1135 commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Quick summary

Where to look

Reading order

Stacked on this PR

Validation

Uh oh!

codecov-commenter commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

johnml1135 commented Sep 15, 2026

Uh oh!

johnml1135 commented Sep 15, 2026

Uh oh!

Uh oh!

claude Bot Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

claude Bot commented Oct 15, 2026

Uh oh!

claude Bot Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

johnml1135 commented Aug 19, 2026 •

edited

Loading

codecov-commenter commented Aug 20, 2026 •

edited

Loading