Each of the 16 coder outputs (8 designs × 2 coders) was parsed and checked by script, and the two coders were compared. The comparison measures reproducibility of the protocol; it does not show that the codes are valid, since both coders are LLMs and can share a bias.
Validation of the outputs
All 16 outputs parse as JSON without repair and cover every unit. Of 925 quotes, 910 are exact (after whitespace normalisation) and 15 are spliced: a sentence joined to a bullet item that is not adjacent in the text, as in “The analyst should be able to adjust: – Minimum edge weight”. Each fragment of a spliced quote is verbatim. No quote is missing from its unit, and no consistency rule of the protocol is violated.
| Design | Coder | Quotes | Exact | Spliced | Not found | Rule violations |
|---|---|---|---|---|---|---|
| D01 | Sonnet | 63 | 63 | 0 | 0 | 0 |
| D01 | Terra | 70 | 65 | 5 | 0 | 0 |
| D02 | Sonnet | 30 | 30 | 0 | 0 | 0 |
| D02 | Terra | 29 | 29 | 0 | 0 | 0 |
| D03 | Sonnet | 67 | 67 | 0 | 0 | 0 |
| D03 | Terra | 68 | 68 | 0 | 0 | 0 |
| D04 | Sonnet | 55 | 55 | 0 | 0 | 0 |
| D04 | Terra | 67 | 67 | 0 | 0 | 0 |
| D05 | Sonnet | 71 | 71 | 0 | 0 | 0 |
| D05 | Terra | 45 | 45 | 0 | 0 | 0 |
| D06 | Sonnet | 67 | 67 | 0 | 0 | 0 |
| D06 | Terra | 68 | 58 | 10 | 0 | 0 |
| D07 | Sonnet | 59 | 59 | 0 | 0 | 0 |
| D07 | Terra | 47 | 47 | 0 | 0 | 0 |
| D08 | Sonnet | 58 | 58 | 0 | 0 | 0 |
| D08 | Terra | 61 | 61 | 0 | 0 | 0 |
The checks are in scripts/analyse_coding.py; per-file results in data/validation_by_file.csv.
Agreement between the coders
Over the 65 units of the eight designs the coders agree on 60/65 (92%) for a stated human action, 62/65 (95%) for a stated machine action, 58/65 (89%) for a named output and 56/65 (86%) for the presence of a control act. For items, control-act F1 is 0.71 (0.72 when judgement and decision acts are merged), parameter F1 is 0.86 and branch F1 is 0.71. Sonnet found 9 loops and Terra 8.
Agreement per design
| Design | Units | Human | Machine | Output | Any act | Acts S / T | Acts F1 | Params S / T | Params F1 | Branches S / T | Branches F1 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| D01 | 9 | 9/9 | 9/9 | 9/9 | 9/9 | 17 / 21 | 0.74 | 6 / 7 | 0.77 | 7 / 7 | 0.86 |
| D02 | 7 | 7/7 | 7/7 | 7/7 | 7/7 | 5 / 5 | 1.00 | 1 / 0 | 0.00 | 1 / 1 | 1.00 |
| D03 | 6 | 5/6 | 5/6 | 5/6 | 5/6 | 19 / 19 | 0.84 | 5 / 5 | 1.00 | 5 / 6 | 0.55 |
| D04 | 8 | 8/8 | 7/8 | 8/8 | 8/8 | 11 / 17 | 0.64 | 5 / 5 | 0.60 | 3 / 1 | 0.50 |
| D05 | 7 | 6/7 | 6/7 | 5/7 | 5/7 | 16 / 8 | 0.67 | 5 / 5 | 1.00 | 11 / 8 | 0.74 |
| D06 | 10 | 7/10 | 10/10 | 10/10 | 7/10 | 21 / 19 | 0.65 | 11 / 10 | 0.86 | 2 / 5 | 0.57 |
| D07 | 8 | 8/8 | 8/8 | 5/8 | 6/8 | 16 / 11 | 0.67 | 3 / 3 | 1.00 | 3 / 3 | 1.00 |
| D08 | 10 | 10/10 | 10/10 | 9/10 | 9/10 | 12 / 10 | 0.64 | 3 / 3 | 1.00 | 9 / 13 | 0.64 |
Human, Machine, Output and Any act: units on which the coders agree. S = Sonnet, T = Terra. F1 matches control acts by unit and type, parameters by name overlap within the design, and branches by unit and type.
Agreement on whole-design measures (8 designs)
| Measure | Pearson r | Spearman rank correlation | Mean, Sonnet | Mean, Terra |
|---|---|---|---|---|
| Units with a human action (share) | 0.80 | 0.73 | 0.85 | 0.77 |
| Units with a machine action (share) | 0.91 | 0.75 | 0.86 | 0.81 |
| Units with a control act (share) | 0.86 | 0.83 | 0.83 | 0.70 |
| Units naming an output (share) | 0.95 | 0.98 | 0.54 | 0.43 |
| Judgement and decision acts per unit | 0.78 | 0.29 | 1.06 | 1.11 |
| Set acts per unit | 0.68 | 0.65 | 0.37 | 0.31 |
| Loops | 0.83 | 0.82 | 1.12 | 1.00 |
| Named tunable parameters | 0.98 | 1.00 | 4.88 | 4.75 |
| Branches | 0.82 | 0.87 | 5.12 | 5.50 |
Judgement and decision acts per unit depend on how finely the coders split acts, so their rank agreement is low and they are not used for conclusions. There is also a constant offset: Sonnet reports more units with a control act and more named outputs than Terra, so the conditions are compared within each coder.
Where the coders differ
- Granularity. One coder keeps a line such as “Check: A; B; C” as one act, the other codes one act per object. The same holds for lists of parameters and of alternatives.
- Conditionals. Data-availability conditions (“if available”) are counted as branches by one coder in some designs and not by the other; branch counts are the noisiest item count.
- Passives and nouns. A passive without an agent (“should be checked against”) or a noun phrase for an action is coded as a human action by one coder in some designs, which the protocol rules out for unlisted verbs.
- Table cells. The rule that every bullet and table cell is read as a sentence is applied unevenly by both coders.
Development of the protocol
- Version 1.0: five criteria (who acts, what the human controls, what loops refine, named parameters, evidence and alternatives) and the JSON output with verbatim quotes.
- Version 1.1: named outputs and provenance statements. Version 1.2: several non-step blocks per design.
- Versions 1.3 and 1.4: rule clarifications after a pilot on D01 (spelling variants of verbs, imperatives name no agent, table cells and bullets read as sentences, decisions need a named human agent, one record per parameter and per conditional, cue list for branches).
- Version 1.5: a wider list of machine verbs (data-processing and map-building verbs), after the pilot on D02 showed that machine actions were under-detected. The unlisted verb rule and the self-check were tightened.
- D01 and D02 were used as pilots under versions 1.2 to 1.4 and were re-run under version 1.5; only the version 1.5 runs are used. D03 to D08 were coded once, under version 1.5. The protocol was not changed after that.