The prose designs of C1 (task only) and C4 (library in prose form) have no ATWL, so features cannot be computed from them. The aspects that an ATWL description makes explicit were therefore coded from the text, under a fixed protocol, by two independent LLM coders, and then compared with measures computed from the ATWL designs of the other conditions. The aspects are: who acts (human or machine), what the human controls, what is repeated and refined, which adjustable parameters are named, which evidence, criteria and alternatives are stated, and which outputs are named.
The coding is not a human judgement. Agreement between two LLM coders shows that the protocol can be applied reproducibly; it does not show that the codes are valid. Quality of the designs is not assessed.
Procedure
- Input. Eight designs (C1 and C4, two tasks, two models). Each is the final answer of the design turn of the corresponding chat, with the model's reasoning block and closing offers removed, split into units at the author's own numbered or lettered steps, and given a neutral id (D01–D08). The coders saw no condition, model, task or study information.
- Coders. Claude Sonnet 5 and GPT-5.6-Terra, each design in its own fresh, independent session, so that each instance saw one design only; the sessions were run in a different random order for each coder. The instructions are the coder prompt (protocol version 1.5).
- Output. One JSON object per design with, for every unit, the stated human and machine actions, the human control acts (set, select, judge, decide, annotate), and whether an output is named; and for the whole design its loops, named parameters, branches and provenance statements. Every code carries a verbatim quote from the text.
- Checks. Every output is parsed, every quote is looked up in the unit it comes from, and the consistency rules of the protocol are checked by script. The two coders are compared on units, items and whole-design measures (reliability).
- Comparison. Conditions are compared within blocks (task × model), separately for each coder, and against the ATWL designs (comparisons).
Designs and coder outputs
| Design | Task | Condition | Model | Units | Words | Text given to the coders | Coder outputs (JSON) |
|---|---|---|---|---|---|---|---|
| D01 | VIS | C4 | Terra | 9 | 1804 | text | Sonnet · Terra |
| D02 | VAST | C1 | Sonnet | 7 | 495 | text | Sonnet · Terra |
| D03 | VAST | C4 | Terra | 6 | 1622 | text | Sonnet · Terra |
| D04 | VIS | C1 | Sonnet | 8 | 745 | text | Sonnet · Terra |
| D05 | VAST | C4 | Sonnet | 7 | 1895 | text | Sonnet · Terra |
| D06 | VIS | C1 | Terra | 10 | 2132 | text | Sonnet · Terra |
| D07 | VIS | C4 | Sonnet | 8 | 1576 | text | Sonnet · Terra |
| D08 | VAST | C1 | Terra | 10 | 1660 | text | Sonnet · Terra |
The condition and model of a design were not known to the coders. The mapping is in data/mapping.csv (with the run order of each coder). The outputs of D05 and D07 from Sonnet arrived wrapped in a code fence, which was removed; nothing else in the outputs was changed.
Files
Comparisons
C4 against C1, model and task, ATWL conditions, the 2×2, the lineage from C4 to C6, C6++ and C7, and the explicitness of prose and ATWL. Also as Word document.
Reliability of the coding
Validation of the outputs, agreement between the coders and the development of the protocol.
Coder prompt (version 1.5)
The instructions given to each coder, with a worked example. Researcher notes (not given to the coders) and the segmentation log.
Data and script
Prose measures, ATWL measures, validation, agreement, all result tables. scripts/analyse_coding.py reproduces them from the designs, outputs and ATWL files of this site.