Comparisons of conditions that the coding of the prose designs makes possible, in the order of the questions they answer. Prose measures come from the two coders; ATWL measures are computed from the 20 ATWL designs. Also available as a Word document.
Summary
- Library in prose form (C4 against C1). Library designs state more iterations (1.4 against 0.75 loops per design), more annotation and logging acts (2.8 against 1.1) and name their outputs more often (share of units 0.61 against 0.36). Only C4 designs contain provenance statements. For loops, annotation and provenance, both coders find no block with the opposite direction.
- What the library does not change: the stated split of labour between human and machine, the number of named tunable parameters, and how often a judgement states what is looked for. No prose design, in either condition, states an exit condition for a loop.
- Model and task matter as much as the library. Terra designs name more tunable parameters (6.3 against 3.4) and have more units with a control act. VAST designs state more branches than VIS designs (6.5 to 7.0 against 3.8 to 4.0). With eight designs these effects cannot be separated from each other for some measures.
- ATWL designs commit to more iteration than prose designs: 1.75 to 4.25 loops per design against 0.75 (C1) and 1.4 (C4). In the 2×2 on loops, ATWL alone (C2 against C1), the library in prose form and the library in ATWL form each add loops in three of four blocks, and none reverses. The library in ATWL form (C3 against C2) also adds 4.0 human judgement and decision transforms, in all four blocks.
- Translation (C4 to C6) makes more of the iteration visible: 2.25 loops per design against 1.4, and every ATWL loop declares an exit condition (55 of 55), where none of the 17 coded prose loop records states one.
- Translation also drops explicit branching. C4 prose designs state 4.1 conditional paths per design on average; the C6 designs contain none outside loops, and C6++ and C7 contain one each.
- Limits: four blocks, so every result is descriptive; cross-format comparisons rest on a mapping between two instruments; quality is not assessed.
Approach
Prose side. Eight designs (C1 and C4, two tasks, two models) coded by two LLM coders, Claude Sonnet 5 and GPT-5.6-Terra, with protocol version 1.5 (coding, reliability). Where a number is shown as a pair, the first value is Sonnet and the second Terra.
ATWL side. Twenty designs (C2, C3, C6, C6++, C7) measured by script from the ATWL text (scripts/analyse_coding.py). The parser reproduces the transform counts, loop counts and human and hybrid shares of the workbook in the ATWL-side analysis for all twenty designs.
Unit of comparison. A block is one task and model (four blocks), each with one design per condition, so differences are taken within blocks. The tables give the mean within-block difference and the number of blocks in which the second condition is higher or lower. With four blocks the smallest possible two-sided exact p-value is 0.125, so no significance test is reported.
Mapping between the two instruments
| Concept | Prose (coded) | ATWL (computed) |
|---|---|---|
| Loops | Loops stated by the text | Loop constructs |
| Human judgement points | Judgement and decision acts | Transforms with intent assess, abstract or generate-knowledge and actor human or hybrid |
| Tunable parameters | Named parameters stated as tunable | Specification artifacts revised by an assign inside a loop |
| Conditional branches | Branches stated (all types) or conditional paths | if blocks outside loops |
| Stated actor, output, exit | Coded per unit or loop | Declared by syntax (actor, output; until for loops) |
Loops are the most comparable concept. Counts of judgement points and parameters depend on how finely each format splits steps, so they are comparable only roughly. Judgement and decision acts of the prose designs are unreliable between the two coders (see the reliability page) and are not used for conclusions; figures based on them are marked †.
Results
Table 1. Library in prose form: C4 against C1 (prose designs, both coders)
| Measure | C1 (Sonnet / Terra) | C4 (Sonnet / Terra) | Mean difference | Blocks higher/lower | Reading |
|---|---|---|---|---|---|
| Loops stated | 0.75 / 0.75 | 1.50 / 1.25 | +0.75 / +0.50 | 3/0 ; 2/0 | consistent increase |
| Annotation / logging acts | 1.50 / 0.75 | 3.00 / 2.50 | +1.50 / +1.75 | 3/0 ; 4/0 | consistent increase |
| Provenance statements | 0.00 / 0.00 | 0.50 / 0.75 | +0.50 / +0.75 | 2/0 ; 3/0 | increase (none in C1) |
| Units naming an output (share) | 0.37 / 0.35 | 0.71 / 0.51 | +0.34 / +0.16 | 3/1 ; 2/1 | increase |
| Branches stated | 3.75 / 5.00 | 6.50 / 6.00 | +2.75 / +1.00 | 2/1 ; 3/1 | higher, but mixed by block |
| Judgement / decision acts † | 6.75 / 9.50 | 9.50 / 7.75 | +2.75 / -1.75 | 3/1 ; 1/3 | unstable (coders differ in sign); not used |
| Named tunable parameters | 5.00 / 4.50 | 4.75 / 5.00 | -0.25 / +0.50 | 2/2 ; 2/2 | no clear effect |
| Units with a control act (share) | 0.83 / 0.73 | 0.84 / 0.66 | +0.01 / -0.06 | 2/1 ; 2/2 | no clear effect |
| Units with human action (share) | 0.85 / 0.78 | 0.84 / 0.76 | -0.01 / -0.02 | 2/1 ; 2/1 | no clear effect |
| Units with machine action (share) | 0.80 / 0.77 | 0.92 / 0.84 | +0.11 / +0.07 | 2/1 ; 3/1 | small increase, mixed |
| Judgements with criterion stated (share) | 0.44 / 0.48 | 0.40 / 0.53 | -0.04 / +0.05 | 2/2 ; 2/2 | no clear effect |
| Loops stating an exit | 0.00 / 0.00 | 0.00 / 0.00 | +0.00 / +0.00 | 0/0 ; 0/0 | none in either condition |
Means of four blocks. Blocks higher/lower counts blocks in which C4 is higher or lower than C1, for Sonnet and then Terra. Taking the lower or the higher of the two coders for each design leaves every direction unchanged: loops +0.50 to +0.75, annotation +1.25 to +2.00, provenance +0.50 to +0.75, named outputs +0.16 to +0.34. † Judgement and decision acts are unreliable between the two coders (rank agreement is low; see the reliability page) and are not used for conclusions; the row is shown for completeness.
Table 2. Model and task contrasts on the prose designs (C1 and C4 pooled; Sonnet / Terra coder)
| Measure | Sonnet designs | Terra designs | VAST designs | VIS designs |
|---|---|---|---|---|
| Units per design | 7.50 / 7.50 | 8.75 / 8.75 | 7.50 / 7.50 | 8.75 / 8.75 |
| Units with a control act (share) | 0.77 / 0.63 | 0.90 / 0.76 | 0.79 / 0.65 | 0.88 / 0.74 |
| Units with machine action (share) | 0.93 / 0.86 | 0.79 / 0.75 | 0.83 / 0.75 | 0.89 / 0.86 |
| Named tunable parameters | 3.50 / 3.25 | 6.25 / 6.25 | 3.50 / 3.25 | 6.25 / 6.25 |
| Branches stated | 4.50 / 3.25 | 5.75 / 7.75 | 6.50 / 7.00 | 3.75 / 4.00 |
| Loops stated | 1.00 / 0.75 | 1.25 / 1.25 | 1.00 / 1.00 | 1.25 / 1.00 |
Four designs per column. The parameter figures are the same for model and task because the two contrasts are confounded here: the one design with 10 to 11 parameters (VIS, Terra, no library) belongs to both the Terra group and the VIS group.
Table 3. ATWL designs by condition (means of four blocks)
| Measure | C2 | C3 | C6 | C6++ | C7 |
|---|---|---|---|---|---|
| Transforms | 18.8 | 24.2 | 35.2 | 43.8 | 31.8 |
| Human share | 0.28 | 0.42 | 0.30 | 0.35 | 0.28 |
| Machine share | 0.59 | 0.54 | 0.60 | 0.58 | 0.54 |
| Hybrid share | 0.13 | 0.05 | 0.09 | 0.07 | 0.17 |
| Loops | 1.75 | 2.75 | 2.25 | 4.25 | 2.75 |
| Loops with an until | 1.75 | 2.75 | 2.25 | 4.25 | 2.75 |
| Specification artifacts | 3.75 | 6.25 | 6.00 | 14.25 | 5.25 |
| Specifications revised in a loop | 1.50 | 3.00 | 2.25 | 5.00 | 2.25 |
| Conditional blocks outside loops | 0.00 | 0.00 | 0.00 | 0.25 | 0.25 |
| Human/hybrid judgement transforms | 3.75 | 5.50 | 6.50 | 8.50 | 6.00 |
| Human/hybrid decision transforms | 2.00 | 4.25 | 4.50 | 6.75 | 4.50 |
C2: ATWL specification and one example. C3: adds the library in ATWL form. C6: faithful translation of the C4 design. C6++: attempted implementation of the C6 self-critique. C7: fresh ATWL redesign from the C4 design.
Table 4a. The 2×2 on concept-level counts (means of four blocks; prose designs are the mean of the two coders)
| Measure | C1 (prose) | C2 (ATWL) | C4 (prose) | C3 (ATWL) |
|---|---|---|---|---|
| Loops per design | 0.75 | 1.75 | 1.38 | 2.75 |
| Human judgement / decision points | 8.12† | 5.75 | 8.62† | 9.75 |
| Tunable parameters (prose) / specifications revised in loops (ATWL) | 3.38 | 1.50 | 3.38 | 3.00 |
| Conditional branches (prose) / conditional blocks outside loops (ATWL) | 4.38 | 0.00 | 6.25 | 0.00 |
Table 4b. Within-block contrasts: mean difference (blocks higher/lower)
| Measure | ATWL, no library (C2−C1) | Library, prose (C4−C1) | Library in ATWL form (C3−C2) | ATWL + library vs prose library (C3−C4) | Interaction |
|---|---|---|---|---|---|
| Loops per design | +1.00 (3/0) | +0.62 (3/0) | +1.00 (3/0) | +1.38 (3/0) | +0.38 (2/1) |
| Human judgement / decision points | -2.38 (1/3)† | +0.50 (3/1)† | +4.00 (4/0) | +1.12 (2/2)† | +3.50 (3/1)† |
| Tunable parameters (prose) / specifications revised in loops (ATWL) | -1.88 (2/2) | +0.00 (2/1) | +1.50 (3/0) | -0.38 (2/2) | +1.50 (2/2) |
| Conditional branches (prose) / conditional blocks outside loops (ATWL) | -4.38 (0/4) | +1.88 (3/1) | +0.00 (0/0) | -6.25 (0/4) | -1.88 (1/3) |
Interaction is (C3−C2) − (C4−C1). Contrasts that cross formats (C2−C1, C3−C4, interaction) mix a change of format with a change of instrument and must be read with the mapping in mind. † Figures based on the judgement and decision acts of prose designs (Table 4a: C1 and C4; Table 4b: every contrast that includes a prose design) are unreliable between the coders and are not used for conclusions. The contrast between the ATWL designs C3 and C2 (+4.00) does not depend on the coding.
Table 5. Lineage: the C4 prose design and the ATWL designs of the same block
| Measure | Block | C4 prose | C6 | C6++ | C7 |
|---|---|---|---|---|---|
| Loops | VAST / Sonnet | 1.0 | 2 | 5 | 3 |
| Loops | VAST / Terra | 2.0 | 2 | 3 | 2 |
| Loops | VIS / Sonnet | 1.5 | 2 | 5 | 3 |
| Loops | VIS / Terra | 1.0 | 3 | 4 | 3 |
| Human judgement / decision points | VAST / Sonnet | 5.5† | 12 | 18 | 20 |
| Human judgement / decision points | VAST / Terra | 15.0† | 11 | 14 | 6 |
| Human judgement / decision points | VIS / Sonnet | 5.0† | 12 | 18 | 9 |
| Human judgement / decision points | VIS / Terra | 9.0† | 9 | 11 | 7 |
| Conditional paths | VAST / Sonnet | 7.0 | 0 | 0 | 1 |
| Conditional paths | VAST / Terra | 3.0 | 0 | 1 | 0 |
| Conditional paths | VIS / Sonnet | 1.0 | 0 | 0 | 0 |
| Conditional paths | VIS / Terra | 5.5 | 0 | 0 | 0 |
| Tunable parameters / specs revised in loops | VAST / Sonnet | 3.5 | 2 | 5 | 3 |
| Tunable parameters / specs revised in loops | VAST / Terra | 1.5 | 2 | 4 | 2 |
| Tunable parameters / specs revised in loops | VIS / Sonnet | 2.5 | 2 | 4 | 2 |
| Tunable parameters / specs revised in loops | VIS / Terra | 6.0 | 3 | 7 | 2 |
C6 is a faithful translation of the C4 design; C6++ is a revision of C6 that keeps most of its artifacts and transforms; C7 is a fresh redesign. C4 values are the mean of the two coders. † The C4 values for judgement and decision points are unreliable between the coders and are not used for conclusions; the C6, C6++ and C7 values are computed from the ATWL text.
Table 6. Explicitness: what prose states and what ATWL declares by syntax
| Measure | C1 prose | C4 prose | ATWL (20 designs) |
|---|---|---|---|
| Units with a stated actor | 0.88 | 1.00 | 1.00 (declared by syntax) |
| Units naming their output | 0.36 | 0.61 | 1.00 (declared by syntax) |
| Loops stating an exit condition | 0 of 6 | 0 of 11 | 55 of 55 declare an until |
| Loops stating what triggers another round | 3 of 6 | 4 of 11 | not coded in ATWL |
| Parameters with an initial value stated | 11 of 38 | 9 of 39 | not comparable |
| Judgements with a criterion stated | 0.46 | 0.47 | no field in ATWL |
Prose figures pool both coders (loops and parameters are counted once per coder). Shares are means over designs and coders.
Reading the results
The library, in prose form, changes what a design commits to in three ways that both coders see: it adds iterations, it adds records of what was done (annotation and logging acts, provenance statements), and it makes outputs more explicit. It does not change how labour is split or how many parameters are named. The additions are of the kind the library workflows contain, for example provenance logging and refinement loops, but this analysis cannot show where each addition comes from.
ATWL adds to iteration and fixes, by syntax, things prose leaves open: an actor, an output and an exit for every loop. Prose designs state exits for none of their loops and name their outputs in 36 to 61 percent of their units. The translation also shows a cost: conditional paths that prose states are not carried into the ATWL. The formal language makes iteration explicit, while the conditional paths that prose states drop out of the ATWL designs.
The library in ATWL form (C3) differs from the language alone (C2) in loops (+1.0), human judgement transforms (+4.0, all four blocks) and revised specifications (+1.5). Part of this is design size, since C3 designs are larger than C2 designs (24 against 19 transforms), but per ten transforms the judgement density is also highest in C3 (4.0 against 3.1).
Limits
- Four blocks and one design per cell. All results are descriptive, and run-to-run variation of the models is not measured.
- Cross-format comparisons depend on the mapping. ATWL splits work into many small transforms, prose into a few stages; ATWL designs are larger (19 to 44 transforms against 6 to 10 units).
- An exit condition declared by syntax is not necessarily precise; the content of the until clauses was not evaluated.
- The coders differ in level (Sonnet reports more units with a control act and more named outputs), so comparisons are made within each coder.
- Judgement and decision act counts are unreliable between coders and are not used for conclusions. Branch counts are the noisiest item count.
- These comparisons describe commitments in the designs, not their quality. Library use measured through citations and methods (C3 against C4, C4 against C5) is covered by the comparison tables.