ATWL-side analysis of the designed workflows

Summary

• Validity. 7 of the 20 designs have errors reported by the ATWL syntax checker (library: 0 of 54): 0 of 4 in C2, 0 of 4 in C3, 2 of 4 in C6, 2 of 4 in C6++ and 3 of 4 in C7; 6 of the 10 Terra designs and 1 of the 10 Sonnet designs. All are dangling references (98) or duplicate artifacts (2), mostly assign targets that are never declared.
• Against the library. The designs have on average 10.4 of the 25 features present, the library 7.4. 10 features differ at q < 0.05 (Fisher exact test, Benjamini–Hochberg): more frequent in the designs are E05, E10, I05, T01, T02, T04, H01 and H03, less frequent I03 and I06. The median Jaccard similarity of the features of a design to its most similar library workflow is 0.53, against 0.58 between library workflows, so the designs are not outliers.
• Conditions. Mean features present: C2 7.5, C3 9.75, C6 10.5, C6++ 12.75, C7 11.5. The overall test (randomisation of condition labels within blocks, 20,000 permutations) gives p = 0.11. With four blocks the result is descriptive; no single feature differs between conditions after correction.
• Factors. On the Jaccard distance of the feature profiles, model explains 15% (p = 0.0015), task 13% (p = 0.006) and condition 15% (p = 0.94). Model and task shape the profiles more than condition does.
• Exploratory contrast (post hoc). Designs made directly with the ATWL materials (C2 and C3, n = 8) against designs derived from the model's prose design (C6, C6++ and C7, n = 12): the derived designs have more transforms (36.9 against 21.5), more checker errors (7 of 12 against 0 of 8) and more features present (11.6 against 8.6). The grouping was defined after seeing the data (see the limits below).

The 25 features

Definitions and library support are in selected_features_2026-10-02.csv; the 54 × 25 matrix of the library is in selected_features_matrix_2026-10-02.csv. H03 is optional in the paper.

Designs by condition

Table 2. Means of the four blocks (task × model) per condition

C2: ATWL specification and one example. C3: adds the 54-workflow library in ATWL form. C6: faithful ATWL translation of the model's own C4 prose design. C6++: attempted implementation of the C6 self-critique. C7: fresh ATWL redesign from the same prose design. An undeclared assign target takes the category of the artifact assigned to it for the features (affects I01, I04, I05 and I07 in 8 designs).

Table 3. Paired contrasts between conditions (paired by block)

Single features that differ in three or more blocks: E01 (generate-knowledge -> define-unit) in C3 - C2; E08 (generate-knowledge -> build-model) in C3 - C2; E10 (assess -> define-unit) in C7 - C3; I07 (knowledge accumulated in a loop) in C6++ - C6. None is significant after correction.

Table 4. Model and task

Table 5. Factors on the feature profiles (PERMANOVA on the Jaccard distance)

Table 6. Exploratory contrast (post hoc): designed directly with ATWL materials against derived from the prose design

The within-block p-values randomise the group labels within the four blocks (exact, all 10⁴ assignments). The direction is the same in all four blocks. Within the derived group all 6 Terra designs have checker errors against 1 of 6 Sonnet designs; the 4 directly designed Terra designs have none.

Designs against the library

Table 7. Prevalence of each feature in the 54 library workflows and in the 20 designs

Shaded rows: q < 0.05 (Fisher exact test). Design counts are not independent of one another: designs share model and task, C6, C6++ and C7 share a prose design.

Features by condition

Limits

• The features measure what an ATWL description makes explicit, not the quality of a design. Quality needs expert ratings or implementation attempts.
• Feature counts depend on workflow size. In the library, size correlates with I02 (Spearman 0.54), T04 (0.50) and I05 (0.48). Designs derived from a prose design are longer, so their higher counts are confounded with size; no size adjustment was made.
• The designs are not independent: blocks share model and task, C6 and C7 share a prose design, and C6++ continues C6. Four blocks make every condition result descriptive. The exploratory contrast was defined after seeing the data, and its six characteristics are not corrected for multiple comparisons in the table (after Bonferroni correction for six tests, errors, transforms and contracted transforms stay below 0.05, warnings and features present do not).
• The parser behind the features and the checker was developed on the library; designs written by an LLM may contain constructs it reads incompletely.
• The ATWL designs differ little from one another on the 25 features, and the features cannot compare ATWL with no ATWL. That comparison is made on the prose designs in the coding analysis.

Files

The features, descriptors and tests were computed with the scripts designed_report.py, condition_analysis.py and run_web_checker.js of the ATWL Workflow Lens package (folder atwl_tools).