RESEARCHER NOTES (protocol version 1.5). Do NOT give this folder's contents to the coders. HOW TO RUN (per call, fresh session, one design per call) message = full text of ../coder_prompt.txt, then a blank line, then the full text of one file from ../designs/ (D01.txt ... D08.txt), pasted inline. Coder 1 uses run_order_coder1 and coder 2 uses run_order_coder2 from mapping.csv. Nothing else is given to the coders (no task text, library, condition or model). Keep each raw output as _.json. FILES IN THIS FOLDER mapping.csv design_id -> task, condition, model, source chat file and turn segmentation_log.txt how each design was cut into units, what was removed or normalised researcher_notes.txt this file; protocol part B follows 1. PREPARING THE INPUT - Segment each design once, using the author's own numbered or lettered steps (Stage, Phase, Step or numbered headings) as units [U1], [U2], ...; sub-headings stay inside their step. Mark the title, introduction, overview and any section that lists deliverables, summaries, sources, data requests, tools or expected outcomes as [U0]. Remove closing offers addressed to the user ("Would you like me to ..."). Remove the model's reasoning blocks; keep only the final answer. - Clean markdown emphasis and normalise special characters once, then freeze. - Give both coders the identical segmented text. Remove file names, condition labels and any header that identifies the condition. - One design per call, in a fresh session. 2. RUNNING - Two independent sessions with no shared context; different model families if possible; lowest temperature. Record model, version, date and protocol version. - Pilot on one or two designs, fix unclear rules, then freeze the protocol. Include pilot designs in the analysis only if the protocol did not change after they were coded; otherwise re-run them. - D01 and D02 were used as pilots under versions 1.2 to 1.4. Re-run both under the final protocol and use only those runs. 3. AUTOMATIC VALIDATION OF EACH OUTPUT - Valid JSON; every unit present exactly once; consistency rules 3 to 5 of the self-check; every quote is an exact substring of the text after whitespace normalisation, and lies within the stated unit. - Invalid output: re-run the same call, at most twice, and log it. - The quote check ignores list markers ("- ") and extra whitespace. If a coder wraps the JSON in code fences, strip them before parsing and note it. 4. COMPARING THE TWO CODERS - Unit-level booleans (human_action, machine_action): raw agreement; Cohen's kappa only on units pooled over all designs (it is unstable for near-constant variables). - Control acts: match by unit and type; report agreement, precision and recall between coders; report evidence_stated and criterion_stated agreement on matched JUDGE/DECIDE acts. - Loops, parameters, branches: match by unit and quote overlap; report match rate, Jaccard agreement of "redefines", and agreement on trigger_stated, exit_stated, tuned, set_by, outcomes_stated. - Most important: agreement on the design-level measures below (rank agreement or ICC over the 8 designs). Item-level disagreement that does not change these measures matters little. - Pilot finding (D01): unit-level indicators agreed well (human action 9/9, machine action 8/9, any control act 9/9, output named 8/9), while item counts (control acts, parameters, branches) differed by a factor of 1.5 to 2.5 between coders and by a similar amount between two runs of the same coder. Prefer unit-level shares and presence indicators as primary measures; report item counts with their agreement, and use the union and the intersection as sensitivity analyses. The "units" field of loops is informational and is not compared. - Disagreements: use the items both coders coded as the primary analysis and the union as a sensitivity analysis; adjudicate only disagreeing items, and report how many there were. 5. DERIVED MEASURES (per design, from the units; global statements are reported separately and not counted) - C1: share of units with a stated actor; human share, machine share and hybrid share (both present) among those units. - C2: control acts per unit by type; share of JUDGE/DECIDE with evidence_stated and with criterion_stated. - C3: number of loops; distribution of "redefines"; share with trigger_stated and exit_stated; distribution of decided_by. - C4: number of named parameters; share tuned; share with initial value or range; distribution of set_by. - C5: number of branches by type; share with outcomes_stated. - C6: share of units that name an output. In ATWL every transform declares outputs by syntax, so this criterion is informative only for the prose side. - PROV: number of provenance statements per design. 6. SCOPE - Code only the prose designs of C1 and C4 (8 designs). Designs of C2, C3, C6, C6++ and C7 are measured from the ATWL. C5 is measured from its enrichment tables. If time is short, the minimum useful set is C1 (actor), C2 (JUDGE with evidence and criterion), C3 (loops with exit) and C6; drop C4 and C5. 7. LIMITS - The verb lists define what counts as a stated actor. A verb on neither list counts as not stated, so actor shares describe stated actors, not all actors. - Agreement between two LLM coders shows that the protocol is reproducible, not that the codes are valid. Both coders can share a bias. A small human check of the disagreeing items (for example 20) should be reported if time allows. - The ATWL side of the same measures is computed formally, not coded. It needs an explicit mapping (for example actor field to C1, human assess/abstract transforms to JUDGE, assign statements in loops to "redefines", specification artifacts to parameters). The mapping is not part of this file and must be fixed before any comparison between prose and ATWL designs.