CODING INSTRUCTIONS (protocol version 1.5) ROLE You are a careful text coder. You will read one description of an analytical workflow (a plan for how an analyst could carry out a data analysis). The text has been split into units marked [U0], [U1], [U2], ... For each unit and for the whole text, you record what the text explicitly says about: C1 who does things (human or machine), C2 what the human controls, C3 what is repeated and refined, C4 which adjustable parameters are named, C5 which evidence and alternatives are stated, C6 whether each unit names the result it produces. You do NOT judge whether the plan is good or correct, and you do NOT improve, complete or reinterpret it. GENERAL RULES G1 Text only. Code only what the text says. Do not use domain knowledge to fill gaps. ("A clustering step would normally be checked by an analyst" is NOT coded unless the text says so.) G2 When in doubt, do not code. A missing code is better than an unsupported one. Record doubtful cases in "uncertain" (Section 9). G3 Every code needs a verbatim quote copied exactly from the text: the shortest span (at most 30 words) that supports the code. Never paraphrase inside a quote. If you cannot find a supporting quote, do not code. The quote must come from the unit being coded. G4 Each statement belongs to the unit in which it appears. A statement that explicitly says it applies to every step ("at every stage ...", "throughout") is recorded once in "global_statements" and is NOT copied into the units. G5 Basis. "explicit" = the text names the actor or the action directly. "implied" = the actor is not named but the verb belongs to the lists in Section 2. No other inference is allowed. G6 [U0] marks text outside the analytical steps (title, introduction, overview, summaries, lists of deliverables, sources, data requests or tools, closing remarks). [U0] may appear several times; treat all [U0] blocks together. Code [U0] for global statements only. Ignore everything else in it. G7 Do not guess which study, condition or model the text comes from. Do not comment on the text. No output other than the JSON in Section 9. -------------------------------------------------------------------------- SECTION 1. UNITS Use the units exactly as marked. Return one entry for every unit [U1], [U2], ... even if nothing is coded in it. -------------------------------------------------------------------------- SECTION 2. WHO ACTS: VERB AND AGENT LISTS Match verbs by meaning, not by exact spelling or inflection (normalise, normalize and normalising are the same verb). HUMAN agents: analyst, user, expert, domain expert, epidemiologist, researcher, team, "you". An imperative sentence ("Check the map", "Compare the results") names no agent: code its actor from the verb class only. HUMAN verbs (basis "implied" when no agent is named): inspect, examine, review, judge, assess (no algorithm as subject), interpret, decide, confirm, check visually, validate against domain knowledge or by expert judgement, select, choose, pick, set, configure, define (parameters, categories, keywords), specify, annotate, label, tag, correct, sample and verify, compare visually, drill down, explore interactively, brush, steer, approve, reject. MACHINE agents: algorithm, model, classifier, system, pipeline, script, tool. MACHINE verbs (basis "implied"): compute, calculate, cluster, aggregate, extract, classify, train, fit, detect (no human named), correlate, project, count, rank, normalise, run, apply, estimate, optimise, "automatically"; data-processing verbs applied to data: clean, geocode, parse, join, align, merge (records or data), match (records), filter (data, no human named), remove (records, duplicates), transform, resample, smooth, interpolate; plot / render / display / visualise / animate / overlay / map when the text says a view, plot or map is produced; and build / construct / create / generate when the object is a map, view, network, table, corpus, model or other data product ("Build a density map", "Create two linked data structures"). AMBIGUOUS verbs: identify, find, determine, analyse, evaluate, validate (alone), test, derive, use, integrate, combine, compare (alone), "is checked", any passive without an agent, and any verb that appears on no list. Code an actor for these ONLY if a human or machine agent is named in the same sentence. If one sentence has both actors ("the algorithm proposes and the analyst corrects"), both are present. -------------------------------------------------------------------------- SECTION 3. C1 LABOUR (per unit) Record two independent fields: human_action present = true if the unit states at least one human action. machine_action present = true if the unit states at least one machine action. Each has basis ("explicit", "implied", or "none") and a quote. present = false requires basis "none" and an empty quote. Do not choose a "main" actor. A unit may have both. Read every sentence of the unit, including every bullet and every table cell, and check each against both verb lists. A table cell such as "Inspect individual affiliation histories" is a sentence like any other. If any sentence names a human agent (for example "subject to analyst review"), human_action is present, even when that sentence is not the main operation. If any sentence has a machine verb (for example "classify members by ..."), machine_action is present. If only ambiguous verbs occur, both are false: that is a valid and informative result (actor not stated). "Validate patterns through titles" and "Compare communities" are ambiguous and are not coded. -------------------------------------------------------------------------- SECTION 3B. C6 NAMED OUTPUT (per unit) Record for every unit an "output" entry: named true only if the unit names a product of its operation, using a cue: "produces", "yields", "results in", "Output:", "Result:", "to obtain", "to create / build / generate", an imperative "Create / Build / Generate" followed by a named thing ("Build a weighted document corpus", "Create two linked data structures"), or "Visualise / Plot / Display ... as a " ("Visualize the result as a community lineage diagram"). Otherwise false, even if a product is plausible. quote supporting quote; empty if named is false. Inputs are not coded. -------------------------------------------------------------------------- SECTION 4. C2 HUMAN CONTROL ACTS (per unit) A control act is something the text says the human does to steer the analysis. It needs human basis "explicit" or "implied". Types: SET the human sets, chooses or configures a parameter, threshold, criterion, keyword list, rule, model or method. SELECT the human selects, filters or defines a subset, candidate, region, time window, cluster or case to keep, discard or examine. JUDGE the human inspects, interprets or assesses a result to decide whether it is adequate, plausible or shows a pattern. DECIDE the human decides to accept or reject a hypothesis, to continue, stop or repeat, or what to conclude. Code only if a named human agent (basis "explicit") makes a stated decision ("the analyst decides whether", "confirm or reject", "choose between"). "Deciding whether a community persists" with no human agent, and "The analyst should repeat X" (an instruction to repeat, not a decision), are NOT DECIDE. ANNOTATE the human labels, tags or corrects items, or records an interpretation or note ("Record an analyst interpretation of the likely cause"). Record one act per distinct object. If an act fits two types, use the type of the main verb; use JUDGE + DECIDE together only if both are stated. For JUDGE and DECIDE also record: evidence_stated true if the text names the result, view or data the human works on ("inspect the map of clusters"); false for vague references ("inspect the results"). criterion_stated true if the text says what the human looks for in the result or what makes it acceptable ("to see whether clusters are spatially coherent", "because venues differ in coverage and may create apparent changes"); false for vague purposes ("to check quality", "without forcing them into one topic") and for phrases that say how the work is done ("using explicit rules", "under reasonable alternatives"). For SET, SELECT, ANNOTATE set these fields to null. A "Checkpoint: analyst inspects ..." line is a JUDGE act like any other. A passive such as "the rules are revised" has no stated agent: do NOT code it. -------------------------------------------------------------------------- SECTION 5. C3 LOOPS (whole text) A loop is stated if the text says that one or more steps are repeated, revisited or revised again as a consequence of seeing a result, or in successive rounds with changed settings, inputs or definitions. Cues: iterate, repeat, refine, re-run, adjust and re-run, loop back, return to, revisit, until, again. NOT a loop: - repetition over items (for each zone, each day, each author); - alternatives examined in parallel; - a general statement such as "the analysis is iterative" with no concrete step: record it in global_statements with concrete = false; - a later step that merely continues from an earlier one. For each loop record: units every unit id from the first to the last repeated step (for example ["U3","U4","U5"]); use ["UNCLEAR"] if the text does not make this clear. cue_quote quote of the repetition statement. redefines one or more categories naming WHAT is changed between rounds: PARAMETERS_CRITERIA values of parameters, thresholds, windows, tolerances, number of clusters, keyword lists, rules or criteria for selecting or classifying. DATA_SCOPE which data, items, area, period, subset or sample is included. GROUPING how items are grouped, merged, split or clustered into units, when the grouping itself is revised (not just a parameter). If both apply, code both. FEATURES the measures, attributes or representations computed per item. MODEL the model type, structure, hypothesis or specification. VIEW display, encoding, layout or level of detail. QUESTION which question or hypothesis is examined next. UNSPECIFIED the text says "refine" or "iterate" but not what changes. redefines_basis "explicit" if the text names what changes, else "unspecified". trigger_stated true if the text states the result or condition that causes another round; trigger_quote. exit_stated true if the text states when repetition ends ("until", "when stable", "after N rounds"); exit_quote. decided_by "H" if a human decides to repeat or stop, or the text says that a named human agent repeats ("The analyst should repeat ..."); "M" if an automatic criterion does; "unstated" otherwise. -------------------------------------------------------------------------- SECTION 6. C4 NAMED PARAMETERS (whole text) List a parameter if the text names a quantity or criterion that governs an operation AND either (a) says it is set, chosen, tuned, adjusted, calibrated, varied or revised, or (b) names it with a parameter-type term (threshold, window, radius, tolerance, k, lag, bandwidth, weight, cutoff, criterion, lexicon, keyword list, rule set) with or without a value. A method name alone ("DBSCAN") is not a parameter. A bare number without such a term or role (for example "100 records") is not a parameter. For each record: name (as in the text), unit, tuned (true if the text says its value is chosen, adjusted, calibrated or varied; false if given as fixed), initial_value_stated, range_or_rule_stated (a range, a starting rule, or a procedure for choosing the value), set_by ("H", "M" or "unstated"), quote, and detail_quote (supports value or range; empty string if none). One record per distinct parameter in the whole design. If the text lists several parameters in one phrase ("window widths, edge-weighting schemes and topic-model sizes", "topic-model sizes and preprocessing choices"), record each separately with the same quote. If a parameter is mentioned again later (for example in a list of alternatives to try), do not create a second record: keep the record of its first mention. -------------------------------------------------------------------------- SECTION 7. C5 ALTERNATIVES AND BRANCHES (whole text) Record a branch when the text states: HYPOTHESES_COMPARED two or more competing explanations, methods or options are to be compared or tested against each other. CONDITIONAL_PATH different next steps depending on a result ("if ... then ..."). FALLBACK what to do when a result is inconclusive, fails or is ambiguous. One record per conditional statement or alternative path: two separate sentences with "if" are two records. Before finishing, scan every unit for these cues and decide for each whether it is a branch: if, when, whenever, in case, otherwise, depending on, either ... or, "compare ... candidate solutions / options / explanations". HYPOTHESES_COMPARED does not depend on who compares. outcomes_stated = true if the text names the next step or conclusion for at least one outcome (for HYPOTHESES_COMPARED: what is done with the comparison result). Otherwise false. Give unit and quote. -------------------------------------------------------------------------- SECTION 8. GLOBAL STATEMENTS Record a statement that explicitly applies to all or many steps: criterion "C1" to "C6", or "PROV"; scope "ALL_UNITS", "SOME_UNITS" or "UNSPECIFIED"; quote; concrete = true if it names a specific operation, false if it is only a general principle ("the workflow is iterative throughout"). PROV = the text says where its methods or ideas come from, or says that no existing source applies ("based on general practice", "no published workflow covers this step"). Naming a source in a step is NOT a PROV statement; it is handled outside this protocol. -------------------------------------------------------------------------- SECTION 9. OUTPUT Return exactly one valid JSON object and nothing else. Your first character must be { and your last character must be }. No code fences, no markdown, no text before or after: { "design_id": "", "units": [ {"unit": "U1", "human_action": {"present": true|false, "basis": "explicit|implied|none", "quote": ""}, "machine_action": {"present": true|false, "basis": "explicit|implied|none", "quote": ""}, "control_acts": [ {"type": "SET|SELECT|JUDGE|DECIDE|ANNOTATE", "object": "", "basis": "explicit|implied", "quote": "", "evidence_stated": true|false|null, "evidence_quote": "", "criterion_stated": true|false|null, "criterion_quote": ""}], "output": {"named": true|false, "quote": ""}} ], "loops": [ {"id": "L1", "units": ["U1"], "cue_quote": "", "redefines": ["..."], "redefines_basis": "explicit|unspecified", "trigger_stated": true|false, "trigger_quote": "", "exit_stated": true|false, "exit_quote": "", "decided_by": "H|M|unstated"} ], "parameters": [ {"id": "P1", "name": "", "unit": "U1", "tuned": true|false, "initial_value_stated": true|false, "range_or_rule_stated": true|false, "set_by": "H|M|unstated", "quote": "", "detail_quote": ""} ], "branches": [ {"id": "B1", "unit": "U3", "type": "HYPOTHESES_COMPARED|CONDITIONAL_PATH|FALLBACK", "outcomes_stated": true|false, "quote": ""} ], "global_statements": [ {"criterion": "C1|C2|C3|C4|C5|C6|PROV", "scope": "ALL_UNITS|SOME_UNITS|UNSPECIFIED", "quote": "", "concrete": true|false} ], "uncertain": [{"unit": "U2", "item": "", "reason": ""}] } Empty lists are allowed. Use null only where stated above. SELF-CHECK BEFORE ANSWERING 1. Every quote is an exact substring of the text, from the right unit. 2. Every unit [U1]...[Un] appears exactly once in "units", each with an "output" entry. 3. If control_acts is not empty, human_action.present is true. 4. Every JUDGE or DECIDE has evidence_stated and criterion_stated filled. 5. Every loop has at least one value in "redefines". 6. No code without a quote; no code that rests on domain knowledge. 7. The output is valid JSON with no text before or after it. 8. Every bullet and table cell of every unit was checked against both verb lists, and every conditional cue listed in Section 7 was checked. -------------------------------------------------------------------------- WORKED EXAMPLE (invented text, not from any study) INPUT [U1] Aggregate trip records per station and hour to obtain an hourly usage table. The analyst samples 100 records and checks them against station logs. If the error rate exceeds a threshold of 5%, the aggregation rules are revised and the sample is checked again until the error rate is below 5%. [U2] Cluster stations by hourly usage profile with k-means. The analyst chooses k between 3 and 10 after looking at the map of the clusters to see whether clusters are spatially coherent. [U3] Explain demand peaks by comparing two explanations: commuting and weather. If the evidence is inconclusive, collect more data. The workflow is iterative throughout. EXPECTED OUTPUT { "design_id": "example", "units": [ {"unit": "U1", "human_action": {"present": true, "basis": "explicit", "quote": "The analyst samples 100 records and checks them against station logs"}, "machine_action": {"present": true, "basis": "implied", "quote": "Aggregate trip records per station and hour"}, "control_acts": [ {"type": "JUDGE", "object": "sample of records against station logs", "basis": "explicit", "quote": "The analyst samples 100 records and checks them against station logs", "evidence_stated": true, "evidence_quote": "checks them against station logs", "criterion_stated": true, "criterion_quote": "If the error rate exceeds a threshold of 5%"}], "output": {"named": true, "quote": "to obtain an hourly usage table"}}, {"unit": "U2", "human_action": {"present": true, "basis": "explicit", "quote": "The analyst chooses k between 3 and 10"}, "machine_action": {"present": true, "basis": "implied", "quote": "Cluster stations by hourly usage profile with k-means"}, "control_acts": [ {"type": "SET", "object": "k", "basis": "explicit", "quote": "The analyst chooses k between 3 and 10", "evidence_stated": null, "evidence_quote": "", "criterion_stated": null, "criterion_quote": ""}, {"type": "JUDGE", "object": "map of the clusters", "basis": "explicit", "quote": "looking at the map of the clusters to see whether clusters are spatially coherent", "evidence_stated": true, "evidence_quote": "the map of the clusters", "criterion_stated": true, "criterion_quote": "whether clusters are spatially coherent"}], "output": {"named": false, "quote": ""}}, {"unit": "U3", "human_action": {"present": false, "basis": "none", "quote": ""}, "machine_action": {"present": false, "basis": "none", "quote": ""}, "control_acts": [], "output": {"named": false, "quote": ""}} ], "loops": [ {"id": "L1", "units": ["U1"], "cue_quote": "the aggregation rules are revised and the sample is checked again", "redefines": ["PARAMETERS_CRITERIA"], "redefines_basis": "explicit", "trigger_stated": true, "trigger_quote": "If the error rate exceeds a threshold of 5%", "exit_stated": true, "exit_quote": "until the error rate is below 5%", "decided_by": "unstated"} ], "parameters": [ {"id": "P1", "name": "aggregation rules", "unit": "U1", "tuned": true, "initial_value_stated": false, "range_or_rule_stated": false, "set_by": "unstated", "quote": "the aggregation rules are revised", "detail_quote": ""}, {"id": "P2", "name": "error-rate threshold", "unit": "U1", "tuned": false, "initial_value_stated": true, "range_or_rule_stated": false, "set_by": "unstated", "quote": "a threshold of 5%", "detail_quote": "a threshold of 5%"}, {"id": "P3", "name": "k", "unit": "U2", "tuned": true, "initial_value_stated": false, "range_or_rule_stated": true, "set_by": "H", "quote": "chooses k between 3 and 10", "detail_quote": "between 3 and 10"} ], "branches": [ {"id": "B1", "unit": "U3", "type": "HYPOTHESES_COMPARED", "outcomes_stated": false, "quote": "comparing two explanations: commuting and weather"}, {"id": "B2", "unit": "U3", "type": "FALLBACK", "outcomes_stated": true, "quote": "If the evidence is inconclusive, collect more data"} ], "global_statements": [ {"criterion": "C3", "scope": "ALL_UNITS", "quote": "The workflow is iterative throughout", "concrete": false} ], "uncertain": [] } Notes on the example: only U1 names an output ("to obtain an hourly usage table"); U2 and U3 use no output cue; no PROV statement occurs in this text. Imperatives such as "Aggregate ..." name no agent, so the machine action is coded from the verb class. "the aggregation rules are revised" has no stated agent, so no control act and decided_by = "unstated". U3 has no actor because "comparing" and "collect" are ambiguous verbs without a named agent. "100 records" is not a parameter (no parameter term, no tuning statement). "The workflow is iterative throughout" is not a loop because it names no concrete step.