## ATWL Workflow Review Report **Paper Title:** DPVis: Visual Analytics With Hidden Markov Models for Disease Progression Pathways **Workflow ID:** DPVis --- ### Overall Assessment **Summary:** This representation captures the overall two-phase structure of DPVis well — model-based state/trajectory characterisation followed by interactive subgroup construction — and shows several sophisticated, well-justified modelling choices (notably the `relational configuration` transition-structure feature and the `vector`-valued, non-exclusive subgroup-membership feature). However, it contains one genuine structural/dataflow bug that silently breaks part of the loop's feedback semantics, and the model-selection loop's degree of interactive, tool-driven respecification is not well supported by the paper's own text, which explicitly identifies this as a desired future capability rather than a demonstrated one. These issues should be addressed before approval. **Recommendation:** - [ ] Approve as-is - [ ] Approve with minor revisions - [x] Requires revision - [ ] Reject - major rework needed --- ### Strengths - `T_infer_visit_states` correctly uses `characterise` (not `define-unit`) for applying the fitted model to assign state labels/posteriors to existing visits — a precise, textbook application of the "characterise vs. define-unit for labels" rule, in exactly the scenario the rule is designed to catch. - The three `characterise` transforms inside the model-selection loop (`T_characterise_states`, `T_infer_visit_states`, `T_characterise_patient_trajectories`) are each correctly scoped to a single entity type (`D_disease_states`, `D_patient_visits`, `D_patient_sequences` respectively) — avoiding the dual-entity conflation seen in other reviewed workflows. - `F_state_transition_profile` is well-modelled as `value structure: relational configuration` with a clarifying `representation form: "directed state-transition relations"`, correctly capturing the paper's transition-constraint mechanism (e.g., no-backward restrictions, Section 3.3) without naming HMM internals. - `F_subgroup_membership` is thoughtfully typed as `value structure: vector` (rather than `atomic`), correctly reflecting that DPVis subgroups are **not** a mutually exclusive partition — patients can belong to multiple simultaneously-retained cohorts (Subgroup List, Section 4.5.1) — a nuance missed by naive copying of the "cluster-label" pattern from other workflows. - `D_patient_visits`'s embedment `{set, time}` is a precise application of the spec's own canonical `D2` example (patient × time events). - Model and visualisation artifacts avoid naming "Hidden Markov Model" or specific view techniques (beeswarm, force-edge bundling, Sankey, BIDE) directly in `manner`/`model type`, maintaining good abstraction level. - `K_model_assessment`'s `actor: hybrid` is a nicely calibrated choice, correctly reflecting that model adequacy in the paper is judged through a combination of quantitative cross-validation and clinical interpretability (Section 3.3). --- ### Critical Issues (Must Fix) #### Issue 1: Loop feedback for "revised clinical variables" is structurally broken **Location:** `T_characterise_visits` (declared before `loop L_model_selection`), `T_specify_model_update`, `T_fit_progression_model` **Problem:** `T_specify_model_update`'s description explicitly states it may "specify revised **clinical variables**... for another model evaluation round," producing `S_model_configuration_r`, which is then assigned back to `S_model_configuration`. However, `T_characterise_visits` — the transform that encodes *which* clinical measures are selected (manner: "encode selected clinical measures," input: `S_model_configuration`) — is declared **outside and before** `loop L_model_selection`, so it only ever runs once, using the original given `S_model_configuration`. Consequently, `F_visit_measures` is never recomputed, and any revised variable selection specified inside the loop has **no effect** on subsequent model fitting — only the "state complexity" and "progression constraints" portions of the specification actually propagate (via direct input to `T_fit_progression_model`). This is a genuine dataflow break, not merely a stylistic placement issue: part of the declared refinement capability is a no-op. **Correction:** ``` # Move T_characterise_visits inside the loop, and update the template: workflow DPVis template: define-unit -> loop(characterise -> build-model -> define-unit -> characterise -> characterise -> characterise -> visualise -> abstract -> assess -> generate-knowledge (specify model)) -> loop(define-unit -> visualise -> abstract -> assess -> generate-knowledge (specify cohort)) -> generate-knowledge ... loop L_model_selection: body: transform T_characterise_visits : intent: characterise manner: "encode selected clinical measures" input: D_patient_visits, S_model_configuration output: F_visit_measures actor: machine description: "Create a visit-level representation from the currently selected clinical measures used to model disease progression" artifact F_visit_measures : feature(D_patient_visits) value structure: vector value type: numeric description: "Selected and encoded clinical measures for each patient visit under the current model configuration" transform T_fit_progression_model : intent: build-model manner: "fit latent disease-progression model" input: D_patient_sequences, D_patient_visits, F_visit_measures, S_model_configuration output: M_progression_model actor: machine ... ``` **Why:** Transforms whose output genuinely depends on an iteratively-updated specification must reside inside the loop that updates that specification (the inverse of the catalogued "loop-invariant transform" issue — here a *variant* transform is incorrectly treated as invariant). I have also added `D_patient_visits` to `T_fit_progression_model`'s input, since the patient-grouping and visit-ordering structure needed to reconstruct per-patient observation sequences for HMM fitting is carried by `D_patient_visits`'s `patient_reference`/`visit_time` features, not meaningfully re-derivable from `D_patient_sequences` alone once visits have been extracted. #### Issue 2: Loop overstates DPVis's demonstrated capability for interactive model respecification **Location:** `loop L_model_selection`, specifically the `then` branch (`T_specify_model_update` → `assign: S_model_configuration := S_model_configuration_r` → re-entry into `T_fit_progression_model`) **Problem:** The loop implies that DPVis itself supports a visually-triggered cycle of assess → respecify → refit for the HMM. The paper directly contradicts this for the *current* system: Section 6.4 ("Users Want to Provide Feedback to the HMM Learning Process") reports that participants explicitly *requested* the ability "to modify states and to establish new states" and "to have more control over the training part," and the paper states as **future work**: *"we plan to extend DPVis so that users can visually supervise the training process of HMMs."* The actual model-selection process described in Section 5.2 is a largely offline cross-validation sweep over 2–20 states, external to the interactive visualisation; DPVis is used to *interpret* the chosen model, not to drive its (re)training interactively. **Correction:** Soften the loop's framing to avoid implying DPVis-internal, visualisation-triggered retraining as an established capability, e.g.: ``` loop L_model_selection: purpose: "Evaluate candidate disease-progression models — trained externally across a range of state counts and variable selections — for clinical interpretability and fit, and select or request a revised configuration for further offline training rounds" until: "..." body: ... transform T_specify_model_update : intent: generate-knowledge manner: "select revised model configuration for a subsequent offline training round" ... description: "Specify revised clinical variables, state complexity, or progression constraints for retraining outside the interactive visualisation, informed by the current assessment" ``` Alternatively, if finer fidelity is not required, retain the current structure but add an explicit note (e.g., in the workflow `description`) that this loop represents a *generalised* capability consistent with the paper's described task (G1, "evaluate usefulness of model instances... by exploring exposed patterns") rather than a literally demonstrated closed interactive loop. **Why:** Per Phase 3 (Paper Alignment), iterative refinement procedures represented as ATWL loops should reflect what the paper actually describes as happening (or as a reasonable generalisation thereof); here the paper's own discussion section explicitly flags the depicted capability as not-yet-implemented, which is a stronger signal than typical limitations language and should not be silently smoothed over. --- ### Recommended Improvements (Should Fix) #### Improvement 1: `S_cohort_exploration` is a premature input to the model-selection visualisation **Location:** `transform T_visualise_model_outputs` **Suggestion:** `S_cohort_exploration` ("layout, grouping, selection, and display settings" for *cohort* exploration) is listed as an input to `T_visualise_model_outputs`, which belongs to the model-selection loop — before `loop L_subgroup_exploration` (which actually produces/refines cohort-exploration settings) has executed. Given the template's strictly sequential two-loop structure, this creates a forward dependency on an artifact that, narratively, hasn't yet been meaningfully configured. ``` # SUGGESTED: remove the forward-reaching dependency transform T_visualise_model_outputs : intent: visualise manner: "coordinated state-summary and progression views" input: D_disease_states, D_patient_visits, D_patient_sequences, F_state_measure_profile, F_state_transition_profile, F_visit_state_label, F_visit_state_posterior, F_patient_state_sequence, F_patient_transition_pattern output: V_model_overview ... ``` **Why:** If the intent is instead to reflect that DPVis is a single simultaneous multi-panel system where cohort filters could already be active during model inspection, this should be stated explicitly in the transform's description rather than left as an unexplained dependency on a not-yet-iterated specification. #### Improvement 2: Frequent state-transition pattern mining (State Transition Pattern List) is only implicitly represented **Location:** `T_characterise_patient_trajectories`, `T_interpret_model_patterns` **Suggestion:** Section 4.2.5 describes a dedicated, partly automated contribution: mining frequent state-transition subsequences (via the BIDE algorithm) and ranking them by subject frequency — distinct from the per-patient trajectory features currently captured by `F_patient_transition_pattern`. Consider adding a machine-actor `abstract` step producing a population-level pattern artifact for frequent transition motifs: ``` transform T_mine_frequent_pathways : intent: abstract manner: "frequent subsequence mining" input: F_patient_state_sequence output: P_frequent_pathways actor: machine description: "Identify the most frequently occurring state-transition subsequences across patients, ranked by subject count" artifact P_frequent_pathways : pattern(F_patient_state_sequence) representation form: "ranked list of frequent transition motifs" description: "Population-level frequent state-transition patterns supporting interpretation of common progression pathways" ``` **Why:** This is an explicitly named, dedicated contribution (its own view panel) rather than a minor detail, and distinguishing machine-mined population-level patterns from human-interpreted ones improves fidelity without over-specifying the mining algorithm. --- ### Minor Suggestions (Optional) - `model type: "latent disease-progression model"` could be sharpened to `"state-space model"`, matching the paper's own index terms ("hidden markov model, state space model") while still avoiding naming HMMs specifically. - `static_attributes` and `outcome_attributes` features on `D_patient_sequences` omit `value type` (optional, but adding e.g. `{categorical, numeric}` would aid clarity given these are explicitly used as cohort-definition criteria in `T_define_patient_subgroups`). - Consider making `T_define_patient_subgroups`'s description explicitly mention that subgroups are **not mutually exclusive** (to justify, alongside `F_subgroup_membership`'s `vector` structure, why this differs from the `atomic` cluster-label pattern used elsewhere). --- ### Alignment with Paper Captured from paper: - [partial] Main methodology phases (visit extraction, HMM-based state inference, trajectory characterisation, and subgroup construction are all represented; the model-selection phase has the structural dataflow break noted in Issue 1) - [partial] Iterative refinement procedures (subgroup-exploration loop is well-aligned with Section 6.2's sensitivity-analysis description; model-selection loop overstates current tool interactivity per Section 6.4 — see Issue 2) - [pass] Key methodological contributions (state characterisation, transition-constraint structure, posterior/uncertainty representation, and flexible non-exclusive subgroup building are all captured at an appropriate abstraction level; frequent-pathway mining is only implicitly represented — see Improvement 2) - [pass] Domain terminology (disease states, trajectories, subgroups — consistent with the paper's own vocabulary) Missing or incorrect: - Broken feedback path for "revised clinical variables" specification (Issue 1) - Overstated interactive model-retraining capability relative to Section 6.4 (Issue 2) - Frequent subsequence mining (State Transition Pattern List) under-represented (Improvement 2) --- ### Conclusion This DPVis workflow demonstrates careful attention to several subtle ATWL distinctions — correct `characterise`-vs-`define-unit` usage for label assignment, non-exclusive subgroup membership modelling, and appropriate relational-configuration typing for transition structure. However, it contains a concrete structural bug (Issue 1) that silently disables part of the model-refinement feedback loop, and its central iterative model-selection mechanism is not well supported by the paper's own account of DPVis's current capabilities (Issue 2). Both should be addressed — the first via a straightforward restructuring (moving `T_characterise_visits` inside the loop), the second via either reframing the loop's narrative or adding an explicit caveat — after which this workflow would be well-positioned for approval.