DPVis: ATWL workflow review

ATWL Workflow Review Report

Paper Title: DPVis: Visual Analytics With Hidden Markov Models for Disease Progression Pathways Workflow ID: DPVis

Overall Assessment

Summary: This representation captures the overall two-phase structure of DPVis well — model-based state/trajectory characterisation followed by interactive subgroup construction — and shows several sophisticated, well-justified modelling choices (notably the relational configuration transition-structure feature and the vector-valued, non-exclusive subgroup-membership feature). However, it contains one genuine structural/dataflow bug that silently breaks part of the loop's feedback semantics, and the model-selection loop's degree of interactive, tool-driven respecification is not well supported by the paper's own text, which explicitly identifies this as a desired future capability rather than a demonstrated one. These issues should be addressed before approval.

Recommendation: - [ ] Approve as-is - [ ] Approve with minor revisions - [x] Requires revision - [ ] Reject - major rework needed

Strengths

T_infer_visit_states correctly uses characterise (not define-unit) for applying the fitted model to assign state labels/posteriors to existing visits — a precise, textbook application of the "characterise vs. define-unit for labels" rule, in exactly the scenario the rule is designed to catch.
The three characterise transforms inside the model-selection loop (T_characterise_states, T_infer_visit_states, T_characterise_patient_trajectories) are each correctly scoped to a single entity type (D_disease_states, D_patient_visits, D_patient_sequences respectively) — avoiding the dual-entity conflation seen in other reviewed workflows.
F_state_transition_profile is well-modelled as value structure: relational configuration with a clarifying representation form: "directed state-transition relations", correctly capturing the paper's transition-constraint mechanism (e.g., no-backward restrictions, Section 3.3) without naming HMM internals.
F_subgroup_membership is thoughtfully typed as value structure: vector (rather than atomic), correctly reflecting that DPVis subgroups are not a mutually exclusive partition — patients can belong to multiple simultaneously-retained cohorts (Subgroup List, Section 4.5.1) — a nuance missed by naive copying of the "cluster-label" pattern from other workflows.
D_patient_visits's embedment {set, time} is a precise application of the spec's own canonical D2 example (patient × time events).
Model and visualisation artifacts avoid naming "Hidden Markov Model" or specific view techniques (beeswarm, force-edge bundling, Sankey, BIDE) directly in manner/model type, maintaining good abstraction level.
K_model_assessment's actor: hybrid is a nicely calibrated choice, correctly reflecting that model adequacy in the paper is judged through a combination of quantitative cross-validation and clinical interpretability (Section 3.3).

Critical Issues (Must Fix)

Issue 1: Loop feedback for "revised clinical variables" is structurally broken

Location: T_characterise_visits (declared before loop L_model_selection), T_specify_model_update, T_fit_progression_model

Problem: T_specify_model_update's description explicitly states it may "specify revised clinical variables... for another model evaluation round," producing S_model_configuration_r, which is then assigned back to S_model_configuration. However, T_characterise_visits — the transform that encodes which clinical measures are selected (manner: "encode selected clinical measures," input: S_model_configuration) — is declared outside and before loop L_model_selection, so it only ever runs once, using the original given S_model_configuration. Consequently, F_visit_measures is never recomputed, and any revised variable selection specified inside the loop has no effect on subsequent model fitting — only the "state complexity" and "progression constraints" portions of the specification actually propagate (via direct input to T_fit_progression_model). This is a genuine dataflow break, not merely a stylistic placement issue: part of the declared refinement capability is a no-op.

Correction:

# Move T_characterise_visits inside the loop, and update the template:

workflow DPVis
  template: define-unit ->
            loop(characterise -> build-model -> define-unit ->
                 characterise -> characterise -> characterise ->
                 visualise -> abstract -> assess ->
                 generate-knowledge (specify model)) ->
            loop(define-unit -> visualise -> abstract -> assess ->
                 generate-knowledge (specify cohort)) ->
            generate-knowledge
  ...

loop L_model_selection:
  body:

    transform T_characterise_visits :
      intent: characterise
      manner: "encode selected clinical measures"
      input: D_patient_visits, S_model_configuration
      output: F_visit_measures
      actor: machine
      description: "Create a visit-level representation from the
        currently selected clinical measures used to model disease
        progression"

    artifact F_visit_measures : feature(D_patient_visits)
      value structure: vector
      value type: numeric
      description: "Selected and encoded clinical measures for each
        patient visit under the current model configuration"

    transform T_fit_progression_model :
      intent: build-model
      manner: "fit latent disease-progression model"
      input: D_patient_sequences, D_patient_visits, F_visit_measures,
             S_model_configuration
      output: M_progression_model
      actor: machine
      ...

Why: Transforms whose output genuinely depends on an iteratively-updated specification must reside inside the loop that updates that specification (the inverse of the catalogued "loop-invariant transform" issue — here a variant transform is incorrectly treated as invariant). I have also added D_patient_visits to T_fit_progression_model's input, since the patient-grouping and visit-ordering structure needed to reconstruct per-patient observation sequences for HMM fitting is carried by D_patient_visits's patient_reference/visit_time features, not meaningfully re-derivable from D_patient_sequences alone once visits have been extracted.

Issue 2: Loop overstates DPVis's demonstrated capability for interactive model respecification

Location: loop L_model_selection, specifically the then branch (T_specify_model_update → assign: S_model_configuration := S_model_configuration_r → re-entry into T_fit_progression_model)

Problem: The loop implies that DPVis itself supports a visually-triggered cycle of assess → respecify → refit for the HMM. The paper directly contradicts this for the current system: Section 6.4 ("Users Want to Provide Feedback to the HMM Learning Process") reports that participants explicitly requested the ability "to modify states and to establish new states" and "to have more control over the training part," and the paper states as future work: "we plan to extend DPVis so that users can visually supervise the training process of HMMs." The actual model-selection process described in Section 5.2 is a largely offline cross-validation sweep over 2–20 states, external to the interactive visualisation; DPVis is used to interpret the chosen model, not to drive its (re)training interactively.

Correction: Soften the loop's framing to avoid implying DPVis-internal, visualisation-triggered retraining as an established capability, e.g.:

loop L_model_selection:
  purpose: "Evaluate candidate disease-progression models — trained
    externally across a range of state counts and variable selections —
    for clinical interpretability and fit, and select or request a
    revised configuration for further offline training rounds"
  until: "..."
  body:
    ...
    transform T_specify_model_update :
      intent: generate-knowledge
      manner: "select revised model configuration for a subsequent
        offline training round"
      ...
      description: "Specify revised clinical variables, state
        complexity, or progression constraints for retraining outside
        the interactive visualisation, informed by the current
        assessment"

Alternatively, if finer fidelity is not required, retain the current structure but add an explicit note (e.g., in the workflow description) that this loop represents a generalised capability consistent with the paper's described task (G1, "evaluate usefulness of model instances... by exploring exposed patterns") rather than a literally demonstrated closed interactive loop.

Why: Per Phase 3 (Paper Alignment), iterative refinement procedures represented as ATWL loops should reflect what the paper actually describes as happening (or as a reasonable generalisation thereof); here the paper's own discussion section explicitly flags the depicted capability as not-yet-implemented, which is a stronger signal than typical limitations language and should not be silently smoothed over.

Recommended Improvements (Should Fix)

Improvement 1: `S_cohort_exploration` is a premature input to the model-selection visualisation

Location: transform T_visualise_model_outputs

Suggestion: S_cohort_exploration ("layout, grouping, selection, and display settings" for cohort exploration) is listed as an input to T_visualise_model_outputs, which belongs to the model-selection loop — before loop L_subgroup_exploration (which actually produces/refines cohort-exploration settings) has executed. Given the template's strictly sequential two-loop structure, this creates a forward dependency on an artifact that, narratively, hasn't yet been meaningfully configured.

# SUGGESTED: remove the forward-reaching dependency
transform T_visualise_model_outputs :
  intent: visualise
  manner: "coordinated state-summary and progression views"
  input: D_disease_states, D_patient_visits, D_patient_sequences,
         F_state_measure_profile, F_state_transition_profile,
         F_visit_state_label, F_visit_state_posterior,
         F_patient_state_sequence, F_patient_transition_pattern
  output: V_model_overview
  ...

Why: If the intent is instead to reflect that DPVis is a single simultaneous multi-panel system where cohort filters could already be active during model inspection, this should be stated explicitly in the transform's description rather than left as an unexplained dependency on a not-yet-iterated specification.

Improvement 2: Frequent state-transition pattern mining (State Transition Pattern List) is only implicitly represented

Location: T_characterise_patient_trajectories, T_interpret_model_patterns

Suggestion: Section 4.2.5 describes a dedicated, partly automated contribution: mining frequent state-transition subsequences (via the BIDE algorithm) and ranking them by subject frequency — distinct from the per-patient trajectory features currently captured by F_patient_transition_pattern. Consider adding a machine-actor abstract step producing a population-level pattern artifact for frequent transition motifs:

transform T_mine_frequent_pathways :
  intent: abstract
  manner: "frequent subsequence mining"
  input: F_patient_state_sequence
  output: P_frequent_pathways
  actor: machine
  description: "Identify the most frequently occurring state-transition
    subsequences across patients, ranked by subject count"

artifact P_frequent_pathways : pattern(F_patient_state_sequence)
  representation form: "ranked list of frequent transition motifs"
  description: "Population-level frequent state-transition patterns
    supporting interpretation of common progression pathways"

Why: This is an explicitly named, dedicated contribution (its own view panel) rather than a minor detail, and distinguishing machine-mined population-level patterns from human-interpreted ones improves fidelity without over-specifying the mining algorithm.

Minor Suggestions (Optional)

model type: "latent disease-progression model" could be sharpened to "state-space model", matching the paper's own index terms ("hidden markov model, state space model") while still avoiding naming HMMs specifically.
static_attributes and outcome_attributes features on D_patient_sequences omit value type (optional, but adding e.g. {categorical, numeric} would aid clarity given these are explicitly used as cohort-definition criteria in T_define_patient_subgroups).
Consider making T_define_patient_subgroups's description explicitly mention that subgroups are not mutually exclusive (to justify, alongside F_subgroup_membership's vector structure, why this differs from the atomic cluster-label pattern used elsewhere).

Alignment with Paper

Captured from paper: - [partial] Main methodology phases (visit extraction, HMM-based state inference, trajectory characterisation, and subgroup construction are all represented; the model-selection phase has the structural dataflow break noted in Issue 1) - [partial] Iterative refinement procedures (subgroup-exploration loop is well-aligned with Section 6.2's sensitivity-analysis description; model-selection loop overstates current tool interactivity per Section 6.4 — see Issue 2) - [pass] Key methodological contributions (state characterisation, transition-constraint structure, posterior/uncertainty representation, and flexible non-exclusive subgroup building are all captured at an appropriate abstraction level; frequent-pathway mining is only implicitly represented — see Improvement 2) - [pass] Domain terminology (disease states, trajectories, subgroups — consistent with the paper's own vocabulary)

Missing or incorrect: - Broken feedback path for "revised clinical variables" specification (Issue 1) - Overstated interactive model-retraining capability relative to Section 6.4 (Issue 2) - Frequent subsequence mining (State Transition Pattern List) under-represented (Improvement 2)

Conclusion

This DPVis workflow demonstrates careful attention to several subtle ATWL distinctions — correct characterise-vs-define-unit usage for label assignment, non-exclusive subgroup membership modelling, and appropriate relational-configuration typing for transition structure. However, it contains a concrete structural bug (Issue 1) that silently disables part of the model-refinement feedback loop, and its central iterative model-selection mechanism is not well supported by the paper's own account of DPVis's current capabilities (Issue 2). Both should be addressed — the first via a straightforward restructuring (moving T_characterise_visits inside the loop), the second via either reframing the loop's narrative or adding an explicit caveat — after which this workflow would be well-positioned for approval.