## ATWL Workflow Review Report **Paper Title:** *DPVis: Visual Analytics With Hidden Markov Models for Disease Progression Pathways* **Workflow ID:** `dpvis-disease-progression-analysis` ### Overall Assessment The representation captures DPVis’s main analytical logic: HMM-based state inference, state and transition interpretation, and iterative cohort construction for outcome analysis. However, it underrepresents the paper’s core subgroup-management capability and has issues in model-selection semantics and cohort-view dataflow. **Recommendation:** - [ ] Approve as-is - [ ] Approve with minor revisions - [x] Requires revision - [ ] Reject — major rework needed ### Strengths - HMM state inference, posterior probabilities, and per-subject state sequences are appropriately represented. - The distinction between state interpretation, transition-pathway interpretation, and cohort-outcome analysis aligns well with tasks T1–T5. - The cohort-refinement loop has a valid explicit assessment, refinement specification, and feedback assignment. - `D_visits` correctly uses `embedment: {set, time}` for longitudinal multi-subject visit records. - State-summary and transition visualisations capture the principal DPVis view categories at an appropriate abstraction level. --- ### Critical Issues (Must Fix) #### Issue 1: Subject identity and sequential model input are insufficiently represented **Location:** `D_visits`, `T_train_select`, `M_hmm` **Problem:** `T_define_subjects` groups visits by subject identifier, but no subject identifier is declared in `D_visits`. Further, the HMM is trained only from `D_visits`, although disease progression modelling depends on the temporal sequence of visits per subject. `D_subjects` is created before model training but is not used to train the model. **Correction:** ```atwl artifact D_visits : entities origin: given internal structure: elementary embedment: {set, time} features: - id: f_subject_id value structure: atomic value type: reference description: "Identifier of the subject to whom the visit belongs" - id: f_observed_vars value structure: vector value type: numeric description: "Observed clinical variables chosen for state inference" - id: f_static_vars value structure: vector value type: {categorical, numeric} description: "Static subject covariates" - id: f_outcome_markers value structure: vector value type: {temporal, categorical} description: "Outcome-event indicators and timing" description: "Longitudinal clinical visit records for multiple subjects" transform T_train_select : intent: build-model manner: "fit hidden Markov model to longitudinal subject sequences" input: D_subjects, D_visits, S_hmm_spec output: M_hmm actor: hybrid description: "Train a hidden Markov model that infers latent disease states and transition dynamics from ordered subject visits" artifact M_hmm : model(D_subjects, D_visits, S_hmm_spec) model type: "hidden Markov model" representation form: "discrete latent states with state-conditional observation distributions and state-transition probabilities" description: "Trained model of latent disease states and temporal transitions across subject visit sequences" ``` **Why:** The subject grouping and within-subject temporal ordering are fundamental to HMM training in DPVis. --- #### Issue 2: Model selection is incorrectly embedded in a `build-model` transform **Location:** `T_train_select`, workflow template **Problem:** The transform both trains models and evaluates/selects them using cross-validated likelihood. Model fitting is `build-model`, but evaluating candidate-model quality is an `assess` activity. The paper explicitly describes model selection through cross-validation, with model usefulness also evaluated using clinical knowledge. **Correction:** Either treat the selected HMM configuration as a given input, or separate fitting from assessment and selection. ```atwl transform T_train_candidates : intent: build-model manner: "fit hidden Markov model candidates across state counts and transition constraints" input: D_subjects, D_visits, S_hmm_spec output: M_hmm_candidates actor: machine description: "Train candidate hidden Markov models for the specified configurations" artifact M_hmm_candidates : model(D_subjects, D_visits, S_hmm_spec) model type: "hidden Markov model candidates" representation form: "candidate HMM parameterisations with differing state counts and transition constraints" description: "Candidate disease progression models for selection" transform T_assess_model_fit : intent: assess manner: "evaluate predictive fit and clinical usefulness" input: M_hmm_candidates, D_subjects output: K_model_fit actor: hybrid description: "Assess candidate models using validation fit and interpretability criteria" artifact K_model_fit : knowledge(M_hmm_candidates) representation form: "model-selection assessment" description: "Assessment identifying an adequately fitting and interpretable HMM configuration" ``` A subsequent `generate-knowledge` transform may produce a selected model specification, followed by final HMM fitting. **Why:** `assess` must represent evaluation of model adequacy; `build-model` should represent fitting the model itself. Update the template if this separation is adopted. --- #### Issue 3: The workflow does not represent retained and comparable cohorts **Location:** `D_cohort`, `F_cohort_membership`, `L_cohort_analysis` **Problem:** DPVis’s central contribution is not merely defining one cohort: users create, retain, refine, name, and compare multiple potentially overlapping subgroups. The current workflow repeatedly replaces one `D_cohort`. It therefore cannot represent the paper’s Subgroup List or cohort comparison. An atomic categorical membership also incorrectly implies a single mutually exclusive cohort assignment. **Correction:** Represent a collection of retained cohorts and multi-cohort membership. ```atwl artifact D_cohorts : entities origin: given internal structure: group/cluster embedment: set description: "Initially empty or previously retained subject cohorts, each defined by a named set of selection criteria" artifact F_subject_cohort_memberships : feature(D_subjects) origin: given value structure: list value type: categorical description: "List of retained cohort memberships for each subject; memberships may overlap" transform T_build_or_update_cohorts : intent: define-unit manner: "interactive query-based cohort creation, refinement, and retention" input: D_subjects, F_state_sequence, D_visits, F_posterior, S_cohort_criteria, D_cohorts, F_subject_cohort_memberships output: D_cohorts_r, F_subject_cohort_memberships_r actor: hybrid description: "Create, update, name, and retain cohorts defined by transition, timing, outcome, or static-variable criteria" artifact D_cohorts_r : entities internal structure: group/cluster embedment: set description: "Updated collection of retained subject cohorts" artifact F_subject_cohort_memberships_r : feature(D_subjects) value structure: list value type: categorical description: "Updated overlapping memberships of subjects in retained cohorts" assign: D_cohorts := D_cohorts_r F_subject_cohort_memberships := F_subject_cohort_memberships_r ``` **Why:** The paper’s T5 task requires building, refining, comparing, exporting, and retaining multiple cohorts—not replacing one subgroup at every loop iteration. --- #### Issue 4: Cohort visualisation omits artifacts used by its encoding **Location:** `T_visualise_cohort`, `V_cohort` **Problem:** The visualisation encoding claims to show per-visit state colours and posterior-probability glyphs, but `F_state_label` is absent. It also shows selected cohort details but omits the membership artifact needed to determine which subjects belong to the cohort. `F_posterior` is an input to the transform but is absent from the visualisation declaration. **Correction:** ```atwl transform T_visualise_cohort : intent: visualise manner: "overlaid outcome-timing density plots, static-variable bar charts, and subject-level timelines" input: D_cohort, F_cohort_membership, F_cohort_outcomes, D_subjects, D_visits, F_state_label, F_posterior output: V_cohort actor: machine description: "Display selected-cohort outcome timing and static-variable distributions relative to the population, with subject-level state-sequence detail" artifact V_cohort : visualisation(D_cohort, F_cohort_membership, F_cohort_outcomes, D_subjects, D_visits, F_state_label, F_posterior) layout: "overlaid density plots, bar charts, and per-subject timelines" form: "dual density curves, horizontal bar charts, and coloured visit timelines with posterior-probability glyphs" encoding: "density curves from cohort and population outcome timing; bar height from static-variable proportions; cohort highlighting from F_cohort_membership; visit-dot colour from F_state_label; glyph size or pie segments from F_posterior" description: "Cohort view relating selected trajectory patterns to outcomes, static variables, and individual subject records" ``` **Why:** Visualisation declarations should reference the data artifacts that determine their marks, selections, and encodings. --- ### Recommended Improvements (Should Fix) #### Improvement 1: Add the Static Variables View as an explicit visualisation **Location:** Before `L_cohort_analysis` The paper includes a Static Variables View that supports cohort creation from sex, genetic profile, family history, and other static variables. The present workflow only includes static-variable bars after a cohort has already been selected. ```atwl transform T_visualise_static_vars : intent: visualise manner: "static-variable distribution view" input: D_subjects, F_subject_outcomes output: V_static_vars actor: machine description: "Display distributions of static subject variables for population overview and cohort selection" artifact V_static_vars : visualisation(D_subjects, F_subject_outcomes) layout: "aligned horizontal bar charts" form: "categorical distribution bars" encoding: "bar length from population proportion; interactive selections define static-variable cohort constraints" description: "Static Variables View for inspecting and selecting subject characteristics" ``` Add `V_static_vars`, `V_state_summary`, and `V_transitions` as inputs to the cohort-building transform if cohort criteria are created through direct interaction with these views. --- #### Improvement 2: Represent frequent state-sequence mining explicitly **Location:** Before `T_visualise_transitions` The State Transition Pattern List is based on frequent sequential-pattern mining. The workflow currently claims a ranked pattern list but does not derive the patterns. ```atwl transform T_mine_transition_patterns : intent: abstract manner: "frequent sequential-pattern mining" input: D_subjects, F_state_sequence output: P_frequent_sequences actor: machine description: "Identify frequent multi-state transition patterns and their prevalence across subjects" artifact P_frequent_sequences : pattern(F_state_sequence) representation form: "ranked frequent state-sequence patterns" description: "Recurring state-transition patterns ranked by the number of subjects containing each pattern" ``` Then include `P_frequent_sequences` in `T_visualise_transitions` and `V_transitions`. --- #### Improvement 3: Correct final knowledge provenance **Location:** `K_findings` ```atwl artifact K_findings : knowledge(P_trajectory_groups, P_associations, D_cohort, K_cohort_quality) representation form: "statements and retained cohort definitions" description: "Documented disease progression trajectory groups, their retained cohort definitions, and associations with outcomes and patient characteristics" ``` If retained multi-cohort management is added, replace `D_cohort` with `D_cohorts`. **Why:** The final synthesis transform consumes `K_cohort_quality`, which should be reflected in the final knowledge artifact. --- ### Minor Suggestions (Optional) - Remove `f_num_visits` from `D_subjects` unless it is consumed by a downstream transform or visualisation. - Consider describing the pathway view more specifically as including Pathway Waterfall, Sankey-style summaries, a chord diagram, and a ranked sequence-pattern list. - The paper uses continuous-time HMMs in its main case study, but generic “hidden Markov model” terminology is acceptable for a reusable workflow representation. - The proposed workflow properly avoids modelling the paper’s future-work feature of directly supervising HMM state creation, since it is not part of the implemented DPVis workflow. --- ### Alignment with Paper | Aspect | Assessment | Notes | |---|---|---| | HMM-based state inference | Partial | Core inference is represented, but training should explicitly use subject sequences. | | State-characterisation task (T1) | Pass | State summary and human interpretation are well represented. | | Transition-pattern task (T2) | Partial | Main transition views are captured; explicit frequent-pattern mining is missing. | | Outcome and subject-detail analysis (T3–T4) | Partial | Cohort outcome and posterior displays are represented, but cohort-view data dependencies need correction. | | Subgroup management (T5) | Fail | The workflow represents one replaced cohort, not multiple retained and comparable cohorts. | | Interactive refinement | Pass | Cohort criteria are refined through a valid assessment and assignment loop. | | Domain terminology | Pass | States, trajectories, subjects, cohorts, outcomes, posterior probabilities, and progression pathways align with the paper. | ### Conclusion The representation is a good account of the HMM-to-visual-analysis pipeline and its principal clinical-analysis goals. Revision is required to make subject-sequence modelling explicit, separate model evaluation from model building, correctly connect cohort visual encodings to their data sources, and—most importantly—represent DPVis’s retained, overlapping, and comparable subgroup management.