workflow disease-progression-pathway-exploration template: generate-knowledge (configure) -> build-model (train HMMs) -> characterise (infer states) -> loop(visualise (feature matrix) -> abstract (interpret states) -> assess (model quality) -> generate-knowledge (adjust) -> build-model -> characterise (reinfer)) -> characterise (mine patterns) -> contextualise (waterfall) -> contextualise (sankey) -> contextualise (time bars) -> contextualise (chord) -> visualise (pathways) -> loop(abstract (identify trajectories) -> generate-knowledge (define subgroup) -> define-unit (execute query) -> characterise (subgroup stats) -> contextualise -> visualise (subgroup) -> abstract (compare) -> assess) -> generate-knowledge (synthesis) description: "Clinical researchers explore disease progression pathways from longitudinal health records using Hidden Markov Models. The workflow integrates HMM-based state discovery with coordinated interactive visualisations (Feature Matrix, Pathway Waterfall, Sankey, Chord Diagram, Pattern List, Subject List, Kernel Densities) to support iterative subgroup construction, trajectory comparison, and clinical hypothesis testing." # ============================================================================ # INPUT # ============================================================================ artifact D_patients : entities origin: given internal structure: elementary embedment: set features: - id: f_visits value structure: vector value type: {numeric, temporal} description: "Sequence of visits at irregular time points; each visit carries timestamp (age) and observed variable values" - id: f_observed_vars value structure: matrix value type: numeric description: "Per-visit observed measurements (e.g., GADA, IAA, IA2A autoantibody levels; growth measures like height, weight, BMI)" - id: f_static_vars value structure: vector value type: {numeric, categorical} description: "Time-invariant attributes: genetic profile (HLA), gender, race, family history, breastfeeding status" - id: f_outcomes value structure: vector value type: {numeric, categorical} description: "Health outcome variables: age at seroconversion, age at diagnosis, disease stage at final visit" description: "Longitudinal patient records from prospective clinical studies (e.g., 559 subjects from DAISY, DiPiS, DIPP birth cohorts); each patient has irregularly-spaced visits with observed biomarker measurements, static demographic/genetic attributes, and outcome indicators" # ============================================================================ # STEP 1: HMM CONFIGURATION # ============================================================================ transform T_configure_hmm : intent: generate-knowledge manner: "select output variables for HMM training (e.g., three autoantibodies: GADA, IAA, IA2A); specify number of hidden states to discover (range 2–20 for model comparison); set transition constraints (no-backward for chronic diseases, allowing only forward or self-loop transitions); choose model variant (discrete HMM or Continuous-time HMM)" input: D_patients output: S_hmm_config actor: human description: "Configure HMM parameters based on clinical knowledge: which variables manifest disease progression, how many latent states to discover, and what constraints on state dynamics" artifact S_hmm_config : specification representation form: "model configuration parameters" description: "HMM configuration: selected output variables, number of states K, transition constraint type (no-backward, free, or custom), model variant (CT-HMM), and cross-validation settings for model selection" # ============================================================================ # STEP 2: HMM TRAINING # ============================================================================ transform T_train_hmm : intent: build-model manner: "train Hidden Markov Model(s) on longitudinal observed data: learn state emission distributions (output probabilities per state), transition probability matrix, and initial state distribution; handle missing data and irregular time intervals natively; train multiple models with different K values for comparison" input: D_patients, S_hmm_config output: M_hmm actor: machine description: "Train HMM(s) to discover hidden disease states from the evolution of observed biomarkers across patient visits" artifact M_hmm : model(D_patients, S_hmm_config) model type: "Hidden Markov Model (CT-HMM variant)" representation form: "trained generative model" description: "Trained HMM with K states: includes emission probability distributions (what observed variable values each state produces), transition probability matrix (which state-to-state transitions are likely), and initial state probabilities; represents a generative model of disease progression dynamics" # ============================================================================ # STEP 3: STATE INFERENCE # ============================================================================ transform T_infer_states : intent: characterise manner: "apply trained HMM to infer the most probable hidden state for each patient visit; compute posterior probability distribution over all K states per visit; assign categorical state label to each visit based on maximum posterior" input: D_patients, M_hmm output: F_state_assignments actor: machine description: "Infer hidden disease states for every patient visit using the trained HMM, producing state labels and posterior distributions" artifact F_state_assignments : feature(D_patients) value structure: vector value type: {categorical, numeric} description: "Per-visit state inference: categorical state label (most probable state), posterior probability distribution over all K states (uncertainty), and visit timestamp; enables construction of state sequences per patient" # ============================================================================ # STEP 4: MODEL EVALUATION AND SELECTION # ============================================================================ loop L_model_selection: purpose: "Evaluate candidate HMM models (different K values) by inspecting whether discovered states are clinically meaningful and the number of states provides adequate resolution; select the best model for detailed exploration" until: "Researcher confirms that the selected model produces interpretable states corresponding to clinical knowledge and provides adequate granularity for trajectory analysis" body: # ======================================================================== # Step 4a: Visualise state characteristics # ======================================================================== transform T_vis_feature_matrix : intent: visualise manner: "render Feature Matrix: rows = observed variables, columns = states; each cell shows mean value colour-coded on diverging blue-red scale; show Feature Distribution on demand with mean, std, histogram, and missing-value bars per cell" input: F_state_assignments, M_hmm output: V_state_summary actor: machine description: "Visualise state characteristics via Feature Matrix and Feature Distribution for clinical interpretation" artifact V_state_summary : visualisation(F_state_assignments, M_hmm) layout: "matrix grid: rows = observed + additional variables, columns = K states; Feature Distribution expandable per row" form: "coloured button-like cells (Feature Matrix); mean ± std error bars with histogram (Feature Distribution)" encoding: "cell colour (diverging blue-red): mean value normalised to min-max range across states; grey shaded bar: proportion of missing values; histogram bar height: visit count in bin" description: "State Summary Panel: Feature Matrix providing overview of state characteristics and Feature Distribution providing details with uncertainty and missing-data indicators" # ======================================================================== # Step 4b: Interpret states clinically # ======================================================================== transform T_interpret_states : intent: abstract manner: "examine Feature Matrix columns to determine what each state represents clinically: which antibodies are positive/negative, which growth measures are elevated; categorise states by number of autoantibodies and specific antibody combinations; identify ambiguous states (intermediate posterior values) suggesting sub-states" input: V_state_summary output: P_state_semantics actor: human description: "Interpret the clinical meaning of each discovered state based on its variable distributions" artifact P_state_semantics : pattern(D_patients) representation form: "clinical state interpretations" description: "Clinical interpretation of HMM states: state categories (e.g., 0-AAB, 1-AAB/IAA, 1-AAB/GADA, 2-AAB, 3-AAB), ambiguous states requiring attention, and relationships between states and known clinical stages" # ======================================================================== # Step 4c: Assess model adequacy # ======================================================================== transform T_assess_model : intent: assess manner: "judge whether states are clinically meaningful and distinguishable; evaluate if K is too low (states conflate distinct clinical conditions) or too high (states are uninterpretable or redundant); compare cross-validation log-likelihood scores across K values; consider whether transition constraints produce clinically plausible dynamics" input: P_state_semantics, V_state_summary output: K_model_assessment actor: human description: "Assess whether the current HMM produces clinically meaningful and interpretable states" artifact K_model_assessment : knowledge(P_state_semantics) representation form: "quality judgment" description: "Model evaluation judgment: states are clinically interpretable and K provides adequate granularity, or model needs adjustment (different K, different output variables, or different constraints)" if K_model_assessment indicates adequate model: then: exit loop L_model_selection else: transform T_adjust_model : intent: generate-knowledge manner: "select a different number of states K from the pre-trained range; or change output variable selection; or modify transition constraints based on clinical reasoning" input: K_model_assessment, S_hmm_config output: S_hmm_config_r actor: human description: "Adjust HMM configuration based on evaluation" artifact S_hmm_config_r : specification representation form: "revised model parameters" description: "Updated HMM configuration (different K, revised output variables, or modified constraints)" transform T_retrain : intent: build-model manner: "retrain HMM with updated configuration (or select pre-trained model with different K from the batch)" input: D_patients, S_hmm_config_r output: M_hmm_r actor: machine description: "Retrain or select alternative HMM" artifact M_hmm_r : model(D_patients, S_hmm_config_r) model type: "HMM variant" representation form: "retrained generative model" description: "Alternative trained HMM with revised parameters" transform T_reinfer : intent: characterise manner: "re-infer states with updated model" input: D_patients, M_hmm_r output: F_state_assignments_r actor: machine description: "Recompute state assignments with new model" artifact F_state_assignments_r : feature(D_patients) value structure: vector value type: {categorical, numeric} description: "Updated state assignments from revised model" assign: S_hmm_config := S_hmm_config_r M_hmm := M_hmm_r F_state_assignments := F_state_assignments_r end loop L_model_selection # ============================================================================ # STEP 5: FREQUENT PATTERN MINING # ============================================================================ transform T_mine_patterns : intent: characterise manner: "apply BIDE algorithm to patient state sequences to extract top-50 most frequently occurring state transition patterns (by unique subject count); filter out single-state patterns" input: F_state_assignments output: F_frequent_patterns actor: machine description: "Mine frequent sequential state transition patterns from all patient state sequences" artifact F_frequent_patterns : feature(D_patients) value structure: vector value type: {categorical, numeric} description: "Ranked list of frequent state transition patterns: each pattern is a sequence of states with associated subject count and frequency; enables pattern-list-based exploration and filtering" # ============================================================================ # STEP 6: PATHWAY ARRANGEMENT AND MULTI-VIEW VISUALISATION # ============================================================================ transform T_arrange_waterfall : intent: contextualise manner: "divide vertical space by K states; position each visit as a dot on horizontal age axis at its state level; apply beeswarm algorithm for density while preserving timestamps; draw trajectory lines between consecutive visits per patient using force-edge bundling to reduce clutter" input: F_state_assignments output: A_waterfall actor: machine description: "Arrange patient state sequences as parallel beeswarm plots with bundled trajectory lines for the Pathway Waterfall view" artifact A_waterfall : arrangement(D_patients) context: "age (horizontal) × state (vertical)" principle: "horizontal axis: patient age at visit; vertical axis: inferred state; dots positioned by beeswarm to show density; trajectory lines bundled by force-edge bundling" description: "Pathway Waterfall arrangement showing individual patient trajectories across states over age with density-preserving layout" transform T_arrange_sankey : intent: contextualise manner: "aggregate state transitions over consecutive visits: vertically stack states per visit with height proportional to subject count; draw paths between consecutive visits with thickness proportional to transition count; support alignment adjustment" input: F_state_assignments output: A_sankey actor: machine description: "Arrange aggregated state transitions as Sankey diagram for Pathway over Observation view" artifact A_sankey : arrangement(D_patients) context: "visit sequence (horizontal) × state (vertical stacks)" principle: "columns: visit indices; stacked nodes: state counts per visit; horizontal paths: transition flows between states; path thickness: number of subjects" description: "Sankey-style arrangement aggregating state transitions over observed visits" transform T_arrange_time_unit : intent: contextualise manner: "aggregate state distributions by configurable time unit (years, months); create stacked bars per time bin showing state proportions; support alignment handle for anchor-point shifting" input: F_state_assignments output: A_time_bars actor: machine description: "Arrange state distributions as stacked bars per time unit" artifact A_time_bars : arrangement(D_patients) context: "age time bins × state" principle: "horizontal axis: age in chosen time units; vertical stacks: state proportions per time bin; alignment adjustable" description: "Time-unit-based stacked bar arrangement of state distributions across patient age" transform T_arrange_chord : intent: contextualise manner: "compute overall state-to-state transition frequencies across all consecutive visit pairs; arrange states as nodes on circular layout sized by visit count; draw directed arcs between states sized by transition frequency" input: F_state_assignments output: A_chord actor: machine description: "Arrange state transitions as chord diagram for overall frequency overview" artifact A_chord : arrangement(D_patients) context: "circular state layout" principle: "nodes on circle perimeter sized by visit count; directed arcs inside circle sized by transition frequency between states" description: "Chord diagram arrangement summarising overall state transition frequencies" transform T_vis_pathways : intent: visualise manner: "render coordinated pathway views: (1) Pathway Waterfall with coloured dots and bundled trajectory lines on dark background; (2) Pathway over Observation as Sankey with alignment handle; (3) Pathway by Time Unit as stacked bars; (4) Chord Diagram with coloured arcs; (5) Pattern List showing BIDE-mined sequences as coloured circle rows with frequency bars; all views support click filtering and linked highlighting" input: A_waterfall, A_sankey, A_time_bars, A_chord, F_frequent_patterns output: V_pathways actor: machine description: "Generate the coordinated State Transitions Panel with multiple pathway views for exploring state transition patterns" artifact V_pathways : visualisation(A_waterfall, A_sankey, A_time_bars, A_chord, F_frequent_patterns) layout: "switchable pathway views in central panel; Pattern List in side panel; all views share consistent state colouring" form: "Waterfall: beeswarm dots + bundled lines; Sankey: stacked nodes + flow paths; Time bars: stacked horizontal bars; Chord: circular arcs; Pattern List: coloured circle sequences + frequency bars" encoding: "dot/node/bar colour: state identity (consistent categorical palette); dot position: age × state; path thickness (Sankey): subject count; arc size (Chord): transition frequency; trajectory line: individual patient path; bar length (Pattern List): pattern frequency" description: "Coordinated pathway views enabling exploration of state transition patterns from overview (Chord, Sankey) to individual trajectories (Waterfall) with frequent pattern ranking" # -------------------------------------------------------------------------- # Initialise accumulating subgroup findings # -------------------------------------------------------------------------- artifact P_subgroup_findings : pattern(D_patients) representation form: "accumulated subgroup associations and anomalies" description: "Initially empty; accumulates findings from subgroup comparisons across exploration iterations" # ============================================================================ # STEP 7: ITERATIVE SUBGROUP BUILDING AND COMPARISON # ============================================================================ loop L_subgroup_analysis: purpose: "Iteratively build, refine, and compare patient subgroups based on state transition patterns, static variables, and outcomes to discover associations between progression trajectories and clinical factors" until: "Researcher has identified and characterised all clinically meaningful trajectory groups, tested relevant hypotheses, and derived actionable clinical insights" body: # ======================================================================== # Step 7a: Identify trajectory groups and patterns of interest # ======================================================================== transform T_identify_trajectories : intent: abstract manner: "inspect Pathway Waterfall for major trajectory groups (distinct vertical bundles transitioning through different state sequences); examine Sankey for volume of transitions; inspect Chord Diagram for dominant transition pairs; check Pattern List for most frequent sequences; identify differences in progression speed (dense early vertical paths = fast progression)" input: V_pathways, V_state_summary, P_subgroup_findings output: P_trajectory_groups actor: human description: "Identify major trajectory groups, their temporal characteristics, and interesting subpatterns informed by prior subgroup findings" artifact P_trajectory_groups : pattern(D_patients) representation form: "identified progression trajectory types" description: "Major trajectory groups identified: e.g., Multi-AAB First (states 0→2), IAA First (states 3→7), GADA First (states 8→10); with noted differences in progression speed, age at transitions, and relative volume" # ======================================================================== # Step 7b: Define subgroup query # ======================================================================== transform T_define_subgroup : intent: generate-knowledge manner: "build subgroup using one or more mechanisms: (a) State Sequence Finder: drag state nodes to create pattern query with constraints on time range, visit position, and posterior probability threshold; (b) Pathway Waterfall: click transition points to filter by specific state transition at specific age; (c) Pattern List: click a frequent pattern to filter matching subjects; (d) Static Variables: click bar segments to filter by demographics/genetics; (e) combine multiple filters across views" input: P_trajectory_groups, V_pathways, P_subgroup_findings output: S_subgroup_query actor: human description: "Specify a subgroup definition through visual query building across multiple views, informed by prior findings" artifact S_subgroup_query : specification representation form: "compound filter: state sequence pattern with temporal/posterior constraints + static variable filters" description: "Subgroup query specification combining state sequence pattern constraints (from State Sequence Finder or Pathway filters), temporal constraints (age ranges for transitions), posterior probability thresholds, and static variable filters" # ======================================================================== # Step 7c: Execute subgroup query # ======================================================================== transform T_execute_subgroup : intent: define-unit manner: "apply compound filter to identify patients matching all specified criteria; compute intersection of subjects satisfying state sequence pattern and static variable constraints" input: D_patients, F_state_assignments, S_subgroup_query output: D_subgroup actor: machine description: "Extract the patient subgroup matching the specified query criteria" artifact D_subgroup : entities internal structure: elementary embedment: set description: "Patient subgroup satisfying the query: subset of patients matching the specified trajectory pattern, temporal constraints, and static variable filters" # ======================================================================== # Step 7d: Compute subgroup characteristics # ======================================================================== transform T_compute_subgroup_stats : intent: characterise manner: "compute outcome distributions for subgroup: kernel density of diagnosis age, kernel density of seroconversion age, static variable distributions; compare with overall population distributions" input: D_subgroup, D_patients output: F_subgroup_profile actor: machine description: "Compute outcome and demographic distributions for the selected subgroup for comparison with the population" artifact F_subgroup_profile : feature(D_subgroup) value structure: vector value type: {numeric, categorical} description: "Subgroup profile: kernel density of diagnosis age, kernel density of seroconversion age, static variable distributions (genetic profiles, demographics); enables comparison with population baseline" # ======================================================================== # Step 7e: Visualise subgroup in context # ======================================================================== transform T_arrange_subgroup : intent: contextualise manner: "overlay subgroup distributions on population distributions: brighter kernel densities inside lighter population curves; highlight subgroup subjects in Subject List View; highlight subgroup trajectories in Pathway Waterfall; update Static Variables bars with subgroup proportions" input: D_subgroup, F_subgroup_profile, F_state_assignments output: A_subgroup_context actor: machine description: "Arrange subgroup data in context of population for comparative visualisation" artifact A_subgroup_context : arrangement(D_subgroup) context: "population baseline for comparison" principle: "overlaid distributions: brighter = subgroup, lighter = population; highlighted trajectories in Waterfall; highlighted rows in Subject List; filtered bars in Static Variables" description: "Contextualised subgroup arrangement enabling visual comparison with overall population" transform T_vis_subgroup : intent: visualise manner: "render Dual Kernel Densities (diagnosis above timeline, secondary outcome below, brighter overlay for subgroup with mean arrows); render Subject List with state-coloured dots and posterior pie charts per visit, onset markers (red rectangles, purple stars); update Pathway views to highlight subgroup; update Static Variables with filtered distributions; add subgroup to Subgroup List with editable name" input: A_subgroup_context, F_subgroup_profile output: V_subgroup actor: machine description: "Visualise subgroup in coordinated views: kernel densities, subject list, pathway highlights, and static variable distributions" artifact V_subgroup : visualisation(A_subgroup_context, F_subgroup_profile) layout: "Dual Kernel Densities (top of Patterns and Outcomes Panel); Subject List (below); pathway views with highlights; Static Variables bars; Subgroup List sidebar" form: "mirrored kernel density curves (above/below timeline); individual subject rows with coloured dots and pie charts; highlighted trajectory lines in Waterfall; coloured bar segments in Static Variables" encoding: "bright density curve: subgroup; light density curve: population; mean arrow: subgroup mean age; dot colour: state; pie slice: posterior probability; red rectangle: disease onset; purple star: seroconversion; bar segment: subgroup proportion of static variable value; Subgroup List bar height: relative subgroup size" description: "Coordinated subgroup visualisation enabling comparison of progression timing, individual trajectories, and demographic/genetic associations" # ======================================================================== # Step 7f: Compare and interpret subgroup # ======================================================================== transform T_compare_subgroups : intent: abstract manner: "compare subgroup's kernel densities with population and other subgroups to identify differences in progression timing; inspect individual Subject List entries to find anomalies (e.g., patients with uncertain posteriors, unexpected state assignments); examine Static Variables for genetic associations; compare trajectory groups' diagnosis/seroconversion distributions" input: V_subgroup, V_pathways, P_subgroup_findings output: P_subgroup_findings_r actor: human description: "Compare subgroup with population and other subgroups to identify associations and anomalies, accumulating with prior findings" artifact P_subgroup_findings_r : pattern(D_patients) representation form: "accumulated subgroup associations and anomalies" description: "Accumulated findings from subgroup comparisons: earlier/later progression compared to population, genetic profile associations, anomalous patients (e.g., diagnosed without autoantibodies suggesting measurement error or novel endotype), differences between trajectory groups" # ======================================================================== # Step 7g: Assess exploration sufficiency # ======================================================================== transform T_assess_exploration : intent: assess manner: "judge whether all major trajectory groups have been identified and characterised; evaluate whether associations with outcomes and static variables have been sufficiently explored; determine if anomalous cases have been investigated; consider if further subgroup refinement would yield new insights" input: P_subgroup_findings_r, P_trajectory_groups, V_subgroup output: K_exploration_state actor: human description: "Assess whether exploration is sufficient or further subgroup analysis is needed" artifact K_exploration_state : knowledge(P_subgroup_findings_r, P_trajectory_groups) representation form: "quality judgment" description: "Assessment of exploration progress: all major trajectories characterised and associations tested, or specific hypotheses or anomalies still requiring investigation" if K_exploration_state indicates sufficient exploration: then: assign: P_subgroup_findings := P_subgroup_findings_r exit loop L_subgroup_analysis else: assign: P_subgroup_findings := P_subgroup_findings_r end loop L_subgroup_analysis # ============================================================================ # STEP 8: KNOWLEDGE SYNTHESIS # ============================================================================ transform T_synthesise : intent: generate-knowledge manner: "compile clinical insights from trajectory exploration and subgroup analysis: confirmed trajectory groups with clinical naming (Multi-AAB First, IAA First, GADA First); associations between genetic profiles and progression speed; anomalous patients suggesting novel endotypes or measurement error; implications for early detection and precision intervention; identified hypotheses requiring further statistical testing or replication in other cohorts" input: P_subgroup_findings, P_trajectory_groups, P_state_semantics, K_exploration_state output: K_clinical_insights actor: human description: "Synthesise clinical knowledge from the disease progression analysis: trajectory types, associations, anomalies, and implications for precision medicine" artifact K_clinical_insights : knowledge(P_trajectory_groups, P_subgroup_findings) representation form: "clinical hypotheses and insights" description: "Synthesised clinical knowledge: (1) confirmed heterogeneous progression pathways distinguished by first autoantibody type; (2) identified rapid progressors (early IAA) with earlier diagnosis; (3) discovered anomalous cases (diagnosed without autoantibodies) suggesting novel endotypes; (4) associations between genetic profiles and trajectory membership; (5) implications for staging presymptomatic T1D beyond current 3-stage model; (6) hypotheses for replication in other cohorts and statistical verification"