workflow som-flow-cluster-exploration template: characterise (preprocess) -> define-unit (SOM) -> characterise (quality) -> characterise (interestingness) -> contextualise (grid) -> visualise (SOM) -> loop(abstract -> assess -> generate-knowledge (adapt) -> characterise -> define-unit -> characterise -> characterise -> contextualise -> visualise) -> visualise (flow graph) -> loop(abstract -> assess -> generate-knowledge (partition) -> define-unit (subset) -> define-unit (child SOM) -> characterise -> contextualise -> visualise -> visualise (flow graph)) -> abstract (reflect) -> generate-knowledge (synthesis) description: "Domain experts explore large time series collections through iterative SOM-based cluster analysis with guided partitioning. The workflow combines interactive SOM refinement, quality-driven guidance, hierarchical data partitioning, and analytic provenance visualisation to support the discovery of cluster structures and metadata relations." # ============================================================================ # INPUT # ============================================================================ artifact D_time_series : entities origin: given internal structure: elementary embedment: set features: - id: f_values value structure: vector value type: numeric description: "Primary time series values (e.g., pitch contour samples)" - id: f_metadata value structure: atomic value type: categorical description: "Categorical metadata attributes (e.g., speaker, nationality, gender, utterance type, repetition)" description: "Collection of time series data items with associated categorical metadata; e.g., recorded pitch contours of speaker utterances with speaker and experiment properties" artifact S_som_config : specification origin: given representation form: "parameter settings" description: "Initial configuration encompassing preprocessing (normalization strategy, interpolation method for equal-length vectors), distance metric (Euclidean, Manhattan, DTW), temporal weighting, and SOM parameters (grid dimensions, training iterations, learning rate, learning radius)" # ============================================================================ # STEP 1: PREPROCESSING # ============================================================================ transform T_preprocess : intent: characterise manner: "apply configured normalization (min/max, logarithmic, semitone, etc.) and interpolation (mean-padding, zero-padding, or linear) to produce equal-length numeric feature vectors" input: D_time_series, S_som_config output: F_vectors actor: machine description: "Transform raw time series into normalized equal-length feature vectors suitable for SOM training" artifact F_vectors : feature(D_time_series) value structure: vector value type: numeric description: "Preprocessed time series vectors of uniform length; normalization and interpolation applied according to configuration" # ============================================================================ # STEP 2: SOM TRAINING # ============================================================================ transform T_train_som : intent: define-unit manner: "train Self-Organizing Map using two-phase process (rough then fine training); assign each data item to its best-matching cell; produce prototype vectors by iterative neighbourhood adaptation" input: F_vectors, S_som_config output: D_som_cells actor: machine description: "Train SOM neural network to cluster data items into grid cells with topology-preserving prototype vectors" artifact D_som_cells : entities internal structure: formation embedment: set features: - id: f_prototype value structure: vector value type: numeric description: "Mean/prototype time series vector for the cell" - id: f_members value structure: set value type: reference description: "Data items assigned to this cell (best-matching unit)" - id: f_grid_position value structure: vector value type: numeric description: "Row and column position in the 2D SOM grid" description: "SOM cells arranged in a 2D grid: each cell contains a prototype vector representing a cluster of similar time series, with trained coordinates preserving topological similarity" # ============================================================================ # STEP 3: QUALITY AND INTERESTINGNESS COMPUTATION # ============================================================================ transform T_compute_quality : intent: characterise manner: "compute per-cell quantization error (mean distance of members to prototype), cell density (member count), and u-matrix values (mean distance to neighbouring cell prototypes)" input: D_som_cells output: F_cell_quality actor: machine description: "Calculate quality measures for each SOM cell to support assessment of clustering fidelity and structure identification" artifact F_cell_quality : feature(D_som_cells) value structure: vector value type: numeric description: "Per-cell quality measures: quantization error, density, and u-matrix distance values indicating cluster boundaries" transform T_compute_interestingness : intent: characterise manner: "for each metadata attribute and each cell, calculate Simpson's diversity index of attribute values among cell members; rank attributes by average interestingness across all cells" input: D_som_cells, D_time_series output: F_interestingness actor: machine description: "Compute interestingness scores measuring strength of relations between SOM cell structure and metadata attributes" artifact F_interestingness : feature(D_som_cells) value structure: vector value type: numeric description: "Per-cell per-attribute interestingness scores (Simpson's index); ranked attribute list by average interestingness" # ============================================================================ # STEP 4: SOM GRID ARRANGEMENT # ============================================================================ transform T_arrange_grid : intent: contextualise manner: "position cells according to their trained 2D grid coordinates; encode inter-cell distances as border widths" input: D_som_cells, F_cell_quality output: A_som_grid actor: machine description: "Arrange SOM cells in their topology-preserving 2D grid layout" artifact A_som_grid : arrangement(D_som_cells) context: "trained SOM topology" principle: "2D grid with cells positioned by trained row/column coordinates; similar cells placed adjacently; distances encoded as border thickness" description: "2D grid arrangement of SOM cells preserving topological neighbourhood relationships from training" # ============================================================================ # STEP 5: SOM VISUALISATION WITH GUIDANCE CUES # ============================================================================ transform T_visualise_som : intent: visualise manner: "render grid with prototype line charts per cell, data item overlays, min/max bandwidth; provide switchable overlays (density, u-matrix, quantization error, meta-clustering, meta-coloring, interestingness); show ranked interestingness thumbnails sidebar and partitioning recommendation cues" input: A_som_grid, F_cell_quality, F_interestingness output: V_som actor: machine description: "Generate composite SOM visualisation with quality overlays, metadata colouring, and guided recommendation cues" artifact V_som : visualisation(A_som_grid, F_cell_quality, F_interestingness) layout: "2D grid of cells; interestingness thumbnails ranked on left sidebar; partitioning cues on right sidebar" form: "tile-based grid with line-chart prototypes per cell; switchable overlays (density heatmap, u-matrix grayscale, meta-colouring with bar/circular/Hilbert pixel filling); recommendation cue icons" encoding: "prototype line: black bold = mean vector, blue thin = member vectors, yellow band = min/max range; density: blue intensity = count; u-matrix: light = similar neighbours, dark = boundaries; meta-colouring: categorical colours per attribute value; interestingness: yellow = related, dark gray = unrelated; quantization error: grayscale darkness = heterogeneity" description: "Rich composite SOM visualisation supporting analysis of cluster structure, quality, and metadata relations with integrated guidance" # ============================================================================ # STEP 6: ITERATIVE SOM REFINEMENT # ============================================================================ loop L_refine: purpose: "Iteratively adapt preprocessing, metric, or SOM parameters and retrain until the SOM adequately captures relevant data structure without confounding effects" until: "Analyst judges that SOM quality is satisfactory and the clustering reflects meaningful data properties rather than artifacts" body: # ======================================================================== # Step 6a: Identify quality issues and confounding effects # ======================================================================== transform T_identify_issues : intent: abstract manner: "inspect prototype vectors for artifacts (e.g., unnormalized pitch height causing gender separation); examine quantization error overlay for imprecise cells; check if topology reflects desired similarity structure or confounding variables" input: V_som output: P_quality_issues actor: human description: "Inspect SOM visualisation to identify quality problems, confounding effects, or inadequate preprocessing" artifact P_quality_issues : pattern(D_som_cells) representation form: "identified deficiencies" description: "Identified quality issues: confounding effects (e.g., gender pitch-height dominance), high quantization error regions, artifacts from inadequate preprocessing, or topology distortions" # ======================================================================== # Step 6b: Assess SOM adequacy # ======================================================================== transform T_assess_som : intent: assess manner: "judge whether identified issues require retraining with different configuration, or whether current SOM adequately represents meaningful data structure for further exploration" input: P_quality_issues, V_som output: K_som_quality actor: human description: "Judge whether the current SOM is adequate or requires adaptation of computations" artifact K_som_quality : knowledge(P_quality_issues) representation form: "quality judgment" description: "Assessment of SOM adequacy: satisfactory for exploration, or requires adaptation (specifying what needs changing)" if K_som_quality indicates SOM adequate: then: exit loop L_refine else: # ==================================================================== # Step 6c: Adapt configuration # ==================================================================== transform T_adapt_config : intent: generate-knowledge manner: "decide on corrective action: change normalization (e.g., add semitone normalization to remove pitch-height effect), adjust distance metric or temporal weighting, modify SOM grid dimensions or training parameters" input: K_som_quality, S_som_config output: S_som_config_r actor: human description: "Specify adapted preprocessing, metric, or SOM parameters to address identified quality issues" artifact S_som_config_r : specification representation form: "revised parameter settings" description: "Updated configuration with corrected preprocessing, metric, or SOM parameters" # ==================================================================== # Step 6d: Reprocess and retrain # ==================================================================== transform T_reprocess : intent: characterise manner: "reapply preprocessing with updated normalization and interpolation" input: D_time_series, S_som_config_r output: F_vectors_r actor: machine description: "Reprocess time series with adapted configuration" artifact F_vectors_r : feature(D_time_series) value structure: vector value type: numeric description: "Reprocessed feature vectors with updated normalization" transform T_retrain_som : intent: define-unit manner: "retrain SOM with updated parameters on reprocessed vectors" input: F_vectors_r, S_som_config_r output: D_som_cells_r actor: machine description: "Retrain SOM with adapted configuration" artifact D_som_cells_r : entities internal structure: formation embedment: set description: "Retrained SOM cells reflecting adapted configuration" # ==================================================================== # Step 6e: Recompute quality and interestingness # ==================================================================== transform T_recompute_quality : intent: characterise manner: "recompute quantization error, density, u-matrix for new cells" input: D_som_cells_r output: F_cell_quality_r actor: machine description: "Recompute quality measures for retrained SOM" artifact F_cell_quality_r : feature(D_som_cells_r) value structure: vector value type: numeric description: "Quality measures for retrained SOM cells" transform T_recompute_interestingness : intent: characterise manner: "recompute Simpson's index for all metadata attributes" input: D_som_cells_r, D_time_series output: F_interestingness_r actor: machine description: "Recompute interestingness for retrained SOM" artifact F_interestingness_r : feature(D_som_cells_r) value structure: vector value type: numeric description: "Recomputed interestingness scores for retrained SOM" # ==================================================================== # Step 6f: Rearrange and re-visualise # ==================================================================== transform T_rearrange_grid : intent: contextualise manner: "position retrained cells in new 2D grid topology" input: D_som_cells_r, F_cell_quality_r output: A_som_grid_r actor: machine description: "Arrange retrained SOM cells in grid layout" artifact A_som_grid_r : arrangement(D_som_cells_r) context: "retrained SOM topology" principle: "same as A_som_grid with updated cells" description: "Grid arrangement for retrained SOM" transform T_revisualise_som : intent: visualise manner: "re-render SOM with all overlays and recommendations" input: A_som_grid_r, F_cell_quality_r, F_interestingness_r output: V_som_r actor: machine description: "Visualise retrained SOM with updated quality and guidance" artifact V_som_r : visualisation(A_som_grid_r, F_cell_quality_r, F_interestingness_r) layout: "same as V_som" form: "same as V_som" encoding: "same as V_som" description: "Updated SOM visualisation after retraining" assign: S_som_config := S_som_config_r F_vectors := F_vectors_r D_som_cells := D_som_cells_r F_cell_quality := F_cell_quality_r F_interestingness := F_interestingness_r A_som_grid := A_som_grid_r V_som := V_som_r end loop L_refine # ============================================================================ # STEP 7: INITIALISE FLOW GRAPH # ============================================================================ transform T_init_flow_graph : intent: visualise manner: "create flow graph canvas with the current SOM as root node; enable cross-graph linking, brushing, and 2D colourmap propagation" input: V_som output: V_flow_graph actor: machine description: "Initialise analytic provenance flow graph with the refined SOM as root node" artifact V_flow_graph : visualisation(V_som) layout: "directed graph canvas; parent SOMs left, children right; zoomable and pannable" form: "embedded SOM grid visualisations as graph nodes; directed edges as data-flow links" encoding: "edge thickness: number of data items flowing; node size: adjustable; 2D colourmap: similarity-preserving colours across all SOMs; linking highlight intensity: shared item count" description: "Analytic provenance flow graph showing hierarchy of SOM analyses with data-flow connections and cross-graph exploration support" # ============================================================================ # STEP 8: ITERATIVE PARTITIONING AND EXPLORATION # ============================================================================ loop L_partition: purpose: "Iteratively partition data into subsets of interest, train child SOMs for detailed analysis, and build up the flow graph until the analyst achieves sufficient understanding of cluster structures and metadata relations" until: "Analyst has identified all relevant patterns, relations, and subgroup differences, or determined that no further partitioning is informative" body: # ======================================================================== # Step 8a: Identify patterns, relations, and interesting subsets # ======================================================================== transform T_identify_structure : intent: abstract manner: "inspect meta-colouring overlays to spot metadata-cluster relations; examine interestingness thumbnails and partitioning cues; identify homogeneous vs. heterogeneous cells; observe cluster boundaries via u-matrix; use linking and brushing across flow graph for comparison" input: V_som, V_flow_graph output: P_data_structure actor: human description: "Identify cluster structures, metadata relations, and interesting subsets worthy of further exploration" artifact P_data_structure : pattern(D_time_series) representation form: "identified clusters and metadata relations" description: "Discovered data structure: cluster groupings, relations between time series patterns and metadata attributes (e.g., nationality correlates with contour shape), heterogeneous cells requiring detail" # ======================================================================== # Step 8b: Assess whether further partitioning is needed # ======================================================================== transform T_assess_exploration : intent: assess manner: "judge whether current level of detail is sufficient to answer analytical questions, or whether partitioning into subsets would reveal additional hidden patterns or enable meaningful comparisons" input: P_data_structure, V_som, V_flow_graph output: K_exploration_state actor: human description: "Assess whether further data partitioning is needed or analysis can conclude" artifact K_exploration_state : knowledge(P_data_structure) representation form: "quality judgment" description: "Assessment of exploration completeness: sufficient insight achieved, or specific partitioning direction identified for deeper investigation" if K_exploration_state indicates exploration sufficient: then: exit loop L_partition else: # ==================================================================== # Step 8c: Select partitioning strategy # ==================================================================== transform T_select_partition : intent: generate-knowledge manner: "choose partitioning approach guided by recommendations: (a) retrain imprecise cells (high qe) as detailed child SOM, (b) split by interesting metadata attribute values, (c) split by meta-clustering labels (k-means on prototypes), (d) manually select cells of interest, (e) apply metadata filter to remove noise or irrelevant items" input: K_exploration_state, V_som output: S_partition_spec actor: human description: "Decide on partitioning strategy based on identified patterns and system recommendations" artifact S_partition_spec : specification representation form: "partitioning directive" description: "Partitioning specification: strategy type (cell selection, attribute split, meta-clustering, filter) and parameters (selected cells, attribute name, filter expression, k value)" # ==================================================================== # Step 8d: Partition data into subset # ==================================================================== transform T_partition_data : intent: define-unit manner: "extract data subset according to partition specification: select members of chosen cells, filter by attribute value, or split by cluster labels" input: D_time_series, D_som_cells, S_partition_spec output: D_subset actor: machine description: "Create data subset by applying the selected partitioning strategy" artifact D_subset : entities internal structure: elementary embedment: set features: - id: f_values value structure: vector value type: numeric description: "Time series values of subset members" - id: f_metadata value structure: atomic value type: categorical description: "Metadata attributes of subset members" description: "Data subset extracted by partitioning: a focused collection of time series for detailed analysis" # ==================================================================== # Step 8e: Train child SOM on subset # ==================================================================== transform T_train_child : intent: define-unit manner: "train new SOM on subset data using default or customised parameters; grid size adjusted to subset cardinality" input: D_subset, S_som_config output: D_child_cells actor: machine description: "Train a child SOM on the partitioned data subset for more detailed cluster analysis" artifact D_child_cells : entities internal structure: formation embedment: set description: "Child SOM cells providing finer-grained clustering of the data subset" # ==================================================================== # Step 8f: Compute quality and arrange child SOM # ==================================================================== transform T_child_quality : intent: characterise manner: "compute quantization error, density, u-matrix, and interestingness for child SOM cells" input: D_child_cells, D_subset output: F_child_quality, F_child_interestingness actor: machine description: "Compute quality and interestingness measures for child SOM" artifact F_child_quality : feature(D_child_cells) value structure: vector value type: numeric description: "Quality measures for child SOM cells" artifact F_child_interestingness : feature(D_child_cells) value structure: vector value type: numeric description: "Interestingness scores for child SOM cells" transform T_arrange_child : intent: contextualise manner: "position child SOM cells in 2D grid topology" input: D_child_cells, F_child_quality output: A_child_grid actor: machine description: "Arrange child SOM cells in grid layout" artifact A_child_grid : arrangement(D_child_cells) context: "child SOM topology" principle: "2D grid preserving topological neighbourhood" description: "Grid arrangement for child SOM" # ==================================================================== # Step 8g: Visualise child SOM and update flow graph # ==================================================================== transform T_visualise_child : intent: visualise manner: "render child SOM with all overlays, recommendations, and prototype visualisations" input: A_child_grid, F_child_quality, F_child_interestingness output: V_child_som actor: machine description: "Visualise child SOM with quality overlays and guidance" artifact V_child_som : visualisation(A_child_grid, F_child_quality, F_child_interestingness) layout: "same structure as V_som" form: "same as V_som" encoding: "same as V_som" description: "Child SOM visualisation for detailed subset analysis" transform T_update_flow_graph : intent: visualise manner: "add child SOM node to flow graph; draw data-flow link from parent showing item count; enable linking and brushing and 2D colourmap comparison across all graph nodes" input: V_flow_graph, V_child_som, V_som, S_partition_spec output: V_flow_graph_r actor: machine description: "Extend analytic provenance flow graph with new child node and parent-child data-flow connection" artifact V_flow_graph_r : visualisation(V_som, V_child_som) layout: "directed graph canvas; parent SOMs left, children right; zoomable and pannable" form: "embedded SOM grid visualisations as graph nodes; directed edges as data-flow links" encoding: "edge thickness: number of data items flowing; node size: adjustable; 2D colourmap: similarity-preserving colours across all SOMs; linking highlight intensity: shared item count" description: "Extended flow graph with new child SOM node and data-flow connection" # Focus shifts to child for next iteration assign: D_time_series := D_subset D_som_cells := D_child_cells F_cell_quality := F_child_quality F_interestingness := F_child_interestingness A_som_grid := A_child_grid V_som := V_child_som V_flow_graph := V_flow_graph_r end loop L_partition # ============================================================================ # STEP 9: REFLECTION AND KNOWLEDGE SYNTHESIS # ============================================================================ transform T_reflect : intent: abstract manner: "use flow graph overview with 2D colourmap linking, cross-SOM brushing, global meta-colouring, and annotations to compare cluster structures across all branches and levels; identify confirmed or rejected hypotheses and cross-branch relations" input: V_flow_graph output: P_analysis_overview actor: human description: "Reflect on the complete analysis provenance by comparing SOM results across the flow graph to identify overarching patterns and verify findings" artifact P_analysis_overview : pattern(D_time_series) representation form: "comparative summary of cluster relations across levels" description: "Overarching patterns identified across the full analysis: confirmed group differences, discovered metadata relations, identified data quality issues, and cross-branch structure comparisons" transform T_synthesise : intent: generate-knowledge manner: "compile confirmed hypotheses, novel discoveries, data quality findings, and open questions into domain knowledge; determine which findings warrant conventional statistical verification" input: P_analysis_overview, P_data_structure, V_flow_graph output: K_findings actor: human description: "Synthesise domain knowledge from the exploratory cluster analysis: confirmed hypotheses, novel patterns, and identified data issues" artifact K_findings : knowledge(P_analysis_overview, P_data_structure) representation form: "expert judgment and analytical conclusions" description: "Synthesised domain insights: confirmed or rejected hypotheses about group differences (e.g., speaker nationality effects on pitch contours), discovered metadata relations, identified data quality issues (coding errors, missing labels), and directions for further statistical verification"