## ATWL Workflow Review Report **Paper Title:** *SOMFlow: Guided Exploratory Cluster Analysis with Self-Organizing Maps and Analytic Provenance* **Workflow ID:** `somflow-guided-cluster-exploration` ### Overall Assessment The representation captures SOMFlow’s iterative SOM analysis cycle well: training a SOM, assigning data to cells, computing quality and metadata-interestingness measures, visual inspection, assessment, strategy adaptation, and data partitioning. However, it misses the paper’s central methodological contribution: retaining, visualising, navigating, and comparing the branching history of SOM analyses in an analytic-provenance flow graph. The current workflow overwrites the current dataset and patterns, so the claimed post-loop reflection on accumulated analysis results is not supported by the declared dataflow. **Recommendation:** - [ ] Approve as-is - [ ] Approve with minor revisions - [x] Requires revision - [ ] Reject — major rework needed --- ### Strengths - The main SOM workflow is correctly represented as `build-model → define-unit → contextualise → characterise → visualise → abstract → assess`. - `M_som` appropriately captures SOMs as a model combining prototype-based quantisation, clustering, and a topology-preserving embedding. - Defining SOM cells and cell-assignment labels through the same `define-unit` transform is appropriate. - `D_cells` as `group/cluster` entities embedded in a `relational structure` appropriately represents SOM cells and their neighbourhood topology. - The representation correctly includes both quantisation quality and metadata-related interestingness as computed guidance measures. - The strategy feedback loop is structurally valid: `T_decide_strategy` produces an updated specification and the `assign:` statement closes the feedback loop. - The input data and case-study terminology—utterances, pitch contours, metadata, SOM cells, and interestingness—align well with the paper. --- ### Critical Issues (Must Fix) #### Issue 1: The analytic-provenance flow graph and branching analysis history are missing **Location:** `L_drilldown`, `T_define_next`, `T_reflect` **Problem:** The paper’s principal contribution is not only sequential drill-down. SOMFlow retains all SOM results, their parent-child relations, filters, splitters, and selected subsets in a branching flow graph. This graph enables comparison, navigation, provenance inspection, annotations, linking and brushing across SOMs, and reflection on prior decisions. The current workflow instead replaces the working dataset: ```atwl assign: D_utterances := D_next ``` It also replaces `P_patterns` and `K_assessment` on every iteration. Consequently, `T_reflect` can access only the most recent pattern and assessment, despite claiming to compare findings “accumulated across the drill-down analysis.” **Correction:** Represent retained SOM analysis runs and their lineage explicitly, and visualise them during the loop. ```atwl artifact D_som_runs : entities internal structure: elementary embedment: relational structure description: "Retained SOM analysis runs, including the initial SOM and all subsequently created, retrained, filtered, or partitioned SOMs" artifact F_run_lineage : feature(D_som_runs) value structure: relational configuration representation form: "directed parent-child analysis flow graph" description: "Relations between SOM runs indicating retraining, filtering, splitting, or selection-based partitioning operations" transform T_record_analysis_step : intent: define-unit manner: "retain current SOM result as a provenance-graph node" input: D_som_runs, F_run_lineage, M_som, D_utterances, S_analysis_strategy output: D_som_runs_r, F_run_lineage_r actor: machine description: "Add the current SOM analysis result and its relation to predecessor runs to the retained analytic-provenance graph" artifact D_som_runs_r : entities internal structure: elementary embedment: relational structure description: "Updated collection of retained SOM analysis runs" artifact F_run_lineage_r : feature(D_som_runs_r) value structure: relational configuration representation form: "directed parent-child analysis flow graph" description: "Updated provenance relations between retained SOM runs" assign: D_som_runs := D_som_runs_r F_run_lineage := F_run_lineage_r ``` Add a provenance visualisation and use it for reflection: ```atwl transform T_visualise_flow : intent: visualise manner: "analytic-provenance flow graph with embedded SOM views" input: D_som_runs, F_run_lineage output: V_som_flow actor: machine description: "Display retained SOM analyses as graph nodes connected by data-flow and partitioning relationships, supporting navigation, comparison, annotation, and provenance inspection" artifact V_som_flow : visualisation(D_som_runs, F_run_lineage) layout: "directed flow graph" form: "graph nodes containing SOM thumbnails and directed links" encoding: "nodes represent retained SOM runs; links represent data-flow relations; link width represents the number of transferred items; annotations and metadata overlays support comparison" description: "Analytic-provenance graph showing the evolution and branching structure of SOM-based cluster exploration" ``` Then revise reflection: ```atwl transform T_reflect : intent: abstract manner: "compare retained SOM results and analysis branches" input: V_som_flow, D_som_runs, F_run_lineage output: P_overall_relations actor: human description: "Compare current and earlier SOM analyses, branches, and data partitions to identify verified relations and assess the analytical process" ``` **Why:** Retaining and visually comparing multiple SOMs is essential to SOMFlow. It cannot be represented as a linear overwrite-only drill-down process. --- #### Issue 2: Several transforms and visual encodings omit required inputs **Location:** `T_compute_cell_measures`, `T_visualise_som`, `V_som` **Problem:** Quantisation error requires the SOM prototypes from `M_som`, but `T_compute_cell_measures` does not consume the model. Likewise, the SOM visualisation claims to show prototype vectors, topology overlays, and items assigned to cells, but it omits both `M_som` and `F_cell_assignment`. **Correction:** ```atwl transform T_compute_cell_measures : intent: characterise manner: "compute quantization error and metadata interestingness per cell" input: D_cells, F_cell_assignment, D_utterances, M_som output: F_cell_quality, F_cell_interestingness actor: machine description: "Compute per-cell quantization error from assigned items and SOM prototypes, and metadata-interestingness scores for identifying relations between cell content and metadata" ``` ```atwl transform T_visualise_som : intent: visualise manner: "SOM grid view with prototype time series, topology overlays, metadata colouring, and ranked attribute recommendations" input: A_cell_layout, D_cells, D_utterances, M_som, F_cell_assignment, F_cell_quality, F_cell_interestingness output: V_som actor: machine description: "Render SOM cells with prototype and assigned-item time series, topology and quality overlays, metadata colouring, and ranked recommendations for further analysis" ``` ```atwl artifact V_som : visualisation(A_cell_layout, D_cells, D_utterances, M_som, F_cell_assignment, F_cell_quality, F_cell_interestingness) layout: "two-dimensional SOM grid" form: "cell glyphs with prototype and item line charts, quality or metadata overlays, and recommendation thumbnails" encoding: "cell position from A_cell_layout; prototype line and training history from M_som; assigned item traces and bandwidth from D_utterances and F_cell_assignment; cell shading from F_cell_quality or f_metadata; density from f_cell_size; ranked thumbnails from F_cell_interestingness" description: "Interactive SOM view supporting inspection of cluster structure, prototype quality, and metadata relations" ``` **Why:** ATWL visualisations should reference the artifacts that provide their marks, overlays, and interaction-relevant membership mappings. --- #### Issue 3: The template and post-loop reflection contradict the represented control flow **Location:** Workflow template; `T_reflect` **Problem:** The template includes `abstract (reflect)` after the drill-down loop. In the paper, reflection occurs continuously through interaction with the provenance graph, not only after the analyst has stopped iterating. Moreover, the current post-loop transform cannot reflect on historical results because all relevant artifacts are overwritten. **Correction:** Move provenance visualisation and reflection into the iterative workflow after recording a new analysis run. For example: ```atwl template: loop(build-model -> define-unit (assign cells) -> contextualise (grid layout) -> characterise (cell measures) -> visualise (SOM view) -> abstract -> assess -> generate-knowledge (strategy) -> define-unit (partition/retain) -> visualise (provenance flow) -> abstract (reflect)) -> generate-knowledge (synthesise) ``` **Why:** Reflection and comparison of prior analysis branches are active analytical tasks supported throughout SOMFlow exploration, not a final operation on only the last SOM result. --- ### Recommended Improvements (Should Fix) #### Improvement 1: Represent machine-generated partitioning recommendations explicitly **Location:** `T_compute_cell_measures`, `T_visualise_som`, `T_decide_strategy` **Problem:** The paper’s guidance does not consist only of displaying quality and interestingness. SOMFlow generates explicit recommendations for: - refining high-quantisation-error cells; - extending a selected cell group with similar neighbours; - splitting data by metadata attributes; - splitting data by meta-cluster labels. The current representation refers generally to “ranked recommendations,” but does not produce an artifact representing these recommendation cues. **Suggestion:** ```atwl transform T_compute_cell_measures : intent: characterise manner: "compute quality, interestingness, similarity, and partitioning recommendations" input: D_cells, F_cell_assignment, D_utterances, M_som output: F_cell_quality, F_cell_interestingness, F_partition_recommendations actor: machine description: "Compute cell quality and metadata-interestingness measures, identify similar neighbouring cells, and derive candidate refinement and partitioning actions" artifact F_partition_recommendations : feature(D_cells) value structure: vector value type: {numeric, categorical} description: "Per-cell and map-level recommendation scores and action categories for refinement, similar-cell selection, metadata splits, and cluster-based splits" ``` Include `F_partition_recommendations` in `V_som` and `T_decide_strategy`. **Why:** Guidance based on quality and interestingness is a major SOMFlow contribution and should be represented as machine-derived analytical support, rather than only as an informal visual description. --- #### Improvement 2: Include manual annotation as an analysis operation **Location:** Adaptation stage in `L_drilldown` **Problem:** The paper supports user-defined cell and data-item annotations, such as marking measurement-error cells as noise or labelling cells as interesting. These labels can then become metadata and influence later filtering, interestingness computation, and analysis. This is important in the reported speech-intonation case study, where manual noise annotation is followed by filtering and retraining. **Suggestion:** ```atwl transform T_annotate_cells : intent: characterise manner: "interactive assignment of user-defined metadata labels" input: D_cells, F_cell_assignment, V_som output: F_user_annotations actor: human description: "Assign analyst-defined labels, such as interesting, uninteresting, or measurement-error, to cells or their member items" artifact F_user_annotations : feature(D_utterances) value structure: atomic value type: categorical description: "User-defined item annotations propagated from selected SOM cells" ``` Use `F_user_annotations` as an input to later measure computation and partitioning. **Why:** Manual data enrichment is part of SOMFlow’s adaptation capabilities and supports the paper’s noise-removal workflow. --- #### Improvement 3: Preserve the data-feature schema when deriving subsets **Location:** `D_next` **Problem:** `D_next` is used as the next iteration’s data input, including for SOM training and metadata-based analysis. However, it declares no `f_pitch_contour` or `f_metadata` features, despite those features being required downstream. **Suggestion:** ```atwl artifact D_next : entities internal structure: elementary embedment: set features: - id: f_pitch_contour value structure: vector value type: numeric description: "Time series profile retained for each selected item" - id: f_metadata value structure: vector value type: categorical description: "Associated metadata retained for each selected item" description: "Selected or retained subset of the current data, preserving time series and metadata for the next SOM analysis" ``` **Why:** The subset must retain the data characteristics consumed by `T_train_som`, `T_compute_cell_measures`, and `T_visualise_som`. --- #### Improvement 4: Align final knowledge provenance with its synthesis inputs **Location:** `K_findings` **Suggestion:** ```atwl artifact K_findings : knowledge(P_overall_relations, K_assessment) representation form: "statements and explanations" description: "Documented findings about discovered clusters, metadata relations, and data-quality issues identified through iterative SOM-based exploration" ``` If the provenance graph is added, also include it: ```atwl artifact K_findings : knowledge(P_overall_relations, K_assessment, D_som_runs, F_run_lineage) ``` **Why:** `T_synthesise` consumes both `P_overall_relations` and `K_assessment`; the resulting knowledge should reference its declared evidence. --- ### Minor Suggestions (Optional) - The paper includes both grid and force-directed views of the SOM topology. The grid layout is the core representation, so `T_grid_layout` is appropriate; the force-directed view may be described as an alternative display mode in `V_som`. - `F_cell_interestingness` is well typed as a numeric vector because it holds one score per metadata attribute. - Consider adding topological separation measures, such as neighbour-distance borders or a U-matrix, if these are important for a particular workflow comparison. They can remain part of `M_som` and `V_som` at the present abstraction level. - The workflow need not include implementation details such as specific normalisation methods, SOM training rates, or named distance algorithms. - The phrase “cluster exploration” is suitable, but SOM cells should not always be equated with final clusters. The paper explicitly notes that a cluster may consist of several similar neighbouring cells. --- ### Alignment with Paper | Aspect | Assessment | Notes | |---|---|---| | SOM-based clustering, quantisation, and 2D embedding | Pass | The core SOM training, cell assignment, and topology-preserving grid are represented appropriately. | | Quality and interestingness measures | Partial | Quantisation error and metadata interestingness are present, but explicit recommendation cues are missing. | | Adaptation of preprocessing, metrics, and SOM parameters | Pass | Represented through `S_analysis_strategy` and iterative strategy refinement. | | Data partitioning and drill-down | Partial | A current subset is selected, but branching and retaining multiple partitions are not represented. | | Manual labels and noise annotation | Partial | Mentioned indirectly through strategy, but no annotation artifact or transform is represented. | | Analytic provenance and flow-graph reflection | Fail | The central retained graph of SOM analyses, links, comparisons, and annotations is absent. | | Reflection and verification tasks | Fail | The final reflection accesses only the current loop state, not accumulated analysis results. | | Domain terminology | Pass | SOM cells, pitch contours, metadata, quality, interestingness, partitions, and drill-down align with the paper. | ### Conclusion The representation accurately models a single iterative SOM-analysis path, including adaptation and data partitioning. Its main deficiency is that it turns SOMFlow into a replace-and-retrain workflow, whereas the paper’s key contribution is a retained, branching, visual analytic-provenance graph of multiple SOM results. Add explicit artifacts and transforms for retained SOM runs, parent-child lineage, flow-graph visualisation, and in-loop reflection. Also correct the missing model and assignment dependencies in quality computation and visualisation.