SOMFlow: ATWL workflow review

ATWL Workflow Review Report

Paper Title: SOMFlow: Guided Exploratory Cluster Analysis with Self-Organizing Maps and Analytic Provenance
Workflow ID: somflow-guided-cluster-exploration

Overall Assessment

The representation captures SOMFlow’s iterative SOM analysis cycle well: training a SOM, assigning data to cells, computing quality and metadata-interestingness measures, visual inspection, assessment, strategy adaptation, and data partitioning.

However, it misses the paper’s central methodological contribution: retaining, visualising, navigating, and comparing the branching history of SOM analyses in an analytic-provenance flow graph. The current workflow overwrites the current dataset and patterns, so the claimed post-loop reflection on accumulated analysis results is not supported by the declared dataflow.

Recommendation:
- [ ] Approve as-is
- [ ] Approve with minor revisions
- [x] Requires revision
- [ ] Reject — major rework needed

Strengths

The main SOM workflow is correctly represented as build-model → define-unit → contextualise → characterise → visualise → abstract → assess.
M_som appropriately captures SOMs as a model combining prototype-based quantisation, clustering, and a topology-preserving embedding.
Defining SOM cells and cell-assignment labels through the same define-unit transform is appropriate.
D_cells as group/cluster entities embedded in a relational structure appropriately represents SOM cells and their neighbourhood topology.
The representation correctly includes both quantisation quality and metadata-related interestingness as computed guidance measures.
The strategy feedback loop is structurally valid: T_decide_strategy produces an updated specification and the assign: statement closes the feedback loop.
The input data and case-study terminology—utterances, pitch contours, metadata, SOM cells, and interestingness—align well with the paper.

Critical Issues (Must Fix)

Issue 1: The analytic-provenance flow graph and branching analysis history are missing

Location: L_drilldown, T_define_next, T_reflect

Problem: The paper’s principal contribution is not only sequential drill-down. SOMFlow retains all SOM results, their parent-child relations, filters, splitters, and selected subsets in a branching flow graph. This graph enables comparison, navigation, provenance inspection, annotations, linking and brushing across SOMs, and reflection on prior decisions.

The current workflow instead replaces the working dataset:

assign:
  D_utterances := D_next

It also replaces P_patterns and K_assessment on every iteration. Consequently, T_reflect can access only the most recent pattern and assessment, despite claiming to compare findings “accumulated across the drill-down analysis.”

Correction: Represent retained SOM analysis runs and their lineage explicitly, and visualise them during the loop.

artifact D_som_runs : entities
  internal structure: elementary
  embedment: relational structure
  description: "Retained SOM analysis runs, including the initial SOM and
    all subsequently created, retrained, filtered, or partitioned SOMs"

artifact F_run_lineage : feature(D_som_runs)
  value structure: relational configuration
  representation form: "directed parent-child analysis flow graph"
  description: "Relations between SOM runs indicating retraining,
    filtering, splitting, or selection-based partitioning operations"

transform T_record_analysis_step :
  intent: define-unit
  manner: "retain current SOM result as a provenance-graph node"
  input: D_som_runs, F_run_lineage, M_som, D_utterances,
    S_analysis_strategy
  output: D_som_runs_r, F_run_lineage_r
  actor: machine
  description: "Add the current SOM analysis result and its relation to
    predecessor runs to the retained analytic-provenance graph"

artifact D_som_runs_r : entities
  internal structure: elementary
  embedment: relational structure
  description: "Updated collection of retained SOM analysis runs"

artifact F_run_lineage_r : feature(D_som_runs_r)
  value structure: relational configuration
  representation form: "directed parent-child analysis flow graph"
  description: "Updated provenance relations between retained SOM runs"

assign:
  D_som_runs := D_som_runs_r
  F_run_lineage := F_run_lineage_r

Add a provenance visualisation and use it for reflection:

transform T_visualise_flow :
  intent: visualise
  manner: "analytic-provenance flow graph with embedded SOM views"
  input: D_som_runs, F_run_lineage
  output: V_som_flow
  actor: machine
  description: "Display retained SOM analyses as graph nodes connected by
    data-flow and partitioning relationships, supporting navigation,
    comparison, annotation, and provenance inspection"

artifact V_som_flow : visualisation(D_som_runs, F_run_lineage)
  layout: "directed flow graph"
  form: "graph nodes containing SOM thumbnails and directed links"
  encoding: "nodes represent retained SOM runs; links represent data-flow
    relations; link width represents the number of transferred items;
    annotations and metadata overlays support comparison"
  description: "Analytic-provenance graph showing the evolution and
    branching structure of SOM-based cluster exploration"

Then revise reflection:

transform T_reflect :
  intent: abstract
  manner: "compare retained SOM results and analysis branches"
  input: V_som_flow, D_som_runs, F_run_lineage
  output: P_overall_relations
  actor: human
  description: "Compare current and earlier SOM analyses, branches, and
    data partitions to identify verified relations and assess the
    analytical process"

Why: Retaining and visually comparing multiple SOMs is essential to SOMFlow. It cannot be represented as a linear overwrite-only drill-down process.

Issue 2: Several transforms and visual encodings omit required inputs

Location: T_compute_cell_measures, T_visualise_som, V_som

Problem: Quantisation error requires the SOM prototypes from M_som, but T_compute_cell_measures does not consume the model. Likewise, the SOM visualisation claims to show prototype vectors, topology overlays, and items assigned to cells, but it omits both M_som and F_cell_assignment.

Correction:

transform T_compute_cell_measures :
  intent: characterise
  manner: "compute quantization error and metadata interestingness per
    cell"
  input: D_cells, F_cell_assignment, D_utterances, M_som
  output: F_cell_quality, F_cell_interestingness
  actor: machine
  description: "Compute per-cell quantization error from assigned items
    and SOM prototypes, and metadata-interestingness scores for
    identifying relations between cell content and metadata"

transform T_visualise_som :
  intent: visualise
  manner: "SOM grid view with prototype time series, topology overlays,
    metadata colouring, and ranked attribute recommendations"
  input: A_cell_layout, D_cells, D_utterances, M_som,
    F_cell_assignment, F_cell_quality, F_cell_interestingness
  output: V_som
  actor: machine
  description: "Render SOM cells with prototype and assigned-item time
    series, topology and quality overlays, metadata colouring, and
    ranked recommendations for further analysis"

artifact V_som : visualisation(A_cell_layout, D_cells, D_utterances,
    M_som, F_cell_assignment, F_cell_quality, F_cell_interestingness)
  layout: "two-dimensional SOM grid"
  form: "cell glyphs with prototype and item line charts, quality or
    metadata overlays, and recommendation thumbnails"
  encoding: "cell position from A_cell_layout; prototype line and training
    history from M_som; assigned item traces and bandwidth from
    D_utterances and F_cell_assignment; cell shading from
    F_cell_quality or f_metadata; density from f_cell_size; ranked
    thumbnails from F_cell_interestingness"
  description: "Interactive SOM view supporting inspection of cluster
    structure, prototype quality, and metadata relations"

Why: ATWL visualisations should reference the artifacts that provide their marks, overlays, and interaction-relevant membership mappings.

Issue 3: The template and post-loop reflection contradict the represented control flow

Location: Workflow template; T_reflect

Problem: The template includes abstract (reflect) after the drill-down loop. In the paper, reflection occurs continuously through interaction with the provenance graph, not only after the analyst has stopped iterating. Moreover, the current post-loop transform cannot reflect on historical results because all relevant artifacts are overwritten.

Correction: Move provenance visualisation and reflection into the iterative workflow after recording a new analysis run. For example:

template: loop(build-model -> define-unit (assign cells) ->
                 contextualise (grid layout) ->
                 characterise (cell measures) ->
                 visualise (SOM view) -> abstract ->
                 assess -> generate-knowledge (strategy) ->
                 define-unit (partition/retain) ->
                 visualise (provenance flow) ->
                 abstract (reflect)) ->
            generate-knowledge (synthesise)

Why: Reflection and comparison of prior analysis branches are active analytical tasks supported throughout SOMFlow exploration, not a final operation on only the last SOM result.

Recommended Improvements (Should Fix)

Improvement 1: Represent machine-generated partitioning recommendations explicitly

Location: T_compute_cell_measures, T_visualise_som, T_decide_strategy

Problem: The paper’s guidance does not consist only of displaying quality and interestingness. SOMFlow generates explicit recommendations for:

refining high-quantisation-error cells;
extending a selected cell group with similar neighbours;
splitting data by metadata attributes;
splitting data by meta-cluster labels.

The current representation refers generally to “ranked recommendations,” but does not produce an artifact representing these recommendation cues.

Suggestion:

transform T_compute_cell_measures :
  intent: characterise
  manner: "compute quality, interestingness, similarity, and partitioning
    recommendations"
  input: D_cells, F_cell_assignment, D_utterances, M_som
  output: F_cell_quality, F_cell_interestingness,
    F_partition_recommendations
  actor: machine
  description: "Compute cell quality and metadata-interestingness measures,
    identify similar neighbouring cells, and derive candidate refinement
    and partitioning actions"

artifact F_partition_recommendations : feature(D_cells)
  value structure: vector
  value type: {numeric, categorical}
  description: "Per-cell and map-level recommendation scores and action
    categories for refinement, similar-cell selection, metadata splits,
    and cluster-based splits"

Include F_partition_recommendations in V_som and T_decide_strategy.

Why: Guidance based on quality and interestingness is a major SOMFlow contribution and should be represented as machine-derived analytical support, rather than only as an informal visual description.

Improvement 2: Include manual annotation as an analysis operation

Location: Adaptation stage in L_drilldown

Problem: The paper supports user-defined cell and data-item annotations, such as marking measurement-error cells as noise or labelling cells as interesting. These labels can then become metadata and influence later filtering, interestingness computation, and analysis.

This is important in the reported speech-intonation case study, where manual noise annotation is followed by filtering and retraining.

Suggestion:

transform T_annotate_cells :
  intent: characterise
  manner: "interactive assignment of user-defined metadata labels"
  input: D_cells, F_cell_assignment, V_som
  output: F_user_annotations
  actor: human
  description: "Assign analyst-defined labels, such as interesting,
    uninteresting, or measurement-error, to cells or their member items"

artifact F_user_annotations : feature(D_utterances)
  value structure: atomic
  value type: categorical
  description: "User-defined item annotations propagated from selected
    SOM cells"

Use F_user_annotations as an input to later measure computation and partitioning.

Why: Manual data enrichment is part of SOMFlow’s adaptation capabilities and supports the paper’s noise-removal workflow.

Improvement 3: Preserve the data-feature schema when deriving subsets

Location: D_next

Problem: D_next is used as the next iteration’s data input, including for SOM training and metadata-based analysis. However, it declares no f_pitch_contour or f_metadata features, despite those features being required downstream.

Suggestion:

artifact D_next : entities
  internal structure: elementary
  embedment: set
  features:
    - id: f_pitch_contour
      value structure: vector
      value type: numeric
      description: "Time series profile retained for each selected item"
    - id: f_metadata
      value structure: vector
      value type: categorical
      description: "Associated metadata retained for each selected item"
  description: "Selected or retained subset of the current data, preserving
    time series and metadata for the next SOM analysis"

Why: The subset must retain the data characteristics consumed by T_train_som, T_compute_cell_measures, and T_visualise_som.

Improvement 4: Align final knowledge provenance with its synthesis inputs

Location: K_findings

Suggestion:

artifact K_findings : knowledge(P_overall_relations, K_assessment)
  representation form: "statements and explanations"
  description: "Documented findings about discovered clusters, metadata
    relations, and data-quality issues identified through iterative
    SOM-based exploration"

If the provenance graph is added, also include it:

artifact K_findings : knowledge(P_overall_relations, K_assessment,
    D_som_runs, F_run_lineage)

Why: T_synthesise consumes both P_overall_relations and K_assessment; the resulting knowledge should reference its declared evidence.

Minor Suggestions (Optional)

The paper includes both grid and force-directed views of the SOM topology. The grid layout is the core representation, so T_grid_layout is appropriate; the force-directed view may be described as an alternative display mode in V_som.
F_cell_interestingness is well typed as a numeric vector because it holds one score per metadata attribute.
Consider adding topological separation measures, such as neighbour-distance borders or a U-matrix, if these are important for a particular workflow comparison. They can remain part of M_som and V_som at the present abstraction level.
The workflow need not include implementation details such as specific normalisation methods, SOM training rates, or named distance algorithms.
The phrase “cluster exploration” is suitable, but SOM cells should not always be equated with final clusters. The paper explicitly notes that a cluster may consist of several similar neighbouring cells.

Alignment with Paper

Aspect	Assessment	Notes
SOM-based clustering, quantisation, and 2D embedding	Pass	The core SOM training, cell assignment, and topology-preserving grid are represented appropriately.
Quality and interestingness measures	Partial	Quantisation error and metadata interestingness are present, but explicit recommendation cues are missing.
Adaptation of preprocessing, metrics, and SOM parameters	Pass	Represented through `S_analysis_strategy` and iterative strategy refinement.
Data partitioning and drill-down	Partial	A current subset is selected, but branching and retaining multiple partitions are not represented.
Manual labels and noise annotation	Partial	Mentioned indirectly through strategy, but no annotation artifact or transform is represented.
Analytic provenance and flow-graph reflection	Fail	The central retained graph of SOM analyses, links, comparisons, and annotations is absent.
Reflection and verification tasks	Fail	The final reflection accesses only the current loop state, not accumulated analysis results.
Domain terminology	Pass	SOM cells, pitch contours, metadata, quality, interestingness, partitions, and drill-down align with the paper.

Conclusion

The representation accurately models a single iterative SOM-analysis path, including adaptation and data partitioning. Its main deficiency is that it turns SOMFlow into a replace-and-retrain workflow, whereas the paper’s key contribution is a retained, branching, visual analytic-provenance graph of multiple SOM results.

Add explicit artifacts and transforms for retained SOM runs, parent-child lineage, flow-graph visualisation, and in-loop reflection. Also correct the missing model and assignment dependencies in quality computation and visualisation.