SmartIterator

SmartIterator (SI) is a visual analytics approach that treats the full sequence of unsupervised grouping results across a parameter sweep as a first-class analytical object. Rather than seeking a single "optimal" parameter configuration, SI guides the analyst through systematic exploration of how data structure emerges, persists, and transforms across configurations—building cumulative understanding that no single run can provide.

The approach is operationalized through IteraScope (IS), a coordinated visual display that combines:

• A quality-metrics chart with semantic color encoding (warm = higher-is-better, cool = lower-is-better)
• A 1D group embedding with Sankey-style transition flows showing splits, merges, and member movements
• Violin plots of membership confidence and outlier scores (globally normalized for cross-group comparison)
• A 2D group embedding with HDBSCAN-detected recurrent archetypes and interactive threshold control
• Word clouds and term-level transition tooltips (for text corpora)
• Domain-linked views (maps, space-time cubes, parallel coordinates, temporal prevalence charts)

Three method-specific workflows are provided—for density-based clustering, partition-based clustering, and NMF topic modeling—each following a common six-phase structure:

Key insight: Each phase contributes to cumulative understanding of data structure. The knowledge gained from studying transitions between configurations—which groups persist, which split, what vocabulary migrates, where boundaries lie—is often as valuable as the final parameter choice itself. The analyst does not merely pick a number; the analyst learns about the data.

Each notebook implements key computational components from the paper (Section 5), including:

• Iterative parameter sweep with parallel execution via ProcessPoolExecutor
• Per-item uncertainty measures (membership probability, outlier score) for each method family
• Per-iteration quality metrics (silhouette, coherence, Davies–Bouldin, noise %, etc.)
• HDBSCAN archetype detection on pooled group-level feature vectors with iterative threshold sweep
• 1D and 2D group embeddings (t-SNE, UMAP, PaCMAP, LocalMAP)
• MDS-based color assignment ensuring consistency across iterations and threshold changes
• Word cloud generation (frequency- and TF-IDF-weighted; transition word clouds with gained/lost terms)
• Temporal prevalence charts (stacked area, line; absolute/proportional; raw/smoothed)

Note: The notebooks do not implement the full IteraScope interactive display (Sankey rendering, violin plots, hover/click interactions, domain-view linking). These are part of the Java-based V-Analytics environment. The notebooks produce the input data files for IS and provide standalone exploratory visualizations.

1. Start Coarse, Refine Progressively

Begin with a wide parameter sweep at coarse resolution (e.g., K = 3…50 with step 1, or ε = 0.05…1.0 with step 0.05) to identify the region of interest. Examine the metrics chart for elbows, peaks, and plateaus; note where complete iterations cluster. Then narrow the range and reduce the step size for a second round. Two to three rounds typically suffice—each informed by the findings of the previous one. This avoids wasting computation on irrelevant regions and gives early orientation before committing to detailed inspection.

2. Let Sankey Bands, Not Metrics Alone, Define Stability

A silhouette peak or coherence plateau narrows the candidate range, but the definitive stability evidence comes from transition flows. A "good" configuration is one where the groups at the selected parameter persist with minimal change at neighboring values—visible as wide, horizontal Sankey bands connecting the same vertical positions across consecutive iterations. Specifically:

• Persistent wide bands: The group is stable—it neither splits nor absorbs neighbors.
• A band splitting into two: Potential meaningful sub-structure emerging at finer granularity. Inspect both children in domain views.
• Multiple thin bands converging into one new group: Possible wastebasket—a group collecting heterogeneous members from multiple sources without a unifying identity.
• A band thinning progressively: The group is dissolving under stricter parameters—its members are borderline.

If a group splits at the very next parameter step, it may be an artifact of the current granularity rather than genuine structure.

3. Use Domain Views to Break Metric Ties

When multiple configurations score similarly on quality metrics, propagate each to domain-linked views and compare. Domain coherence frequently discriminates between statistically equivalent alternatives:

• Spatial data: Do clusters form geographically contiguous regions, or are they scattered? Contiguity is a powerful external validator.
• Temporal data: Do groups exhibit temporal coherence—emerging at specific times, persisting over defined periods?
• Text data: Do word clouds reveal interpretable themes, or are top terms generic and overlapping?
• Multivariate data: Do parallel-coordinates profiles show clearly distinct patterns, or do cluster profiles overlap substantially?

The domain view often reveals the "right" answer instantly when metrics are ambiguous.

4. Inspect What Changes, Not Only What Persists

Transition-class propagation is most informative when applied to understand losses: which members leave a group at the next parameter step, and where do they go? The spatial, temporal, or semantic distribution of lost members reveals the nature of the transition:

• Losses at cluster edges (spatially coherent): The finer configuration is trimming genuine boundary members—the core structure is preserved.
• Losses scattered randomly: The configuration change may be over-fitting to noise.
• Losses forming a coherent sub-group: A genuine sub-structure is emerging—consider whether the finer granularity captures meaningful differentiation.
• Losses with distinctive vocabulary (topic modeling): A sub-theme is separating out—examine whether it merits its own topic.

This "what changes" perspective often generates more domain insight than examining static group composition at a single configuration.

5. Treat Archetype Completeness as Necessary, Not Sufficient

A "complete" iteration (containing representatives of all HDBSCAN-detected archetypes) captures the full diversity of recurrent patterns—making it a strong candidate. However:

• Completeness alone does not guarantee interpretability or confidence. Always verify through Phases 3–5.
• An iteration missing one archetype may still be preferable if domain context explains the absence (e.g., a noise archetype capturing only preprocessing artifacts, or a rare pattern that only appears under extreme parameter settings).
• If no iteration is complete, consider whether the archetype threshold is set too aggressively. Lower the min_cluster_size to allow more archetypes—some of which may represent genuine but infrequent patterns.
• Use the HDBSCAN threshold sweep chart to find a stability plateau where small changes do not alter the archetype count.

6. Verify Seed Robustness for Partition-Based Methods

K-means and other partition-based methods depend on random initialization. Once a promising K is identified:

• Run 20–30 seeds at the selected K.
• Inspect the Sankey view: horizontal bands with no splits or merges = robust result.
• Check silhouette variation: fluctuation > 0.02 across seeds suggests instability.
• In the 2D embedding, verify tight archetype clusters with negligible seed-to-seed drift.
• If one seed produces a structurally different result (visible as crossed Sankey bands), investigate whether it represents a legitimate alternative interpretation or an inferior local minimum.

For topic modeling (NMF), seed sensitivity is typically lower due to the non-negative constraint, but verification remains good practice.

7. Document Rejections Explicitly

When an alternative configuration is rejected, record:

• Which phase provided the discriminating evidence (metrics? transitions? violins? domain views?).
• What specific observation led to rejection (boundary ambiguity? wastebasket pattern? spatial fragmentation?).
• What the rejection reveals about data structure (e.g., "K=24 rejected because the additional clusters shave off geographic periphery without introducing new demographic archetypes—confirming that Europe's demographic typology is adequately captured at K=20").

Documented rejections serve two purposes: they strengthen confidence in the final selection (by demonstrating that alternatives were considered and found wanting), and they contribute to cumulative understanding (by revealing what the data does not support).

8. When in Doubt, Keep Multiple Candidates

The SI workflow does not require converging to a single "winner." If two configurations offer complementary views of the data—e.g., a coarse typology (K=7) and a fine-grained one (K=20)—both may be worth retaining as analytical lenses for different purposes. The knowledge gained from understanding how the coarse groups differentiate into fine-grained ones is itself a valuable analytical product.