SmartIterator

Supervising unsupervised data grouping — the IteraScope display

Using SmartIterator

Explore freely, or reproduce the paper

Open a data set and use Explore… in the top bar: choose the method, what to sweep (e.g. the number of clusters, a neighbourhood radius, a random seed, a resolution) and in which range, which features to group on, and press Run. The computation happens in your browser (Web Workers); the result is added to the sweep menu under Your explorations. The sweeps that the paper walks through are listed under Paper scenarios (precomputed), and Start from in the Explore dialog copies the settings of any of them into the form, so you can re-run or vary them. Results from your own runs may differ in detail from the precomputed ones (library defaults, random seeds, sampling in large data).

Partition methods for numeric data: k-means, k-medoids, k-medians, Gaussian mixture (soft assignments: confidence is a posterior probability) and Ward (one dendrogram cut at every K, so partitions are nested). When a sweep is over seeds, the runs have no natural order, and Order runs in the top bar reorders them along X: by similarity (similar runs next to each other) or best-first by any quality metric.

Lecture data sets. The two cherry-blossom data sets (weekly counts, photo locations) come from the INM433 practicals. For numeric methods, Scaling offers z-score, min–max or none. For located items (the photos, and the VAST messages) DBSCAN uses the recipe of the lecture with the real spatial distance: the great-circle distance in km between the locations (not a distance in degrees), a radius in km, optionally the difference in days with its own radius; relative distances are combined by maximum, Euclidean or Manhattan. A space-time cube (drag to rotate; with and without noise) shows when the clusters were active. And Items: members of the clusters of the active iteration runs a second round, for example space only on the members of a space-time clustering. The colour menu includes arbitrary colours to compare with similarity-aware colours. See docs/lecture5-guide.md.

Other tools: Guide (the six-phase workflow, checkpoints and a decision record), Annotate (callout letters, group labels such as “24.7”, notes; export figures as PNG), and ?mode=baseline&study=1 for the study mode.

Add your own data demonstrator

Please read first: this is a demonstrator, not a product. Loading your own data works, but only part of what you see is generic.

  • Generic (any data): the quality-metric curves, the Sankey transitions, the violins of assignment confidence and outlier score, recurrent archetypes, the group embedding, group comparison, the workflow guide and the annotation layer.
  • Specific to the bundled use cases: the domain views. They depend on information the converter cannot invent. A choropleth map needs polygon geometry (the EU example ships a simplified NUTS-3 geometry prepared with a dedicated script from the Eurostat boundaries; a table you add gets parallel-coordinates profiles only). A point map needs coordinates in a projection that can be drawn linearly and, for a basemap, an image with known bounds that you supply (no map tiles, no projection handling; longitude/latitude are only corrected by cos(latitude)). The time view assumes a short period (a few days, hourly bins). Word clouds assume English text with a fixed 500-term vocabulary (300 words + 200 bigrams); other languages, other vocabularies or other domain-specific text processing need code changes. The network layout is a plain spring layout.
  • Not provided: data validation beyond basic checks, other geographic or time views, coordinate-system conversion, persistence of your explorations between sessions, or support for very large data (see limits below). Error messages are minimal. Only desktop Chromium has been tested.
  • Adding a use-case-specific view means writing code (see src/60_domain.js, src/75_vast.js, src/88_authors.js as templates). The method side (sweeps, scores, archetypes) is independent of the domain views.

A data set is a folder data/<id>/ (a meta.json and a payload.json.gz) listed in data/datasets.json. You do not need to precompute anything: the converter below writes the folder from a CSV file, and the browser computes the first sweep when the data set is opened. Nothing is uploaded anywhere; the CSV is read on your machine.

  1. Put your CSV somewhere and run the converter (Python 3 with numpy, pandas; scikit-learn for text; networkx for networks):
Your dataCommandMethods offered
Table — one row per item, numeric columns
python3 tools/add_dataset.py table regions.csv \
  --id regions --features pct_young,pct_old,income \
  --name-col name [--also gdp,population]
k-means, k-medoids, k-medians, Gaussian mixture, Ward, DBSCAN. Views: parallel-coordinates profiles.
Points — located items, optionally with a time
python3 tools/add_dataset.py points events.csv \
  --id events --lon lon --lat lat [--time when] \
  [--basemap map.png --bounds XMIN XMAX YMIN YMAX]
DBSCAN with the great-circle distance in km (time optional), k-means, k-medoids, k-medians, Gaussian mixture, Ward (position in km and time can be grouped on). Views: map, space-time cube and hourly time view (with a time), season view.
Text — one row per document
python3 tools/add_dataset.py text docs.csv \
  --id docs --text-col abstract \
  [--title-col title] [--year-col year]
NMF topic modelling. Views: word clouds, topics over time.
Network — one row per edge, optional node table
python3 tools/add_dataset.py network edges.csv \
  --id net --source from --target to [--weight w] \
  [--nodes nodes.csv --node-col name --text-col about]
Louvain community discovery, NMF. Views: node-link network, word clouds. Without node texts, a node’s profile is the set of hubs it is linked to.
  1. Start the server (python3 serve.py) and reload this page: the new data set appears as a card. Opening it computes a first sweep with default settings (about 5–15 s); change anything with Explore….
  2. Common options: --title, --description, --credit (shown on the card), --noun (what one item is called, e.g. regions), --order. To remove a data set, delete its folder and its id in data/datasets.json.

Limits in the browser. Tested with up to 60,000 points (DBSCAN, k-means; Ward is limited to 5,000 items), 3,800 documents (NMF) and 7,100 nodes with 24,600 edges (Louvain); at most 80 iterations per sweep. DBSCAN on located items uses a grid for the neighbour search (tested with 60,000 points); DBSCAN on more than two table features is limited to 8,000 items. Larger data may be slow or fail: sample it before conversion.

Without Python. Any tool can write the files directly. The minimum is meta.json ({"id","title","description","source","family","domain","items","sweeps":[],"payload":"payload.json.gz","payload_bytes","payload_gz_bytes","built"}) and a gzipped payload.json of the form {"schema":"smartiterator-payload/1","meta":{…},"items":{…},"sweeps":[]}. The fields of items for each data type, and the format of precomputed sweeps (which you can also ship to skip the first computation), are in docs/payload-schema.md; tools/build_eu.py, build_vis.py, build_vast.py and build_authors.py are worked examples of precomputed data sets.

Sharing. The whole folder is static: host it on any web server or GitHub Pages. Data in a hosted copy are public to everyone who can open the page.