Supervising unsupervised data grouping — the IteraScope display
This is a research demonstrator accompanying the paper, not a finished system. SmartIterator treats the sequence of grouping results across a parameter sweep as one analytical object. Choose a data set to open its sweep. Each data set was computed once with the Python reference engine; the page then runs entirely in your browser (nothing is sent anywhere).
Companion to: G. Andrienko & N. Andrienko, SmartIterator: Visual Analytics Workflows for Supervising Unsupervised Data Grouping.
Open a data set and use Explore… in the top bar: choose the method, what to sweep (e.g. the number of clusters, a neighbourhood radius, a random seed, a resolution) and in which range, which features to group on, and press Run. The computation happens in your browser (Web Workers); the result is added to the sweep menu under Your explorations. The sweeps that the paper walks through are listed under Paper scenarios (precomputed), and Start from in the Explore dialog copies the settings of any of them into the form, so you can re-run or vary them. Results from your own runs may differ in detail from the precomputed ones (library defaults, random seeds, sampling in large data).
Partition methods for numeric data: k-means, k-medoids, k-medians, Gaussian mixture (soft assignments: confidence is a posterior probability) and Ward (one dendrogram cut at every K, so partitions are nested). When a sweep is over seeds, the runs have no natural order, and Order runs in the top bar reorders them along X: by similarity (similar runs next to each other) or best-first by any quality metric.
Lecture data sets. The two cherry-blossom data sets (weekly counts, photo locations) come from the INM433 practicals. For numeric methods, Scaling offers z-score, min–max or none. For located items (the photos, and the VAST messages) DBSCAN uses the recipe of the lecture with the real spatial distance: the great-circle distance in km between the locations (not a distance in degrees), a radius in km, optionally the difference in days with its own radius; relative distances are combined by maximum, Euclidean or Manhattan. A space-time cube (drag to rotate; with and without noise) shows when the clusters were active. And Items: members of the clusters of the active iteration runs a second round, for example space only on the members of a space-time clustering. The colour menu includes arbitrary colours to compare with similarity-aware colours. See docs/lecture5-guide.md.
Other tools: Guide (the six-phase workflow, checkpoints and a decision record), Annotate (callout letters, group labels such as “24.7”, notes; export figures as PNG), and ?mode=baseline&study=1 for the study mode.
Please read first: this is a demonstrator, not a product. Loading your own data works, but only part of what you see is generic.
src/60_domain.js, src/75_vast.js, src/88_authors.js as templates). The method side (sweeps, scores, archetypes) is independent of the domain views.A data set is a folder data/<id>/ (a meta.json and a payload.json.gz) listed in data/datasets.json. You do not need to precompute anything: the converter below writes the folder from a CSV file, and the browser computes the first sweep when the data set is opened. Nothing is uploaded anywhere; the CSV is read on your machine.
numpy, pandas; scikit-learn for text; networkx for networks):| Your data | Command | Methods offered |
|---|---|---|
| Table — one row per item, numeric columns | python3 tools/add_dataset.py table regions.csv \ --id regions --features pct_young,pct_old,income \ --name-col name [--also gdp,population] | k-means, k-medoids, k-medians, Gaussian mixture, Ward, DBSCAN. Views: parallel-coordinates profiles. |
| Points — located items, optionally with a time | python3 tools/add_dataset.py points events.csv \ --id events --lon lon --lat lat [--time when] \ [--basemap map.png --bounds XMIN XMAX YMIN YMAX] | DBSCAN with the great-circle distance in km (time optional), k-means, k-medoids, k-medians, Gaussian mixture, Ward (position in km and time can be grouped on). Views: map, space-time cube and hourly time view (with a time), season view. |
| Text — one row per document | python3 tools/add_dataset.py text docs.csv \ --id docs --text-col abstract \ [--title-col title] [--year-col year] | NMF topic modelling. Views: word clouds, topics over time. |
| Network — one row per edge, optional node table | python3 tools/add_dataset.py network edges.csv \ --id net --source from --target to [--weight w] \ [--nodes nodes.csv --node-col name --text-col about] | Louvain community discovery, NMF. Views: node-link network, word clouds. Without node texts, a node’s profile is the set of hubs it is linked to. |
python3 serve.py) and reload this page: the new data set appears as a card. Opening it computes a first sweep with default settings (about 5–15 s); change anything with Explore….--title, --description, --credit (shown on the card), --noun (what one item is called, e.g. regions), --order. To remove a data set, delete its folder and its id in data/datasets.json.Limits in the browser. Tested with up to 60,000 points (DBSCAN, k-means; Ward is limited to 5,000 items), 3,800 documents (NMF) and 7,100 nodes with 24,600 edges (Louvain); at most 80 iterations per sweep. DBSCAN on located items uses a grid for the neighbour search (tested with 60,000 points); DBSCAN on more than two table features is limited to 8,000 items. Larger data may be slow or fail: sample it before conversion.
Without Python. Any tool can write the files directly. The minimum is meta.json ({"id","title","description","source","family","domain","items","sweeps":[],"payload":"payload.json.gz","payload_bytes","payload_gz_bytes","built"}) and a gzipped payload.json of the form {"schema":"smartiterator-payload/1","meta":{…},"items":{…},"sweeps":[]}. The fields of items for each data type, and the format of precomputed sweeps (which you can also ship to skip the first computation), are in docs/payload-schema.md; tools/build_eu.py, build_vis.py, build_vast.py and build_authors.py are worked examples of precomputed data sets.
Sharing. The whole folder is static: host it on any web server or GitHub Pages. Data in a hosted copy are public to everyone who can open the page.