The workflow supports data analysts in discovering patterns within multidimensional datasets by incorporating their tacit domain knowledge into embedding projections. The core mechanism externalises knowledge as sample class labels and adds a classification loss to the embedding network, pulling same-class samples with similar features closer together. This produces more compact (significant) and class-consistent (interpretable) visual structures than unsupervised embeddings alone, enabling systematic pattern discovery through iterative knowledge refinement and structure explanation.
The input is a multidimensional dataset (e.g., countries with monthly COVID-19 case counts, or customers with purchase behaviours) where each sample has data features for embedding and descriptive attributes for knowledge externalisation. The analyst selects a descriptive attribute (e.g., continent, income level, median age) and discretises its value range into intervals to form sample groups — this converts tacit knowledge ("geographically proximate countries should have similar epidemic situations") into explicit class labels. The Knowledge Editor provides a tree-based interface where groups are encoded as coloured child nodes of the root, with K-means clustering of attribute-value bars as a reference that the analyst can manually adjust.
The embedding network takes the class labels and data features as input. It jointly optimises a reconstruction loss (preserving data feature similarity) and a classification loss (encoding class membership) weighted by a Classification Loss Ratio (CLR) parameter α. The resulting embeddings are projected to 2D via UMAP, producing a projection where same-class samples with similar features form compact clusters, while counterexamples (same class but different features) remain separated. The analyst can interactively adjust CLR: low values (5–10%) reveal only the strongest patterns; high values (>30%) emphasise class structure and expose outliers that resist grouping.
The analyst inspects the projection for visual structures — clusters and outliers — made significant and interpretable by the knowledge incorporation. They lasso-select one or two visual structures and invoke the Pattern Explainer, which trains a classifier to distinguish the selected structures and ranks attributes by SHAP importance. Embedding Factors (EFs — dimensions of the data feature) explain how structures differ ("countries in this cluster have more confirmed cases in February"), while Classification Factors (CFs — grouping attribute values) explain which groups the structures represent ("these are European countries").
Based on findings, the analyst assesses whether further exploration is needed. They may refine knowledge by creating subclasses (clicking a tree node to sub-divide that group on another attribute), filtering out irrelevant samples, deleting classes, adjusting CLR, or selecting a completely different attribute for knowledge externalisation. Each refinement triggers recomputation of embeddings and projection, potentially revealing finer-grained patterns (e.g., within the high-income group, customers with and without children have different spending on wine). This cycle repeats — externalising knowledge, embedding, exploring, explaining, refining — until all patterns of interest have been identified and characterised.
The analyst synthesises findings accumulated across multiple perspectives and refinement levels: confirmed group-level patterns (e.g., continental differences in epidemic severity), discovered subgroup distinctions (e.g., high-income parents reduce gold and meat spending), identified outliers (e.g., low-income customers without children who spend like high-income ones), and correlations between descriptive attributes and data feature patterns.