This paper presents a visual analytics system for semi-automatic annotation of eye-tracking fixation data with AOI (Area of Interest) labels. It combines image-based fixation representation, dimensionality reduction, an active learning classifier, and uncertainty-aware visualization to support efficient labeling with trust building in the ML predictions.
Input. Multiple recordings, each comprising a video stimulus and gaze data from one participant. Gaze data consists of spatio-temporal 2D coordinates mapped onto the video. The system supports both stationary and mobile (head-mounted) eye tracking.
Preprocessing. Three automated steps prepare the data: (1) image-based fixation detection aggregates consecutive gaze samples into fixations by binning visually similar image patches (similarity threshold 1.1); (2) for each gaze sample, an image thumbnail is cropped from the video frame centred at the gaze point; (3) a pretrained ResNet18 extracts a 512-dimensional feature vector from each thumbnail, serving as input to both the dimensionality reduction and the classifier.
Initial Labeling. A subset of fixation thumbnails is projected into a 2D embedding via unsupervised UMAP. The analyst identifies clusters of visually similar thumbnails, multi-selects fixations belonging to the same semantic region, and assigns AOI labels. AOI categories may be predefined or identified iteratively. After initial labels exist, a semi-supervised re-projection (UMAP with labels or LDA) improves cluster separation. The resulting embedding forms the analyst's mental map for subsequent phases.
Retrospective Analysis. The analyst explores the labelled embedding using an interactive lens (incoming mode: show fixations with samples in brushed area; outgoing mode: show all samples of fixations whose centre is in the brushed area). A multi-class heatmap (per-class KDE, colour-coded) reveals class distributions. An overlap indicator highlights regions where class densities overlap, signalling potential label ambiguity or classification difficulty. The analyst fixes erroneous labels and identifies semantic relations between classes (e.g., fixations straddling adjacent AOIs).
Classifier Training. A classification head (FC layers with early fusion of N gaze patches per fixation) is trained on the manually labelled fixations using the frozen ResNet18 features. The model predicts a probability distribution over C AOI classes for each fixation.
Active Learning Loop. New (unlabelled) fixations are streamed into the system and classified. Prediction uncertainty is quantified as entropy of the output distribution. The system suggests the most uncertain predictions for manual inspection. EyeFlower glyphs—circular thumbnails surrounded by coloured petals whose inner growth encodes class confidence—convey the classifier's output per fixation. The analyst inspects suggested candidates, approves correct predictions or corrects erroneous ones, and the classifier is fine-tuned with increased sample importance on corrected instances. The cycle repeats over multiple iterations, with retrospective checks on high-confidence predictions for trust calibration, until the analyst trusts the classifier sufficiently.
Trust Building. Trust is built through multiple cues: (a) new fixations placed correctly inside existing clusters with high-confidence EyeFlowers; (b) decreasing overlap areas in the multi-class heatmap; (c) shortening uncertainty candidate lists across iterations; (d) high-confidence correct predictions on known-difficult inputs (trust gain); (e) ability to drill down to individual gaze samples for explainability.