The workflow supports eye tracking analysts in annotating Areas of Interest (AOIs) in mobile eye tracking recordings — a traditionally time-consuming task because each recording must be labeled individually since AOIs are only visible when in the participant's field of view. The distinguishing characteristic of the approach is that annotation is performed on segment thumbnails projected into a 2D plane (via UMAP), rather than by drawing bounding shapes on video frames. Similar thumbnails cluster together spatially, so the analyst can label many segments across multiple recordings simultaneously through visual grouping.
Input consists of multiple videos from the world-view camera of mobile eye tracking glasses (from one or multiple participants) with gaze coordinates mapped to the video coordinate system. For each video frame, one gaze coordinate is retained (sampling rate matched to video frames), and a quadratic gaze thumbnail (100–200 px width, ≈10–20% of image width, approximating the foveated area) is extracted at the gaze position — these thumbnails become the new units of analysis. For each thumbnail, a 4096-dimensional feature vector is computed using the FC6 layer of AlexNet — a generalizable image descriptor used for both segmentation similarity and projection.
Consecutive gaze thumbnails are compared using inverted cosine distance between their feature vectors. Two slitscans are generated for each video: a local gaze-guided slitscan (vertical scanline through the center of each gaze thumbnail) showing eye movement, and a global slitscan (scanline in the center of the world-view frame) showing body/head movement. An initial segmentation is computed using a default similarity threshold, then rendered together with the slitscans and similarity plot. The analyst iteratively adjusts the per-video similarity threshold (and, if needed, the thumbnail size): values above the threshold mark segment boundaries. Segment boundaries appear as gray boxes on the timeline. The loop continues until each segment corresponds to a coherent dwell on a single AOI (typically summarizing many redundant frames of the same content into one representative segment thumbnail from the middle frame of the segment).
The feature vectors of the representative segment thumbnails from all videos are first collected into a pooled set, and then jointly embedded into 2D using UMAP with cosine distance. The resulting 2D coordinates are contextualized as a spatial arrangement of thumbnail images (projection view), and per-video scarf plots are contextualized as horizontal timelines of colored segment boxes. Similar thumbnails from different recordings and time spans are placed in spatial proximity, forming visual clusters that correspond to AOIs. The number-of-neighbors hyperparameter k can be adjusted (higher k reveals more global structure; lower k yields tighter local clusters). Non-deterministic projection results are compared across runs for stability. Overlapping thumbnails are resolved on demand: when the analyst selects a group, selected elements are rearranged into an occlusion-free grid for cleaning and labeling.
The analyst defines a list of AOI labels (name + color), either in advance (if AOIs are known) or in a data-driven way as clusters are examined. Multiple thumbnails are selected in the projection (typically a visual cluster), transformed to a grid for validation, and — through two successive knowledge-generating actions — the analyst first picks or defines the appropriate AOI label, then assigns it to the cleaned selection. Labeled thumbnails receive a semi-transparent color overlay and immediately appear as colored bars in the scarf plot timelines of the corresponding videos. Selection is bi-directional: segments can be selected on the timeline to locate them in the projection, and thumbnails in the projection can be selected to jump to their video frame. Labeled elements can be hidden in the projection to focus on remaining segments. The process iterates until all relevant segments are labeled.
The result is a set of annotated scanpaths (colored scarf plots) showing which AOIs each participant investigated and in which order. This forms input for subsequent analysis methods (dwell times, scanpath comparison, statistical AOI analysis) and can serve as ground-truth labels for supervised machine learning approaches to AOI recognition in future recordings.