The paper presents reCAPit, a modular multimodal framework for recording, analysing, and reporting findings from collaborative design workshops. The workflow combines AI-based artifact extraction with interactive visual analysis to help workshop organisers understand how collaboration shaped outcomes.
Data acquisition. The framework records up to four data streams from a shared creative working area: top-down and side video, audio from a scene microphone, gaze from mobile eye-tracking glasses worn by participants, and timestamped digital notes taken by an observer. Predefined static areas of interest (AOIs) partition the working area into semantically meaningful regions.
Artifact extraction. From video, background subtraction combined with hand tracking detects activity (pointing, drawing, interacting) within each AOI. From audio, the Whisper speech-recognition model produces a transcript; speakers are manually assigned and grouped into roles. From gaze, fixation hits on AOIs quantify per-participant visual attention over time. A two-step multimodal segmentation partitions the workshop temporally: first, the PELT change-point algorithm segments the multivariate activity/attention time series into coarse intervals; then, within each interval, text embeddings (multilingual GTE model) of transcript chunks are compared by cosine similarity to detect topic shifts, yielding refined topic segments. An LLM generates an initial title for each segment.
Visualisation. The extracted artifacts are presented in coordinated views: a multimodal streamgraph displays activity and/or attention over time with AOIs as categories; a timeline shows topic segments as shaded areas, participant utterances as Gantt-chart blocks colour-coded by role, and scarf plots for gaze; heatmaps spatially aggregate activity or attention over selected periods; and topic cards summarise each segment with title, quotes, screenshots, notes, and donut-chart statistics of speaker time, activity, and attention distribution.
Interactive exploration. The analyst enters an iterative cycle comprising four complementary strategies: (1) Topic overview—browse topic cards to grasp the workshop's thematic structure; (2) Multimodal inspection—examine the streamgraph to identify phases of concentrated activity or shared attention and drill down into raw data (video, transcript, gaze); (3) Filtering—apply keyword search to surface relevant topic segments; (4) Authoring—populate topic cards with selected quotes, video screenshots (optionally with heatmap overlays), personal notes, and bookmarks. The cycle repeats with different keywords, time foci, and data sources until the analyst has assembled a coherent summary that documents how workshop outcomes were derived.