Appendix D: Questionnaire

Evaluation of the notebook and the web demonstrator on public Flickr data. A single anonymous questionnaire in five parts. All answers are selections from options.

Summary of responses (Appendix E) · Supplementary materials

Procedure, participation and ethics

Students worked with the notebook on public Flickr data. Afterwards they could try the web demonstrator. No tasks were set for it. Both were used as delivered.

Participation was entirely voluntary. The questionnaire is a PDF form. Students who wanted to take part filled it in and uploaded the anonymized PDF to an anonymous FTP server that does not record IP addresses or user names. The form contains no names, e-mail addresses or participant codes. The study relies on voluntary, completely anonymous feedback, and no ethics approval was sought.

Information and consent (first page of the form)

☐ I have read the information and agree to take part.

Part A: Background

#ItemFormat
A3Have you applied a clustering algorithm to data yourself (outside a tutorial)?Yes / No; if yes, select the description that fits
A4Before this class: had you ever run the same clustering method with several different parameter settings on the same dataset?Yes / No; if yes, select how you decided which result to use
A5Self-rated familiarity with Python / scikit-learn1 (none) – 5 (fluent)
A6Self-rated familiarity with clustering validity indices (silhouette, SSE, etc.)1–5
A7Experience with Jupyter or Colab notebooks1 (none) – 5 (daily use)
A8Experience with interactive web-based visualization tools1 (none) – 5 (daily use)
A9Did you attend the lecture on clustering?Yes / No

There is no pre-test, because the questionnaire is voluntary and given once, so it does not measure change.

Part B: The notebook

#ItemFormat
B1How far did you get in class?Part 1 only / Part 1 and part of Part 2 / Parts 1 and 2 / less than Part 1
B2Did you continue with the notebook after the class?No / Yes, Part …
B3The time in class was sufficient.Yes / Partly / No

Usefulness of components

“How useful was this component for understanding how the clustering result depends on the parameters?” 1 (not useful) – 5 (very useful) / did not use.

#Component
B4aSweep plots (number of clusters, noise share, SSE, silhouette)
B4bMaps and space–time cubes
B4cPersistence and transition tables and plots (comparing runs)
B4dSilhouette (assignment confidence) plots
B4eCalendar and profile plots of the weekly groups
B4fInteractive sliders
B4gPrediction checkpoints (predict, then reveal)
B4hThe notes “What this shows”
B4iAudit exercise (Part 3)

Complexity, guidance and interaction

Agreement, 1 = strongly disagree … 5 = strongly agree. Items marked (R) are negatively worded.

#Statement
B5I knew what to do next.
B6There was too much information at once. (R)
B7The available options were manageable.
B8The notes helped me understand what to look at.
B9I needed more guidance on how to interpret the plots. (R)
B10I could compare results from different parameter settings.
B11I could see how groups changed at neighbouring parameter values.
B12The interface responded quickly enough when I changed parameters.
B13The prediction checkpoints helped me check my understanding.
B14I could use the notebook without reading the code.
B15Mental demand of working with the notebook (1 very low, 5 very high).
B16I understood what persistence and transitions showed.
B17I understood what assignment confidence showed.

Part C: The web demonstrator (only if used)

Everybody answers C1. C2 is for “No”, C3–C16 for “Yes”.

#ItemFormat
C1Did you use the web demonstrator?Yes / No
C2If no: why not?Select one: did not know how to start / not interested / other: focused on the notebook / …
C3How long did you use it in total?< 15 min / 15–30 / 30–60 / > 60
C4What did you do? (select all)Select all that apply: looked at the precomputed sweeps of the Flickr data / ran my own sweeps (Explore) / looked at other example data sets

Usefulness of components

As in B4, for the web demonstrator: 1 (not useful) – 5 (very useful) / did not use.

#Component
C5aQuality-metric charts
C5bTransition (Sankey) view
C5cConfidence violin plots
C5dGroup embedding (projection of the groups)
C5eMap and space–time cube
C5fCalendar and profile views
C5gControls to change parameters
C5hExplore mode (running my own sweeps)
C5iWorkflow guide

Complexity, guidance and interaction

Agreement, 1 = strongly disagree … 5 = strongly agree. Items marked (R) are negatively worded.

#Statement
C6I knew what to do next in the web demonstrator.
C7There was too much information at once in the web demonstrator. (R)
C8The available options in the web demonstrator were manageable.
C9The web-demonstrator notes helped me understand what to look at.
C10I needed more guidance on how to interpret the web-demonstrator plots. (R)
C11I could compare results from different parameter settings in the web demonstrator.
C12I could see how groups changed at neighbouring parameter values in the web demonstrator.
C13The web demonstrator responded quickly enough when I changed parameters.
C14The workflow guide helped me decide what to look at next.
C15I could start without help.
C16Mental demand of working with the web demonstrator (1 very low, 5 very high).

Part D: Comparison (only if both tools were used)

For each question: Notebook / Web demonstrator / No difference.

#Question
D1Which helped you more to understand how the result changes with the parameters?
D2Which helped you more to understand assignment confidence?
D3Which gave you better guidance on what to do next?
D4Which made it easier to try your own ideas?
D5Which would you prefer for learning about clustering in a course?
D6Which would you prefer for analysing your own data?

All students used the notebook first and the web demonstrator second. Preferences cannot be separated from order and from familiarity with the data.

Part E: Concepts and comprehension

Conceptual statements

Agreement, 1 = strongly disagree … 5 = strongly agree. Items marked (R) are reverse-keyed: agreement indicates the single-run view.

#Statement
E1A clustering result is best judged by computing a validity index and taking the highest value. (R)
E2Two parameter settings that produce similar validity scores can support meaningfully different conclusions about the data.
E3If a cluster structure appears only for one parameter setting and disappears for neighbouring settings, that is evidence against relying on it.
E4Once the optimal number of clusters has been determined, examining other numbers adds little information. (R)
E5Knowing how confidently each individual item is assigned to its cluster is important for interpreting the result.
E6It can be a valid analytical outcome to conclude that none of the tested configurations should be used.

Self-reported change

“Because of the class, I would now …” Agreement, 1 = strongly disagree … 5 = strongly agree.

#Statement
E7… run the same method with several parameter settings before choosing one.
E8… compare the groups across settings and not only the scores.
E9… check how confidently the items are assigned to their groups.

Comprehension probes

Each probe is answered by selecting one of several answer options. Every option has a score of 0, 1 or 2. The options chosen by respondents are listed in Appendix E.

#Prompt
E10What would you advise before relying on a clustering result from one DBSCAN parameter setting?
E11What does it mean if a group occurs only at one parameter setting, for example at K = 6 but not neighbouring values?
E12What can low assignment confidence indicate?

The probes test understanding at the time of the questionnaire. They do not show learning gains.

Flow and skip logic

Parts C and D are filled in only by students who used the web demonstrator.

PartSectionShown toItems
ABackgroundAll7
BNotebookAll17 (B4 has 9 ratings)
CWeb demonstratorC1 for everybody. C2 only if “No”. C3–C16 only if “Yes”.up to 16
DComparisonOnly if C1 = Yes6
EConcepts and comprehensionAll12