Procedure, participation and ethics
Students worked with the notebook on public Flickr data. Afterwards they could try the web demonstrator. No tasks were set for it. Both were used as delivered.
Participation was entirely voluntary. The questionnaire is a PDF form. Students who wanted to take part filled it in and uploaded the anonymized PDF to an anonymous FTP server that does not record IP addresses or user names. The form contains no names, e-mail addresses or participant codes. The study relies on voluntary, completely anonymous feedback, and no ethics approval was sought.
Information and consent (first page of the form)
☐ I have read the information and agree to take part.
Part A: Background
| # | Item | Format |
|---|---|---|
| A3 | Have you applied a clustering algorithm to data yourself (outside a tutorial)? | Yes / No; if yes, select the description that fits |
| A4 | Before this class: had you ever run the same clustering method with several different parameter settings on the same dataset? | Yes / No; if yes, select how you decided which result to use |
| A5 | Self-rated familiarity with Python / scikit-learn | 1 (none) – 5 (fluent) |
| A6 | Self-rated familiarity with clustering validity indices (silhouette, SSE, etc.) | 1–5 |
| A7 | Experience with Jupyter or Colab notebooks | 1 (none) – 5 (daily use) |
| A8 | Experience with interactive web-based visualization tools | 1 (none) – 5 (daily use) |
| A9 | Did you attend the lecture on clustering? | Yes / No |
There is no pre-test, because the questionnaire is voluntary and given once, so it does not measure change.
Part B: The notebook
| # | Item | Format |
|---|---|---|
| B1 | How far did you get in class? | Part 1 only / Part 1 and part of Part 2 / Parts 1 and 2 / less than Part 1 |
| B2 | Did you continue with the notebook after the class? | No / Yes, Part … |
| B3 | The time in class was sufficient. | Yes / Partly / No |
Usefulness of components
“How useful was this component for understanding how the clustering result depends on the parameters?” 1 (not useful) – 5 (very useful) / did not use.
| # | Component |
|---|---|
| B4a | Sweep plots (number of clusters, noise share, SSE, silhouette) |
| B4b | Maps and space–time cubes |
| B4c | Persistence and transition tables and plots (comparing runs) |
| B4d | Silhouette (assignment confidence) plots |
| B4e | Calendar and profile plots of the weekly groups |
| B4f | Interactive sliders |
| B4g | Prediction checkpoints (predict, then reveal) |
| B4h | The notes “What this shows” |
| B4i | Audit exercise (Part 3) |
Complexity, guidance and interaction
Agreement, 1 = strongly disagree … 5 = strongly agree. Items marked (R) are negatively worded.
| # | Statement |
|---|---|
| B5 | I knew what to do next. |
| B6 | There was too much information at once. (R) |
| B7 | The available options were manageable. |
| B8 | The notes helped me understand what to look at. |
| B9 | I needed more guidance on how to interpret the plots. (R) |
| B10 | I could compare results from different parameter settings. |
| B11 | I could see how groups changed at neighbouring parameter values. |
| B12 | The interface responded quickly enough when I changed parameters. |
| B13 | The prediction checkpoints helped me check my understanding. |
| B14 | I could use the notebook without reading the code. |
| B15 | Mental demand of working with the notebook (1 very low, 5 very high). |
| B16 | I understood what persistence and transitions showed. |
| B17 | I understood what assignment confidence showed. |
Part C: The web demonstrator (only if used)
Everybody answers C1. C2 is for “No”, C3–C16 for “Yes”.
| # | Item | Format |
|---|---|---|
| C1 | Did you use the web demonstrator? | Yes / No |
| C2 | If no: why not? | Select one: did not know how to start / not interested / other: focused on the notebook / … |
| C3 | How long did you use it in total? | < 15 min / 15–30 / 30–60 / > 60 |
| C4 | What did you do? (select all) | Select all that apply: looked at the precomputed sweeps of the Flickr data / ran my own sweeps (Explore) / looked at other example data sets |
Usefulness of components
As in B4, for the web demonstrator: 1 (not useful) – 5 (very useful) / did not use.
| # | Component |
|---|---|
| C5a | Quality-metric charts |
| C5b | Transition (Sankey) view |
| C5c | Confidence violin plots |
| C5d | Group embedding (projection of the groups) |
| C5e | Map and space–time cube |
| C5f | Calendar and profile views |
| C5g | Controls to change parameters |
| C5h | Explore mode (running my own sweeps) |
| C5i | Workflow guide |
Complexity, guidance and interaction
Agreement, 1 = strongly disagree … 5 = strongly agree. Items marked (R) are negatively worded.
| # | Statement |
|---|---|
| C6 | I knew what to do next in the web demonstrator. |
| C7 | There was too much information at once in the web demonstrator. (R) |
| C8 | The available options in the web demonstrator were manageable. |
| C9 | The web-demonstrator notes helped me understand what to look at. |
| C10 | I needed more guidance on how to interpret the web-demonstrator plots. (R) |
| C11 | I could compare results from different parameter settings in the web demonstrator. |
| C12 | I could see how groups changed at neighbouring parameter values in the web demonstrator. |
| C13 | The web demonstrator responded quickly enough when I changed parameters. |
| C14 | The workflow guide helped me decide what to look at next. |
| C15 | I could start without help. |
| C16 | Mental demand of working with the web demonstrator (1 very low, 5 very high). |
Part D: Comparison (only if both tools were used)
For each question: Notebook / Web demonstrator / No difference.
| # | Question |
|---|---|
| D1 | Which helped you more to understand how the result changes with the parameters? |
| D2 | Which helped you more to understand assignment confidence? |
| D3 | Which gave you better guidance on what to do next? |
| D4 | Which made it easier to try your own ideas? |
| D5 | Which would you prefer for learning about clustering in a course? |
| D6 | Which would you prefer for analysing your own data? |
All students used the notebook first and the web demonstrator second. Preferences cannot be separated from order and from familiarity with the data.
Part E: Concepts and comprehension
Conceptual statements
Agreement, 1 = strongly disagree … 5 = strongly agree. Items marked (R) are reverse-keyed: agreement indicates the single-run view.
| # | Statement |
|---|---|
| E1 | A clustering result is best judged by computing a validity index and taking the highest value. (R) |
| E2 | Two parameter settings that produce similar validity scores can support meaningfully different conclusions about the data. |
| E3 | If a cluster structure appears only for one parameter setting and disappears for neighbouring settings, that is evidence against relying on it. |
| E4 | Once the optimal number of clusters has been determined, examining other numbers adds little information. (R) |
| E5 | Knowing how confidently each individual item is assigned to its cluster is important for interpreting the result. |
| E6 | It can be a valid analytical outcome to conclude that none of the tested configurations should be used. |
Self-reported change
“Because of the class, I would now …” Agreement, 1 = strongly disagree … 5 = strongly agree.
| # | Statement |
|---|---|
| E7 | … run the same method with several parameter settings before choosing one. |
| E8 | … compare the groups across settings and not only the scores. |
| E9 | … check how confidently the items are assigned to their groups. |
Comprehension probes
Each probe is answered by selecting one of several answer options. Every option has a score of 0, 1 or 2. The options chosen by respondents are listed in Appendix E.
| # | Prompt |
|---|---|
| E10 | What would you advise before relying on a clustering result from one DBSCAN parameter setting? |
| E11 | What does it mean if a group occurs only at one parameter setting, for example at K = 6 but not neighbouring values? |
| E12 | What can low assignment confidence indicate? |
The probes test understanding at the time of the questionnaire. They do not show learning gains.
Flow and skip logic
Parts C and D are filled in only by students who used the web demonstrator.
| Part | Section | Shown to | Items |
|---|---|---|---|
| A | Background | All | 7 |
| B | Notebook | All | 17 (B4 has 9 ratings) |
| C | Web demonstrator | C1 for everybody. C2 only if “No”. C3–C16 only if “Yes”. | up to 16 |
| D | Comparison | Only if C1 = Yes | 6 |
| E | Concepts and comprehension | All | 12 |