Appendix E: Summary of responses

Summary of 20 voluntary, anonymous questionnaire responses on the notebook and the web demonstrator. Values use a five-point scale unless stated otherwise.

Questionnaire (Appendix D) · Supplementary materials

Overview

20responses included
14used the web demonstrator
6did not use the web demonstrator
14completed the comparison questions for both tools
5.35 / 6mean total score of the three comprehension probes (maximum 6)

The page reports counts and distributions. It uses no significance tests and makes no claims about the effects of the notebook or the web demonstrator.

A – Background

Self-reported background of the 20 respondents.

ItemAnswerCountShare
A3 Applied a clustering algorithm outside a tutorialYes840%
No1260%
A4 Before, ran the same method with several parameter settings on the same dataYes420%
No1680%
A9 Attended the lecture on clusteringYes20100%
12345
ItemDistributionnMeanMedian
A5 Familiarity with Python / scikit-learn (1 none, 5 fluent)
5
8
5
2
203.203
A6 Familiarity with clustering validity indices (1 none, 5 fluent)
1
8
6
5
202.753
A7 Experience with Jupyter or Colab notebooks (1 none, 5 daily use)
5
6
4
5
203.453
A8 Experience with interactive web-based visualization tools (1 none, 5 daily use)
3
4
9
4
202.703

B – Notebook

Asked of all 20 respondents. Usefulness (B4) is rated from 1 (not useful) to 5 (very useful). Agreement items use 1 (strongly disagree) to 5 (strongly agree). Mental demand (B15) uses 1 (very low) to 5 (very high).

ItemAnswerCountShare
B1 How far did you get in class?Part 1 only630%
Part 1 and part of Part 21260%
Parts 1 and 2210%
B2 Continued with the notebook after the class?No1155%
Yes, Part 2945%
B3 The time in class was sufficient.Yes630%
Partly840%
No630%

Usefulness of components (B4)

ItemDistributionnMeanMedian
B4a Sweep plots
3
6
11
204.405
B4b Maps and space–time cubes
4
6
4
6
203.603.5
B4c Persistence and transition displays
1
4
6
9
204.154
B4d Assignment confidence (silhouette) plots
5
5
7
3
203.403.5
B4e Calendar and profile plots
2
3
6
5
4
203.303
B4f Interactive sliders
4
6
8
2
203.403.5
B4g Prediction checkpoints
2
2
7
4
5
203.403
B4h The notes “What this shows”
1
4
5
10
204.204.5
B4i Audit exercise (Part 3)
4
7
6
3
203.403

Complexity, guidance and interaction (B5–B17)

Bars show the responses as given, from 1 (strongly disagree) to 5 (strongly agree), also for negatively worded and reverse-keyed items. For these items, high values are the unfavourable answer.

ItemDistributionnMeanMedian
B5 I knew what to do next.
7
3
10
204.154.5
B6 There was too much information at once. (negatively worded)
6
8
6
203.003
B7 The available options were manageable.
9
4
7
202.903
B8 The notes helped me understand what to look at.
6
8
6
204.004
B9 I needed more guidance on how to interpret the plots. (negatively worded)
6
8
6
203.003
B10 I could compare results from different parameter settings.
4
13
3
203.954
B11 I could see how groups changed at neighbouring parameter values.
4
13
3
202.953
B12 The interface responded quickly enough when I changed parameters.
5
11
4
203.954
B13 The prediction checkpoints helped me check my understanding.
8
9
3
203.754
B14 I could use the notebook without reading the code.
6
8
6
203.003
B15 Mental demand of working with the notebook (1 very low, 5 very high).
3
6
8
3
203.554
B16 I understood what persistence and transitions showed.
3
7
7
3
203.503.5
B17 I understood what assignment confidence showed.
1
3
9
3
4
203.303

C – Web demonstrator

Fourteen respondents used the web demonstrator. Six did not.

ItemAnswerCountShare
C3 Total time with the web demonstrator< 15 min429%
15–30 min321%
30–60 min17%
> 60643%
C4 What students did (several answers possible)Looked at precomputed Flickr sweeps14100%
Ran own sweeps (Explore)964%
Looked at other example data sets536%

Reasons for not using it

ItemAnswerCountShare
C2 Reason for not using itFocused on the notebook350%
Not interested233%
Did not know how to start117%

Usefulness of components (C5), 14 users

ItemDistributionnMeanMedian
C5a Quality-metric charts
1
4
3
6
144.004
C5b Transition (Sankey) view
1
4
3
6
144.004
C5c Confidence violin plots
3
5
5
1
143.293
C5d Group embedding
2
2
6
4
143.864
C5e Map and space–time cube
1
7
3
3
143.573
C5f Calendar and profile views
1
3
6
4
143.644
C5g Controls to change parameters
2
3
6
3
143.714
C5h Explore mode
2
2
6
4
143.864
C5i Workflow guide
1
5
5
3
143.714

Complexity, guidance and interaction (C6–C16), 14 users

Bars show the responses as given, from 1 (strongly disagree) to 5 (strongly agree), also for negatively worded and reverse-keyed items. For these items, high values are the unfavourable answer.

ItemDistributionnMeanMedian
C6 I knew what to do next in the web demonstrator.
5
6
3
143.864
C7 There was too much information at once in the web demonstrator. (negatively worded)
1
6
7
144.434.5
C8 The available options in the web demonstrator were manageable.
6
7
1
143.644
C9 The web-demonstrator notes helped me understand what to look at.
3
8
3
144.004
C10 I needed more guidance on how to interpret the web-demonstrator plots. (negatively worded)
5
4
5
144.004
C11 I could compare results from different parameter settings in the web demonstrator.
5
6
3
143.864
C12 I could see how groups changed at neighbouring parameter values in the web demonstrator.
8
2
4
143.713
C13 The web demonstrator responded quickly enough when I changed parameters.
4
9
1
143.794
C14 The workflow guide helped me decide what to look at next.
2
8
4
144.144
C15 I could start without help.
2
6
6
143.293
C16 Mental demand of working with the web demonstrator (1 very low, 5 very high).
2
8
4
143.143

D – Preferences of students who used both tools

Answered by the 14 students who used both tools. All of them used the notebook first, so the answers are preferences in this sequence and not an isolated comparison of the tools.

NotebookNo differenceWeb demonstrator
QuestionDistributionNotebookNo differenceWeb demonstrator
D1 Which helped you more to understand how the result changes with the parameters?
7
1
6
716
D2 Which helped you more to understand assignment confidence?
7
2
5
725
D3 Which gave you better guidance on what to do next?
5
3
6
536
D4 Which made it easier to try your own ideas?
4
4
6
446
D5 Which would you prefer for learning about clustering in a course?
3
3
8
338
D6 Which would you prefer for analysing your own data?
3
3
8
338

E – Concepts and comprehension

E1–E9 use 1 (strongly disagree) to 5 (strongly agree). E1 and E4 are reverse-keyed: agreement reflects a single-run or single-optimum view. E7–E9 are self-reports of intended change.

Bars show the responses as given, from 1 (strongly disagree) to 5 (strongly agree), also for negatively worded and reverse-keyed items. For these items, high values are the unfavourable answer.

ItemDistributionnMeanMedian
E1 A clustering result is best judged by computing a validity index and taking the highest value. (reverse-keyed)
5
10
5
202.002
E2 Two parameter settings that produce similar validity scores can support meaningfully different conclusions about the data.
4
11
5
204.054
E3 If a cluster structure appears only for one parameter setting and disappears for neighbouring settings, that is evidence against relying on it.
8
4
8
204.004
E4 Once the optimal number of clusters has been determined, examining other numbers adds little information. (reverse-keyed)
9
7
4
201.752
E5 Knowing how confidently each individual item is assigned to its cluster is important for interpreting the result.
1
8
11
204.505
E6 It can be a valid analytical outcome to conclude that none of the tested configurations should be used.
4
8
8
204.204
E7 Because of the class, I would now run the same method with several parameter settings before choosing one.
6
8
6
204.004
E8 Because of the class, I would now compare the groups across settings and not only the scores.
6
10
4
203.904
E9 Because of the class, I would now check how confidently the items are assigned to their groups.
4
10
6
204.104

Comprehension probes (E10–E12)

Each probe was answered by selecting one of several answer options. Each option has a score of 0, 1 or 2. The table lists the options that respondents chose.

Probe and answer chosenScoreCount
E10 Advice before relying on one DBSCAN result (mean score 1.70)
I would check neighbouring values and use the transition display to see whether the group persists. I would also inspect its contents.28
One run is not enough. I would vary epsilon and min_samples and see whether similar groups recur; then inspect the data context before making a claim.24
I would rerun DBSCAN with several parameter settings, compare the clusters across nearby settings, and inspect whether they make sense on the map. I would also check noise and assignment confidence.22
I would try a few settings and compare the silhouette scores before choosing.16
E11 Meaning of a group that occurs at only one setting (mean score 2.00)
It probably means the group is not stable. I would check the neighbouring runs and whether its members make sense together.212
If the group occurs only at K=6, it may be specific to that setting. I would compare K=5 and K=7 and inspect the transitions.28
E12 Meaning of low assignment confidence (mean score 1.65)
Low confidence means some points are close to other clusters or the group may be heterogeneous. I would inspect them and test other settings.28
The group may overlap with another group or contain mixed items; I would inspect the points and consider changing parameters.25
It means the algorithm is unsure which group is correct.17
Total score across the three probesDistributionMeanMedian
Total score, 0–6
3
7
10
5.355.5

Reading and limits