Instructor notes: Lecture 5 practical (Smart clustering of cherry blossom photos)
Tested with Python 3.12, scikit-learn 1.8, ipywidgets 8.1: the whole notebook runs top to bottom in about 30 s (a clustering takes about 0.3 s; the interactive tool about 1.5 s per click). All numbers below are from that run.
DBSCAN results are deterministic; k-means numbers (Part 2) and the MDS colours can differ slightly with other scikit-learn versions.
Timing (60 minutes)
| Stage | Content | min |
|---|
| 1 | Look at the data (map, season) | 5 |
| 2 | Distance for space and time: flat earth vs great circle, radii, relative distances, combine rule, checkpoint | 10 |
| 3 | First run: 25 km, 7 days, N = 7 | 5 |
| 4 | Parameter sensitivity: interactive tool, sweeps | 12 |
| 5 | optional Same K, same groups? (Jaccard recurrence) | 8 |
| 6 | Assess and revise: round 2 (space only) | 12 |
| 7 | Season by place and year | 8 |
| 8 | Wrap-up, one written paragraph | 3 |
Differences from the earlier version of this practical
- The distance is now computed exactly as in the lecture (slides 44-45) and in the old practical: space = great-circle distance in km, time = difference in days, each divided by its radius, then combined (max, Euclid, Manhattan). The earlier version of this notebook scaled x, y and t as three coordinates; that is identical for the Euclidean rule but makes the maximum and Manhattan rules draw a square in space instead of a circle. All Part 1 numbers below are therefore new (for example 106 instead of 108 clusters in the first run).
- Stage 2 explains flat earth vs real spatial distance with real numbers, and how the user-defined radii enter.
- Every map has static space-time cubes next to it (view as in the earlier practicals; density-based results: one cube with and one without noise).
- Quality indicators of the density-based runs now include SSE and explained variance (silhouette, SSE and explained variance are measured with one fixed ruler, 25 km and 7 days, for the clustered photos only).
- The wrap-up of Part 2 carries the eight recommendations of the SmartIterator page.
Answer key, Part 1
- Stage 1: 9,703 photos; 477 (2016) to 1,438 (2011) per year; day of year 5th/50th/95th percentile = Feb 26 / Apr 06 / Jun 19.
- Stage 2: example pair = photos 1220 and 9003 (Newark and Manhattan, 13 and 18 April 2008): flat earth 26.1 km, great circle 20.0 km, projected 19.9 km, 4.99 days. With Rs = 25 km the flat-earth distance would make them non-neighbours, the real distance makes them neighbours.
Over 67,391 random pairs 1-60 km apart: flat earth is longer by +17% (median; +0% to +34% for 90% of the pairs); the projected x_km, y_km differ by -0.0% (median; -1.2% to +1.3% for 98%).
Checkpoint: only the maximum rule makes the pair neighbours (relative distances 0.80 and 0.71; max 0.80, Euclid 1.07, Manhattan 1.51). Shapes: square, circle, diamond in the plane of the relative distances; cylinder, ellipsoid, double cone in the space-time cube.
Points to make: degrees are angles (1 degree of longitude is 85 km at 40N, 71 km at 50N); the radii are your parameters; st_cluster() only uses the projected coordinates to find candidates, the great-circle distance decides.
- Stage 3: 106 clusters, 30.2% noise, sizes 6 to 496. The eight largest clusters are all Washington DC, one per spring 2007-2014 (496, 420, 398, 393, 354, 290, 253, 214 photos). Quality: silhouette 0.83, SSE 9,277, explained variance 99.994% (almost 100% because the clusters are tiny compared with a continent and ten years).
- Stage 4 sweeps (K / noise %): Rs 10-50 km: K 113,105,105,106,110,111,112,113,112; noise 35 to 26. Rt 1-14 d: K 124,140,121,106,103,101,106,110,105,107,107,105,106,111; noise 52 to 25. N 3-12: K 265,184,140,123,106,95,93,90,85,85; noise 22 to 34.
Combine rule: max 106 / 30.2%, Euclid 105 / 31.2%, Manhattan 102 / 33.1%. The K axis of the Rs panel spans only 105-113: a good moment to talk about axis ranges.
Silhouette: rises with N (0.67 to 0.85) as noise rises; with Rt lowest at 1 day (0.70), highest at 6 days (0.84), 0.80 at 14 days; with Rs 0.78-0.84.
SSE: grows with Rs (7,267 to 13,480) and Rt (1,056 to 18,013), falls with N (12,124 to 7,617); lowest when most photos are noise. Explained variance 99.989% to 99.999% in every run: it cannot guide the choice (it is designed for partition-based methods).
- Stage 5: 94 of 106 clusters recur in all nine Rs runs; the others are small (median 8 photos vs 39). N = 12: 82 keep a counterpart; N = 3: 100.
- Stage 6: 13 clusters have photos in the DC window, ten are large (one per spring 2007-2016: 214, 290, 354, 393, 496, 420, 398, 253, 209, 157 photos), 12 have their centre in the window. Colours of the ten large clusters shade from violet (2007) to light blue (2016): similar place and season, different year.
Round 2: 14 places, all 6,770 round-1 photos belong to a place. Rs sweep of round 2 (10-50 km): 16, 14, 14, 14, 14, 13, 12, 12, 12.
Places: DC 3,201; New York 878; Vancouver 743; San Francisco 541; Seattle 461; Toronto 348; Portland 296; Philadelphia 172; then Boston 46, Los Angeles 38, Macon 24 and three tiny ones (8, 7, 7).
- Stage 7: median dates Portland Mar 23, Seattle Mar 30, DC Apr 3, Vancouver Apr 7, Philadelphia Apr 13, San Francisco Apr 14, New York Apr 21, Toronto May 5. Narrowest window DC (20 days), widest San Francisco (Feb 15 - Apr 22).
Checkpoint answer: no single early year. Earliest year: 2012 in DC (median day 83; next is 2016 with 85), New York, Philadelphia and Toronto; 2010 in Vancouver (70.5), Seattle (70) and Portland (66); 2007 in San Francisco.
Caveat to say aloud: photo dates are a proxy for blossom dates and photo counts reflect people, not trees.
Answer key, Part 2 (k-means, min-max scaling, seed 10, 10 restarts)
- 521 weeks, 92 empty; NE mean 9.9 photos/week, SE 0.5. K = 7 sizes: 416, 49, 27, 10, 7, 7, 5 (group 0 = quiet weeks).
- K sweep: variance explained 40% (K = 2) to 88% (K = 15), no elbow; silhouette 0.78 (K = 2), 0.79 (K = 3), 0.66-0.68 for K = 4-14, 0.54 at K = 15. K = 3 sizes 477 / 37 / 7.
- Peak week per region and year: NE 12-15, NW 10-15, GL 15-20, SW week 16 in all ten years (a fixed-date event is a plausible hypothesis, not a result), SE unreliable (week 2 in 2015 and 2016).
- K = 7 to 8: the quiet group keeps 98% of its weeks; groups of 10, 7 and 5 weeks carry over; the 27-week group and one 7-week group split. K = 8 sizes: 412, 41, 29, 12, 11, 7, 5, 4.
- Low-confidence weeks (silhouette < 0.25): group 0 2%, 49-week group 41%, 27-week group 81%, 10-week group 60%.
- Scaling: no scaling 93.0% variance explained but 97% of the spread from NE, ARI 0.65 vs min-max; z-score 76.3%, ARI 0.97. Seeds (one start each): median ARI 0.94 (K = 7), 0.85 (K = 15), 0.78 (K = 25); silhouette over the 30 seeds 0.65-0.68 (K = 7), 0.50-0.69 (K = 15), 0.42-0.61 (K = 25).
- GL = Great Lakes (regions NW, SW, NE, SE, GL are shown with their names in the figures and tables).
Colours (Part 2 vs Part 1)
- Part 2: colours are shared across runs. The mean profiles of the groups of all runs (K = 2-15, 119 groups) are scaled per region, placed together in one MDS map (random_state 9) and coloured by position (four-corner interpolation; corners yellow, cyan, red, blue). A persistent group keeps its colour.
Measured: the quiet group's colour differs by at most 5 (RGB distance, maximum 441) between any two K; 65 persistent group pairs (at least 80% of the weeks in both directions) have a median colour distance of 0 and a maximum of 21, against a median of 146 for random pairs of groups from different runs.
Side effect: similar groups get similar colours, so the quiet group (group 0) and the 49-week group (group 1) are two close greens in the calendar. This is by design (lecture slides 69-72); the box plots and the group numbers tell them apart.
- Part 1: colours are computed anew for every run (profile: x, y, mean time, mean day of the year; then MDS and four-corner colours), so colours must not be compared between runs. The notebook explains what the colours show after Stage 3 and again in Stage 6. The demonstrator uses one embedding for all runs.
Wrap-up recommendations (end of Part 2)
Eight recommendations from https://geoanalytics.net/and/SmartIterator/ (Practical Recommendations), each mapped to the stage where students did it. Recommendation 6 is worded differently from the page: the page suggests that a silhouette fluctuation above 0.02 across seeds signals instability. In the paper's EU example the silhouette varied by only 0.012 over 30 seeds while the groups agreed only moderately (median adjusted Rand index 0.53), so the notebook says: look at the score and at the groups. In this notebook's weekly data the silhouette does swing with the seed (0.50 to 0.69 at K = 15).
Part 3 (audit)
Categories are those of the course manual. Passage 1 (weekly counts): sycophancy, overstated certainty, unacknowledged confounds, erased uncertainty, fabricated or unseen detail, causal/prescriptive. Passage 2 (photos): sycophancy, overstated certainty, false precision, unacknowledged confounds, erased uncertainty, causal/prescriptive.
Marking is not needed: the checklist reveals its own analysis. Collect the rewrites and the synthesis paragraph if you want to see understanding.
Likely misconceptions
- "106 clusters = 106 hotspots": they are space-time concentrations (one place in one spring); round 2 shows 14 places.
- "Degrees are kilometres": the flat-earth error of Stage 2; one degree of longitude is shorter than one degree of latitude.
- "A high explained variance means good clusters": here it is 99.99% in every run.
- "A higher silhouette means a better clustering": it rises when loose points become noise.
- "No noise in round 2 means a good result": round 2 starts from points that were already dense.
- "Stable K means stable groups": Stage 5.
Data and privacy
cherry_photos.csv was derived from public Flickr photo metadata. Only longitude, latitude and the time of taking were used; no user names, ids, titles, tags or URLs. Coordinates were projected to km (Albers equal-area, standard parallels 29.5 and 45.5, origin 96 W, 23 N); t_days is days since 2007-01-01. The photo set includes some Canadian places (Vancouver, Toronto). The 3- and 6-year files of the earlier practical are subsets of this file.
weekly_counts_by_regions.csv holds aggregated counts. The base map is made from Natural Earth data (public domain).
Direct links to the data sets: ?ds=cherry-photos and ?ds=cherry-weeks (for example demo/?ds=cherry-photos).
Agreement: the demonstrator uses the same distance recipe (great-circle distance in km, optional time difference), so the first run of the photos gives the same 106 clusters and 30.2 % noise as the notebook. Do not expect every number to match across implementations: libraries, random seeds and floating-point details differ (for example, the demonstrator's k-means for Part 2 uses 5 random starts per run, the notebook 10, so partitions at high K can differ slightly).
Stage 4 sweeps = the first three sweeps of Cherry blossom photos; round 2 = the fourth sweep, or Explore with Items: members of the clusters of the active iteration and time switched off.
Part 2 = Cherry blossom photos per week (calendar, box plots, transition view, Order runs for the seed sweep).
The demonstrator's silhouette is computed on a reference sample, so values differ slightly from the notebook.
Possible extensions
Progressive refinement of loose clusters (lecture slide 65); HDBSCAN as an alternative to DBSCAN; a year filter for Round 1 (the 3- and 6-year files of the earlier practical).