Iterative Trace Partitioning in Practice: a Worked Example on the BPI Challenge 2013 (Incidents) Log

A step-by-step case study of one analysis session in TracePrism, kept as a supplementary reference alongside the TVCG submission and as teaching material for students learning the tool and the underlying method. Third in the series, after Road Traffic Fines and Sepsis Cases.

Getting oriented: dataset overview and the process graph

STEP 0BPI Challenge 2013 (incidents) records IT incident-management activity at Volvo IT / VINST: 7,553 traces, only 13 distinct activities, median length 6 events (range 2–123). Unlike Fines or Sepsis, this log's outcome is not a separate business result — it is the highest of three tiered support lines (1st/2nd/3rd) an incident's handling ever escalated to, directly mirroring one of the challenge's own original questions ("does this incident get pushed to a higher support line?"). The outcome mix is skewed but less extreme than Sepsis: line_1_only 60.2% (resolved without ever leaving first-line support), reached_line_2 32.6%, reached_line_3 7.2% (the rarest outcome — the small minority of incidents escalated all the way to specialist support). Case attributes are organizational rather than clinical: distinct teams involved, team handoffs (a "ping-pong" proxy for the challenge's own handoff question), whether the incident was ever parked in a "Wait – User" sub-status, the starting support line, and the highest support line reached (excluded from modeling for the reason given above).

Step 1 — An obvious group: the quick single-call resolution

STEP 1BPI2013 has one common short trace variant clearing the ≥20-occurrence bar for an "obvious group": Accepted_In_Progress → Accepted_In_Progress → Completed_In_Call — an incident picked up, worked on directly, and closed out on a call without ever leaving first-line support. 1,749 traces (23.2% of the whole dataset), 98.3% pure line_1_only (1,720 of 1,749). This was committed immediately, for the same reason obvious groups are committed first in the other two case studies: it is a free, zero-modeling win that narrows what Step 2's topic model has to explain, and at 23.2% of the dataset it is the single largest committed group across the whole session.

Step 2 — Topic round: K=14 on the 5,804-trace remainder

STEP 2With the obvious group removed, the encoding recommender was run against the remaining 5,804 traces. Its unfiltered top-ranked candidate combined every declared attribute facet, including the excluded highest support line reached (sharp 0.674, NMI 0.059, score 0.366) — per the note above, that facet was blocked. With it excluded, the switches actually turned on were activities + 2-grams + distinct teams involved + team handoffs + ever in "Wait User" status + starting support line, closely tracking the next-best legitimate candidates in the same table (activities + 2-grams + attribute-augmented activity, sharp 0.668/NMI 0.054/score 0.361, and activities + 2-grams + distinct teams involved, sharp 0.661/NMI 0.053/score 0.357).

A K-sweep from K=3 to K=20 was run and seven candidate K's were compared under a dominant-outcome-share ≥ 0.85 (and n ≥ 15) rule; coverage rose from 1,326 of 5,804 traces at K=8 to a peak of 1,475 at K=14, then drifted back down through K=16–20 (1,446, 1,440, 1,407) as the extra splits added more topics without adding qualifying coverage — the textbook "diminishing returns" signature described in the app's own K-sweep guidance. K=14 was taken forward.

At K=14, four of the fourteen topics cleared the purity bar and were committed: T1 (616 traces, 100% pure line_1_only), T6 (473 traces, 91.5% pure reached_line_2), T9 (251 traces, 90.4% pure line_1_only), and T13 (135 traces, 100% pure reached_line_3) — notably, a perfectly pure topic for the rarest outcome, on the very first modeling round. The other ten topics ranged 46–70% purity and were left for the next round.

Step 3 — Diminishing returns: three more rounds find nothing further

STEP 3The 4,329 traces left after Step 2 were re-scored (top-ranked candidate this round: activities + 2-grams + gap-augmented 2-grams, sharp 0.674, NMI 0.018, score 0.346 — no excluded facet involved this time). A K-sweep under the same purity logic as Step 2, but progressively widened and loosened to make sure nothing was missed, was run three times over the same 4,329-trace remainder: K=4–16 at ≥0.80 purity/n≥12, then K=3–12 at the same threshold with n≥10, then K=2–8 with n≥8. Every single K tried, across all three passes, produced zero qualifying topics. The best any individual topic ever reached in these three rounds was 67.7% dominant-outcome share — informative as a topic, but well short of even a relaxed purity bar, and this pattern held whether K was small or large. This is a sharper, more absolute version of the "thin round" both companion case studies eventually hit (Sepsis's Step 4 still found one 7-trace topic; here, nothing at all cleared the bar at any K from 2 to 16) — a clean, unambiguous, and early diminishing-returns signal that further topic modeling on this remainder was not going to pay off, so the session moved directly to Stage 5.

Step 4 — UMAP + k-means (N=24) on the 4,329-trace residue

STEP 4Topics were refit at K=10 purely as a feature space (not evaluated for purity in their own right — the K=10 choice itself came from comparing K=4/6/8/10/12 as candidate feature spaces and taking the one whose downstream clustering covered the most traces), then UMAP-embedded and clustered with k-means swept across N=8–28. At the standard 0.80 homogeneity bar used throughout the other two case studies, this sweep also came back completely empty, at every N from 8 to 20 — the residue resisted clustering just as thoroughly as it resisted topic modeling. Relaxing the bar to 0.75 (the loosest threshold used anywhere in this session, and looser than either companion study needed) finally surfaced real structure: coverage peaked at 881 of 4,329 traces (20.3%) at N=24, across four qualifying clusters, ahead of every other N tried (0–662 traces).

N=24 was taken forward and re-fit directly (rather than read off the sweep) before committing. All four qualifying clusters — 190, 283, 248, and 160 traces, at 75.8%, 78.4%, 75.0%, and 75.6% homogeneity respectively — turned out to be line_1_only-dominant. Neither reached_line_2 nor reached_line_3 ever formed a homogeneous cluster at any N tried, at any threshold down to 0.75 — a mirror image of Step 2, where the very first topic round found a perfectly pure reached_line_3 topic essentially for free. Whatever distinguishes a clean single-line resolution from one that escalates seems to survive both NMF and k-means once the obvious group and Step 2's four topics are removed; the four Step 4 clusters mostly separate different flavors of still-tidy line_1_only handling from each other, not escalation from non-escalation.

The four qualifying clusters were committed directly from the cluster table.

Final knowledge base and what was left unexplained

Table 1 lists every group committed during the session, in commit order, with its size, share of the full dataset, dominant outcome, and provenance.

Table 2 breaks down the 3,448-trace residue by outcome, compared against its share of the whole dataset.

Takeaways for students