Expert Evaluation (R1)

ATWL supplementary materials — how do authors of visual analytics approaches judge the ATWL representation of their own workflow?

← ATWL supplementary materials

What this page is

Ten experts evaluated the LLM-extracted representation of their own visual analytics (VA) approach: the workflow summary and the ATWL representation, with its code view and graphical views. This page reports the closed-form ratings in aggregated form and a summary of the free-text comments. It accompanies the expert evaluation in the revised paper. The individual responses are not published; the section “Protection of participants” explains why and how.

Main results

Participants and questionnaire

All ten participants gave their consent on the questionnaire. Each evaluated the representation of a workflow of their own, extracted with the LLM-supported procedure of the paper. The questionnaire had 14 items: five statements rated on a five-point agreement scale (Q1, Q4, Q5, Q11, Q13), one multiple-choice item on the conditions for using ATWL (Q12), and eight free-text items (Q2, Q3, Q6–Q10, Q14). The item numbers below are those of the questionnaire.

Protection of participants

The participants are researchers, and the workflows they judged come from published papers. Detailed comments could therefore reveal who they are. To protect them, this page contains only information that cannot be traced to an individual.

Because of this generalisation, some remarks that were specific to a single workflow cannot be reported. Complete anonymity cannot be guaranteed for a small group of experts, which is why the summaries are kept deliberately general.

Ratings

I disagreeI somewhat disagreeUndecidedI somewhat agreeI agree

Q1. The workflow summary reflects what the paper actually describes.

Mean 4.9, SD 0.3 (n = 10)

Q4. The ATWL representation accurately captures the structure of the workflow of my VA approach.

Mean 4.8, SD 0.4 (n = 10)

Q5. A graphical representation helps in understanding the ATWL representation.

Mean 4.5, SD 0.8 (n = 10)

Q11. A public library of ATWL workflows would be of value for the community.

Mean 4.6, SD 0.5 (n = 10)

Q13. I would consider representing future workflows in ATWL (with LLM-supported extraction procedure) as a supplementary artifact for my papers.

Mean 4.1, SD 0.7 (n = 10)

Numbers in the bars are counts of participants. Means are computed with the values 1 (disagree) to 5 (agree).

Q12. Conditions for using ATWL to represent one’s own VA approach (multiple answers possible; number of participants)

Access to a substantial library of existing ATWL representations for reference
8
Consistently accurate workflow extraction by the LLM (minimal correction needed)
8
Ease of verifying that the ATWL representation faithfully captures the original workflow
8
Low effort to run ATWL generation (with an LLM)
8
Availability of an interactive graphical viewer
7
Fast generation of ATWL representation (< 1 hours)
7
A user-friendly editor with syntax highlighting and validation
5
Adoption of ATWL by the broader VA research community
4
Availability of a tool to check syntactic correctness of ATWL representations
3

Free-text comments (summary)

The comments were summarised by the authors. Themes are listed in no particular order and are not ranked by frequency; “most”, “several”, and “individual” are used instead of exact counts.

Workflow summary (Q2: missing parts, Q3: falsely added parts)

Participants found the summaries accurate, concise, and complete at the intended level of detail. Several mentioned minor gaps, all concerning secondary aspects: optional or alternative choices that the approach offers, data or visualisation types that the paper mentions only briefly, feedback links between analysis phases, and the assignment of views to tasks. Several considered such omissions appropriate for a summary.

Almost all found nothing falsely added. One participant noted that a concluding step in the summary had been inferred from the use cases of the paper rather than from the approach itself.

Accuracy of the ATWL representation (Q6: interpretation errors, Q7: missing parts, Q8: falsely added parts)

Interpretation errors. Most participants reported none. Individual remarks concerned the intent assigned to a step, loops whose exit condition was modelled differently from the human-driven process, artifacts referenced after earlier steps had modified them, parts of the workflow judged redundant, and artifact names that are not informative.

Missing parts. The representation was seen as complete but low-level; it makes explicit details that the papers leave implicit. Suggested additions were a clearer marking of human decisions, feedback links between phases, preparatory steps that secure the quality of a model, and user-driven changes of the visual representation together with the provenance they may create.

Falsely added parts. Most found none. Participants added that dense graphs cannot be verified edge by edge, and that an implicit step in a loop (judging whether the results suffice) may not be something users do consciously.

Positive remarks (Q9)

Improvement opportunities (Q10)

Other feedback (Q14)

Reading and limits