← ATWL supplementary materials
What this page is
Ten experts evaluated the LLM-extracted representation of their own visual analytics (VA) approach: the workflow summary and the ATWL representation, with its code view and graphical views. This page reports the closed-form ratings in aggregated form and a summary of the free-text comments. It accompanies the expert evaluation in the revised paper. The individual responses are not published; the section “Protection of participants” explains why and how.
Main results
- Accuracy. All ten participants at least somewhat agreed that the workflow summary reflects the paper (nine fully) and that the ATWL representation captures the structure of their approach (eight fully). Most found no interpretation errors and no falsely added parts; the individual issues they raised are summarised below.
- Graphical representation. Seven participants fully agreed that it helps in understanding the representation, one somewhat agreed, and two were undecided. The code view and the graphical views were seen as complementary.
- Library. All participants agreed that a public library of ATWL workflows is valuable for the community (six fully).
- Own use. Eight participants would consider representing future workflows in ATWL as a supplementary artifact for their papers; two were undecided.
- Conditions for use. Low effort of LLM-supported generation, accurate extraction, ease of verification, and access to a library of existing representations were each named by eight participants.
- Criticism. The main points concern the complexity and readability of large representations, the learning curve of the language, and a few modelling details.
Participants and questionnaire
All ten participants gave their consent on the questionnaire. Each evaluated the representation of a workflow of their own, extracted with the LLM-supported procedure of the paper. The questionnaire had 14 items: five statements rated on a five-point agreement scale (Q1, Q4, Q5, Q11, Q13), one multiple-choice item on the conditions for using ATWL (Q12), and eight free-text items (Q2, Q3, Q6–Q10, Q14). The item numbers below are those of the questionnaire.
Protection of participants
The participants are researchers, and the workflows they judged come from published papers. Detailed comments could therefore reveal who they are. To protect them, this page contains only information that cannot be traced to an individual.
- Ratings are reported in aggregated form only.
- Free-text comments are not published. They are summarised across participants, and the summaries were generalised so that they do not refer to particular workflows, tools, or personal circumstances.
- No statement is attributed to a person, and statements are not linked to each other or to ratings.
- The individual responses and any material that could identify a participant are neither published nor shared.
Because of this generalisation, some remarks that were specific to a single workflow cannot be reported. Complete anonymity cannot be guaranteed for a small group of experts, which is why the summaries are kept deliberately general.
Ratings
Q1. The workflow summary reflects what the paper actually describes.
Mean 4.9, SD 0.3 (n = 10)
Q4. The ATWL representation accurately captures the structure of the workflow of my VA approach.
Mean 4.8, SD 0.4 (n = 10)
Q5. A graphical representation helps in understanding the ATWL representation.
Mean 4.5, SD 0.8 (n = 10)
Q11. A public library of ATWL workflows would be of value for the community.
Mean 4.6, SD 0.5 (n = 10)
Q13. I would consider representing future workflows in ATWL (with LLM-supported extraction procedure) as a supplementary artifact for my papers.
Mean 4.1, SD 0.7 (n = 10)
Numbers in the bars are counts of participants. Means are computed with the values 1 (disagree) to 5 (agree).
Q12. Conditions for using ATWL to represent one’s own VA approach (multiple answers possible; number of participants)
Free-text comments (summary)
The comments were summarised by the authors. Themes are listed in no particular order and are not ranked by frequency; “most”, “several”, and “individual” are used instead of exact counts.
Workflow summary (Q2: missing parts, Q3: falsely added parts)
Participants found the summaries accurate, concise, and complete at the intended level of detail. Several mentioned minor gaps, all concerning secondary aspects: optional or alternative choices that the approach offers, data or visualisation types that the paper mentions only briefly, feedback links between analysis phases, and the assignment of views to tasks. Several considered such omissions appropriate for a summary.
Almost all found nothing falsely added. One participant noted that a concluding step in the summary had been inferred from the use cases of the paper rather than from the approach itself.
Accuracy of the ATWL representation (Q6: interpretation errors, Q7: missing parts, Q8: falsely added parts)
Interpretation errors. Most participants reported none. Individual remarks concerned the intent assigned to a step, loops whose exit condition was modelled differently from the human-driven process, artifacts referenced after earlier steps had modified them, parts of the workflow judged redundant, and artifact names that are not informative.
Missing parts. The representation was seen as complete but low-level; it makes explicit details that the papers leave implicit. Suggested additions were a clearer marking of human decisions, feedback links between phases, preparatory steps that secure the quality of a model, and user-driven changes of the visual representation together with the provenance they may create.
Falsely added parts. Most found none. Participants added that dense graphs cannot be verified edge by edge, and that an implicit step in a loop (judging whether the results suffice) may not be something users do consciously.
Positive remarks (Q9)
- ATWL provides a structured common ground for communicating, comparing, and reasoning about workflows that are otherwise described loosely.
- It is useful as a basis for reproducing and reimplementing workflows from papers, and for teaching.
- Making the data abstraction explicit and distinguishing human from machine roles were appreciated.
- The code view (with commented blocks, syntax highlighting, and a consistent structure) and the graphical views were seen as complementary: some participants followed the steps more easily in the code and the summary, others valued the graph for the relations between artifacts and transforms.
Improvement opportunities (Q10)
- Control of the complexity of large representations: aggregation, filtering, and interactive links between a high-level and a detailed view.
- Clearer node naming and grouping of transforms and artifacts into semantic steps.
- Visual distinction of human, machine, and hybrid components.
- Stable layout and clear colour coding when expanding and collapsing the graph.
- Clarification of the purposes of the different views.
- Integration into common code editors and cross-reference aids.
- Natural-language summaries or simplifications for less experienced users.
- Clearer support for parallel steps.
Other feedback (Q14)
- The representation is accurate but requires effort to learn, so its community may remain a specific group within VA research.
- A more general, domain-independent representation of a workflow would improve reusability.
- A large library would allow analysing the set of workflows as a dataset and comparing one’s own workflow with existing ones.
- LLM support was seen as useful in both directions: assisting the writing of representations and running automatic checks and revisions.
- A representation must stay digestible; mechanisms for reducing complexity are key for complex workflows.
- Onboarding material (such as the introductory slides) could be offered inside the viewer.
Reading and limits
- The group is small (ten participants) and the experts judged representations of their own approaches. The ratings are indicative and not suitable for statistical inference.
- Free-text comments were summarised and generalised by the authors; the summaries do not contain every remark.
- The evaluation checks whether representations are faithful from the authors’ point of view. It does not test whether other people would extract or read the workflows in the same way (see also the Re-Extraction Check).