Our study was conducted according to the following methodology. Two groups of researchers had different roles according to their major areas of expertise; they can be called ‘technology group’ and ‘evaluation group’.
The evaluation group, consisting of four experts in information visualization and evaluation of visual displays and user interfaces, has previously collected data about eye movements of people performing a particular type of tasks with the use of different tree diagrams, namely, finding the least common ancestor of several marked leaf nodes. The evaluation group tried to analyze the collected eye movement data data using standard techniques traditionally used for eye movement analysis but was disappointed by the limited opportunities provided by these techniques and, as a consequence, by insufficient insights that could be gained. Particularly, the researchers could not explore the temporal aspect of the data.
The technology group, consisting of two experts in geographic visualization and analysis, has large experience in developing methods for analysis of geographical movement data and has earlier occasionally applied their methods also to eye movement data. However, no systematic study has been done before. The two groups decided to cooperate in performing such a study and conveying their experiences and recommendations to the visualization research community.
Since the evaluation group was not experienced in using geographic analysis methods, it was decided that they should not spend their time on the application of methods and tools and concentrate instead on evaluating the method results. It was the task of the technology group to apply various methods to the data provided by the evaluation group and create illustrated reports about the results. The evaluation group posed various questions about users’ viewing behaviors they wanted to find answers for. The technology group tried to give substantiated answers to these questions in the reports. The evaluation group studied the reports of the technology group, tried to interpret the results based on their expertise, and evaluated the utility of the methods. New questions often arose from studying the reports and discussions. The technology group tried to find answers also to these questions by means of the available methods. The groups had one half-day and one full-day face-to-face meetings and in the remaining time communicated through Internet.
The questions posed by the evaluation groups can be considered as tasks given to the technology group. The technology group did not only try to fulfill these tasks but also, based on their experiences with geographic movement data, suggested other possible questions and tasks that could make sense for eye movement data. The evaluation group typically judged these questions as relevant.
The eye movement data were acquired in a controlled laboratory experiment previously conducted by Burch et al. The task for this experiment was to find the least common ancestor for a set of marked leaf nodes in node-link tree diagrams. Tree layout, tree orientation, and number of marked leaf nodes were chosen as independent variables and task completion time and accuracy were recorded as dependent variables. This study emphasized high accuracy, i.e., participants were instructed to answer as correctly as possible, resulting in an overall accuracy of more than 97 percent. By focusing on a high degree of accuracy, participants had to apply some kind of reliable visual task solution strategy to come up with correct answers.
The study was based on a repeated-measures design in within-subjects style. A Tobii T60 XL eye tracking system was used to record eye movement data, i.e., fixation durations and saccades, along with completion times and accuracies. Participants sat in front of a TFT screen at a resolution of 1920x1200 pixels at a distance given by the calibration feature of the eye tracking system. A minimum of 10 pixels covering and 30 ms fixation duration were chosen as key parameters, i.e., if a person’s gaze stayed for at least 30 ms in the same region of 10 pixels width and height, this was registered as an eye fixation. The 38 participants came from Western countries, with a preference for left-to-right reading direction.
In the experiment, participants were shown tree data stimuli in three different node-link layouts: traditional, orthogonal, and radial. The non-radial diagrams had four different root orientations (top, right, bottom, left). Furthermore, the number of marked leaf nodes varied between three, six, and nine. There were three blocks each containing stimuli with the same layout. Inside each block, the stimuli were randomized. The stimuli data was generated by a stochastic algorithm before running the study. The test system displayed the next stimulus when the participant confirmed to proceed (continue-on-demand). In total, there were 2,052 individual tests.
The task for the participants was to locate the least common ancestor of a set of red colored leaf nodes. Once the location of the correct node was found, the participant had to confirm it by a mouse click. This task was chosen because the hierarchy had to be understood and a strategy had to be applied to answer correctly. The differences between the tree layouts, orientations, and number of marked leaf nodes might cause different task solution strategies. The applied strategies were of special interest for the researchers.