The Speaker, Betty Boothroyd, rebuked an M.P. for using a cardboard diagram in the Commons to explain overseas aid figures. She said “I have always believed that all Members of this House should be sufficiently articulate to express what they want to say without diagrams.”[Guardian 7/12/94] p. 2

little is known about the cognitive value of any graphical representations, be they good old-fashioned (e.g. diagrams) or more advanced (e.g. animations, multimedia, virtual reality). In our paper, we critique the disparate literature on graphical representations, focussing on four representative studies. Our analysis reveals a fragmented and poorly understood account of how graphical representations work, exposing a number of assumptions and fallacies. As an alternative we propose a new agenda for graphical representation research. This builds on the nascent theoretical approach within cognitive science that analyses the role played by external representations in relation to internal mental ones. We outline some of the central properties of this relationship that are necessary for the processing of graphical representations. Finally, we consider how this analysis can inform the selection and design of both traditional and advanced form of graphical technology. p. 2

Virtual reality and visualization1, as means of representing and interacting with information, are very much at the forefront of technological development. An overriding intuition is that much can be gained computationally from interacting with virtual reality simulations or visualizing from three-dimensional dynamic images (Fairchild et al., 1993). p. 3

a long line of assumptions about graphical technological advancements p. 3

• static pictures and diagrams are better than sentential representations2 • 3-D representations are better than 2-D ones • solid modelling is better than wire-frame modelling • colour is better than black and white images • animated diagrams are more effective than static images • interactive graphics are better than non-interactive graphics • Virtual reality is better than animation. p. 3

Such generalisations about the benefits of advanced graphical technologies over good old fashioned representations, however, beg the question of what is actually gained cognitively from having more explicit, dynamic and interactive representations of information. Why, for example, should an animated diagram that changes in response to user interaction be more effective at facilitating problem-solving than static diagrams? Why not the other way round, where static diagrams are more effective than animations or non-interactive graphics p. 3

1The term visualization is defined as “mechanisms by which humans perceive, interpret, use and communicate visual information” (McCormick and DeFanti, 1987). p. 3

The value of different graphical representations3, be they good old-fashioned or technologically-advanced, cannot be assessed adequately from our intuitions. To be effective a number of interdependent factors need to be considered, such as the level of experience with the graphical representation, the knowledge domain and the type of task. p. 4

Whilst there have been numerous empirical studies investigating different aspects of graphical representations there has been little attempt to integrate the findings into an analytic framework. p. 4

What is needed, therefore, is a more systematic approach for evaluating the merits of different kinds of graphical representations, one that is theoretically-driven and which accounts for the cognitive processing when people interact with them. p. 4

3 Graphical representations include diagrams, maps, plans, animations and virtual reality and are distinct from propositional/sentential representations and formal notation(cf. Larkin and Simon, 1987). p. 4

Why is there a lack of a suitable, explicit processing model? One reason may be that the form of graphical representation does not lend itself to systematic computational analyses. The theoretical frameworks and formal notations that have been developed for analysing verbal language are not applicable to the syntactically- and semantically-dense properties of graphical representations (Goodman, 1968). p. 5

Another reason, as we argue later, may be that p. 5

there seems to be a pervasive (and possibly unwarranted) assumption that graphical representations must work in a certain way because of their figural nature. Thus many studies are almost ‘black-box’ in their approach to psychological mechanisms. Some, however, have attempted to look systematically at the effective perceptual features of graphical representations. For example Winn (1993) analysed diagrams in terms of a model of visual search, focusing on strategies for extracting information. His analysis identifies the importance of external features such as the spatial distribution and discriminability of elements of the diagram. He also points to important cognitive processes such as knowledge of content and symbol conventions in the reading process. Winn states that such accounts must as yet be insufficient. He points to a lack of graphical representation-specific research on search strategies but we would emphasise equally the paucity of work on determining how graphical representations are themselves represented and how this interacts with the kinds of high-level cognitive processes, such as applying knowledge of content, that Winn rightly emphasises. p. 6

we need to ask what is the nature of the relationship between graphical representations and internal representations and to consider how graphical representations are used when learning, solving problems and making inferences. Such an enterprise means working towards a detailed description of cognitive mechanisms. In this respect we would point to the kind of account offered by theorists such as Larkin and Simon (1987) and Koedinger and Anderson (1990). These offer a model of diagram use which explicitly presents the main elements of a theoretical account, viz. a statement of (i) the properties of the graphical representation; (ii) the way that these properties may be represented internally and (iii) computational processes that mediate between the two. These models thus demonstrate the kind of systematic approach to graphical representation we are advocating. However, we would also argue that focusing primarily on internal representations, as these models do, is not enough - it misses out much of the cognitive processing that goes on when interacting p. 6

with graphical representations and, hence, a useful account of their value. These models, thus do not match a second desideratum: an account which analyses more fully the interplay between internal and external when carrying out a cognitive task. p. 7

Within cognitive science, in general, there has been a move towards promoting the need to analyse the interaction between internal and external representations. In a special issue on situated action in the journal of Cognitive Science, Vera and Simon (1993) stress that, “A fundamental problem for cognitive modellers is to interleave internal and external states in order to achieve naturalistic behaviour” (p12). Norman (1988, 1993) has for several years been describing cognition in terms of ‘knowledge in the head’ and ‘knowledge in the world’. Larkin (1989) has also shifted her thinking from Larkin’s and Simon’s (1987) earlier computational model of diagram use - that focused primarily on internal representations - to considering the role played by external displays in cognitive problem- solving. Others, like Cox and Brna (1995) have been examining specifically the cognitive effects of external representations in reasoning tasks. External representations, here, may refer to both linguistic and graphical forms. p. 7

In our examination of the emerging literature on internal/external representations we have abstracted three central characteristics which we consider as an useful analytic framework from which to explicate aspects of external cognition. These are computational offloading, re-representation and graphical constraining: p. 8

(i) computational offloading - This refers to the extent to which different external representations reduce the amount of cognitive effort required to solve informationally equivalent problems. For example, Larkin and Simon (1987) point to the greater efficiency in geometry problem-solving for diagrams over sentential forms through their ability to provide direct perceptual recognition of geometric relations. Explicitly representing the problem state in diagrams in this way enables solutions to be more readily ‘read-off’. In contrast, solutions for the same problems represented as sentential descriptions typically are implicit and so have to be mentally formulated. This requires a greater computational effort. p. 8

(ii) re-representation - This refers to how different external representations, that have the same abstract structure, make problem-solving easier or more difficult. For example, Zhang and Norman (1994) describe carrying out the same multiplication task using roman or arabic numerals. Both represent the same formal structure, but the former is much harder for people, used to working with the decimal system, to manipulate to reach the solution (e.g. LXV111 x X is much more difficult to solve than 68 x 10). p. 8

(iii) graphical constraining - this refers to the way graphical elements in a graphical representation are able to constrain the kinds of inferences that can be made about the underlying represented world. This characterisation is a term developed in recent work on the value of diagrams for solving formal logic problems by Stenning and colleagues (e.g. Stenning and Tobin, 1995; Stenning and Oberlander, 1995). A central idea is that the p. 8

This is similar to representational guidance, I think Stenning and Oberlander have been cited before, have to read them. p. 8

relations between graphical elements in a graphical representation are able to map onto the relations between the features of a problem space in such a way that they restrict (or enforce) the kinds of interpretations that can be made. The closer the coupling between the elements in the visual display and the represented world, the more tractable the inferencing. p. 9

Although the characterisations might appear to overlap, we see them more as complementary; computational offloading highlights the cognitive benefits of graphical representations, re-representation relates to their structural properties and graphical constraining to possible processing mechanisms. p. 9

we wish to clarify three theoretical issues: (i) the erroneous equivalence of external and internal structure, particularly where it justifies an implicit processing model; (ii) to identify existing, explicit processing models which are of the right type although they do not go far enough in the direction we want and (iii) to clarify the concept of external cognition as referring to the totality of the relationship between external representation, internal representation and their interaction (processing). p. 9

We focus our analysis on two main classes of graphical representations which span traditional and developing technologies: (i) static diagrams and other illustrations that have a role as adjuncts to text (or oral language) and (ii) animations. We then extend the lessons of the analysis to (iii) the area of virtual reality. p. 9

The term ‘representation’ has a variety of different meanings, depending on the context. A common distinction is between representation as process, and representation as product, as the outcome of this process. Process concerns the transformations and preservations that occur in deriving the representation from what is being represented. Description of product is typically concerned with structural characterisations of the representation, for example as image-like, mental model or propositional. Confusion might arise since the two senses, process and product, may be used interchangeably. In fact the two cannot always be easily separated, since characterisation of structural properties is usually related to a particular processing model. Here we shall discuss representation in both senses. p. 10

The classes of static diagrams and animations are considered distinctive, in so far as they have been identified as having different characteristics in the literature. It is acknowledged, however, that there is likely to be some overlap between what constitutes a diagram and an animation, especially for displays that comprise of both static and animated components. There are many different exemplars of diagrams and no single accepted taxonomy that can be conveniently employed to describe them, although there is good evidence emerging for stable classification strategies (Cox and Brna, 1993; Lohse et al., 1991). There may, in fact, not be a single, criterial feature for the term ‘diagram’. p. 10

distinction between representations like graphs, describing quantitative data in two dimensions, and other, less-constrained types. We would probably subscribe to that view but it is not crucial for us here. We would rather adopt a position similar to that of Winn (1987, p. 153) who treats diagrams as representations with the function of being “.. to describe whole processes and structures, often at levels of great complexity”. p. 11

Animations are equally difficult to define and, again, there is, as yet, no single theoretically- or even empirically-grounded classification scheme available. Animations - be they computer, film, video or other media-based -differ from static diagrams in presenting a series of rapidly changing static displays, giving the illusion of temporal and spatial movement. This can be achieved through a range of techniques. For example in ‘multi- dimensional’ animation interdependent objects appear to move in relation to each other; in ‘partial’ animation certain parts of a display move whilst the rest of the display remains static; in ‘artificial’ animation implicit movement is made explicit or processes normally invisible to the eye are made visible. While not an exhaustive classification, we can see the diversity of animation ‘types’. p. 11

The third class of graphical representations that we examine is virtual reality or virtual environments. These are computer-generated graphical simulations, intended to create “the illusion of participation in a synthetic environment rather than external observation of such an environment” (Gigante, p.3, 1993). Images are displayed stereoscopically to the user, via a head-mounted display. Objects within this field of vision can be interacted with via a dataglove or other input device, use of a virtual reality headset can change the field of vision in the virtual world and users can ‘fly’ around the virtual world through gesturing. A major motivation for virtual reality systems is to enable people to become ‘immersed in the experience’ of interacting with external representations (Kalawsky, 1993). However this is difficult to operationalise (Sheridan, 1992) and there is no taxonomy of types of virtual reality immersion. p. 11

Work on static diagrams represents a considerable corpus of research from which it is hard to make generalisations. Winn (1987), reviewing the field, notes that there is an interaction between (at least) ability level, diagram format and task type to be considered in drawing conclusions across studies. p. 12

Larkin and Simon (1987) analysed examples taken from classic physics (pulleys and weights) and geometry (theorem proving) textbooks. Their aim was to develop computational models that allowed a contrast between processing of ‘sentential’ and ‘diagrammatic’ representations which contained the same information about the problem. In p. 12

the first case elements appear in a single sequence, while in the second they are indexed by their location in two-space. Their theoretical analysis suggests that a diagram “preserves explicitly the information about the topological and geometric relations among the components of the problem, while the sentential representation does not” (p.66). p. 13

What is an attention manager? p. 13

The approach taken by Larkin and Simon provides an explicit formalism. The elements of their system are (i) data structures that represent the problem to be solved (ii) productions that contain knowledge of the laws of the domain (the ‘program’) and (iii) an attention manager. They propose that a diagrammatic data structure may differ markedly from an informationally-equivalent sentential one through affording the possibilities of easier search. For example, similar attributes may be clustered at the same spatial location making it easier to recognise them. p. 13

Diagrams, therefore, provide simultaneous information about the location of components in a form that enables objects and their relations to be easily tracked and maintained. This greatly reduces the need to search and recognise. In contrast, sentential representations of the same problem can not provide the same external memory cues. Solving certain kinds of problems using sentential representations thus incurs a much greater computational load, particularly to keep track of how the solution is progressing. Moreover, much of the necessary information to derive a solution is not available from the sentential descriptions and so has to be formulated explicitly. Hence, far more has to be computed mentally to determine possible states and their consequences, than when carrying out the same tasks using problem diagrams such as the one shown in Figure 1. p. 13

In describing their model Larkin and Simon (1987) noted that there was one respect in which diagrams could not be supposed to have an intrinsic advantage - inference-making. p. 13

They observed that when data structures are informationally equivalent “Inference is largely independent of representation..” (p. 71). Bauer and Johnson-Laird (1993), however, postulated that for certain kinds of problems, diagrams should help reasoning, a claim based on Johnson-Laird’s (1983) model theory of deductive reasoning. They investigated the role of external representations, in the form of schematic diagrams, on the solving of deductive reasoning tasks. The problems were double-disjunctive reasoning, which require reasoners to keep track of various alternative states in order to solve them. Because of the difficulty of taking into account many models of the premises subjects are known to perform poorly on these types of problems. Bauer and Johnson-Laird hypothesised that providing diagrams should enable reasoners to keep track of alternative models. p. 14

Bauer’s and Johnson-Laird’s (1993) first reported attempt at developing a schematic diagram to make explicit the alternative possibilities was largely unsuccessful. This they attributed to their using arbitrary and abstract icons for representing explicitly the alternatives, which were found to be of no help to the reasoner (see Figure 2). Their second attempt, however, was more successful. Two types of more concrete diagrams were constructed: one based on an electrical circuit and the other a jigsaw. In both examples, a particular problem-solving context was provided from which to make the deductions. The instructions for the circuit representation of the problem was couched in terms of switches and lights being on or off in the circuit whilst the instructions for the jigsaw representation were expressed in terms of completing a path from one side of the figure to the other. This involved inserting shapes, corresponding to specific people specified in the reasoning problem, into slots in the path, corresponding to particular places (see Figures 3a & 3b). In both examples, therefore, the subjects were required to solve the reasoning task by mentally transforming parts of the diagram. In doing so, the solvers no longer need to solve the problems entirely in their head but can work them out by interacting with the diagrams. p. 14

Interesting the idea of interacting with the diagram. Wonder how this would translate to a knowledge representation, where a student is reading different notes, taking notes, moving notes around, etc. p. 14

Indeed, the results showed that performance was significantly better and faster when using the diagrammatic representations than when solving the same problems using sentential representations. The findings seem to provide further support for the important role of diagrams as external memories, enabling a picture of the whole problem to be maintained simultaneously, whilst allowing the solver to work through the interconnected parts ( cf. Larkin, 1989; Larkin and Simon, 1987; Zhang and Norman, 1994). Although it could be argued that a sentential representation can also act as an external memory aid, the extent of the ‘computational offloading’ is considerably less. The reason being that the problem states and its solution are more explicitly represented in the diagram than in the sentential representation, meaning that less inferencing is required. p. 15

Diagrams as external memories. Similar to more abstract things - the desire to "have it all displayed at once"... Spatial memory seems important. Any studies about the size of the diagram, or about the size of display - working on screen vs. working with multiple print-outs that you can shuffle around etc? Wonder if this is also connected to the issue of scrolling vs paging - with paging you know where things are on each page. p. 15

Consider first the Larkin and Simon (1987) account. Their principal concern is with diagrammatic internal representations, which as Parkes (1993, p. 37) points out provide “ …. access to the properties of the pictures which are posited to facilitate more efficient computations..”. Larkin and Simon (p. 66) describe such representations as having the property of corresponding “ .. on a one-to-one basis, to the components of a diagram describing the problem”. However, their account leaves open the question of (i) how this is produced in human beings and (ii) what work is being done by the external and the internal representations respectively. Consider the following quote (p. 92): “We have seen that formally producing perceptual elements does most of the work of solving the geometry problem. But we have a mechanism - the eye and the diagram - that produces exactly these perceptual results with little effort. We believe that the right assumption is that diagrams and the human visual system provide, at essentially zero cost, all of the inferences we have called ‘perceptual’”. It is hard to get a precise understanding of how ‘perceptual inferences’ p. 15

might work but Larkin and Simon do offer a suggestive analysis of the metaphorical sense of (mentally) ‘seeing’ as referring to inferences that are akin to perceptual experiences in being based on “productions with great computational efficiency” (p.71). This conceptual issue notwithstanding there is a further problem. As Koedinger and Anderson (1990, p.518) note the Larkin and Simon argument “… is based on an assumption that perceptual inferences are generally easier than symbolic inferences” but “ … it is possible that perceptual inferences appear easier because, in general, they have been much more highly practised than symbolic inferences”. In other words the value of diagrams in such situations is strongly related to experience and expertise, of the individual having ‘operators’ that match the display (cf. Larkin, 1989). Novice physicists, for example, will not make the same inferences as experts from the same diagram (cf. Anzai, 1991). p. 16

The Bauer and Johnson-Laird (1993) explanation of what subjects are doing when using diagrams to solve the logic problems is also debatable. It seems unlikely that they are using the diagrams to compare alternative models, as postulated by Johnson-Laird’s model theory. Alternatively, it appears that the external representations (the combination of the diagram and analogical instructions) have re-represented the original deduction problems into simpler and different tasks. In their own words, Bauer and Johnson-Laird (1993) explain: “in the case of the diagrammatic problems, the subjects form a visual representation of the diagram, and in their mind’ s eye they can imagine moving the pieces or switches (i.e. they carry out visual transformations of images). Bypassing the construction of the meanings of verbal premises and manipulating visual images appears to reduce the load on working memory and to speed up the process of inference.” (p373). p. 16

Therefore, rather than facilitating the reasoner to envisage all the alternative possibilities inherent in the premises it seems more likely that the combination of instruction and diagram is constraining a particular way of conceptualising and solving the logic problem. This account fits in better with Stenning's and Oberlander’s (1995) theory of specificity which p. 16

differs from the model theory by postulating that “graphical representations such as diagrams limit abstraction and thereby aid processibility” (p.2). This is achieved through the information available in the diagram restricting the possible interpretations of the problem and in so doing guiding the reasoner to make the correct solution. Thus, certain diagrams are more effective than others because they exploit better the constraining properties of varying graphical forms. For example, Stenning and Tobin (1994) claim that Euler's Circles (see Figure 4) are more effective than 3-D cube diagrams in helping subjects solve logic problems because the geometrical constraints of the intersecting circles represent the logical constraints much better. In other words, a diagram is more likely to afford a particular reading of the problem and way of solving it than a sentential representation because it is less expressive (i.e. decreases indeterminacy). Having built a mental model of the combined external representations (the instructions and the diagram) that satisfies the premise in their own minds, it is unlikely that the solvers will then build alternative, but equally plausible, models of the problem (Cox and Brna, 1995). p. 17

So diagrams constrain the possibilities, this could be good (speed up problem solving) but if you choose the wrong visualization, doesn't it also lead to the risk of choosing the wrong solution? In line with our discussion in class, wonder if there has been research on offering multiple different visualizations to users - do these users do better (speed/accuracy) than users who only have one randomly assigned visualization? p. 17

In Hegarty’s study the graphical representations used were static canonical diagrams of p. 17

pulley systems whilst in Kaiser et al.’s study, both static and animated canonical graphical representations were used to depict objects falling, being severed, or being displaced from various dynamical systems (e.g. pendulums and moving planes). The primary aim of Hegarty’s study was to ascertain the extent and form of mental animation that occurs when making judgements about the motion of pulley systems. In contrast, the main objective of Kaiser et al.’s study was to determine how external animations enable more effective judgements to be made about the trajectories of moving objects compared with static diagrams. In both studies, subjects’ were required to reason through initially comprehending a verbal problem together with a static or dynamic graphical representation used to convey the problem state, and then predict correctly future states or trajectories of part of the system depicted in the graphical representation. p. 18

Hegarty’s central idea is mental animation, which involves simulating mentally, in a serial manner, components in the graphical representation of the pulley system. An obvious reason for this is that we are unable to animate all parts of the diagram at once, due to the constraints of working memory. It also seems plausible, given that we can only perceive the working of certain aspects of a real world pulley system at any one time, depending what is in our field of view at that time. With real-world pulleys, however, the motion of each part is always available; we need only to follow the way the components move to make judgments about them. Moreover, we can do this in a haphazard way. With diagrams, however, Hegarty argues we make inferences about the motion of the static parts by following the temporal order of the causal chain of events from input to output. p. 18

This seems to be only about diagrams that directly depict physical reality. What about statistical graphs, or even completely abstract mind maps etc? Hope they will be mentioned. p. 18

In contrast to Hegarty’s approach, Kaiser et al. (1992), explain reasoning about mechanical systems in terms of what the external representation does for the learner. Like Hegarty, they stress the importance of information being processed sequentially, but in terms of the external representation being able to ‘temporally parse a multi-dimensional problem into unidimensional components’ (p. 671). In doing so, they propose that the distinct state changes that have to be recognised to make correct judgements about the system are made more obvious through an animation than with a static display. The idea that the external representation does the ‘temporal parsing’, rather than the problem-solver having to do it, is illustrated with an example of common-sense reasoning about the C-shaped tube physics problem (based on McCloskey et al., 1980). The main finding is that when the problem is represented as a static 2-D representation (see Figure 5), students often incorrectly infer that the projected motion of the ball on exiting the curved tube continues in a curvilinear trajectory. Kaiser et al. found the same effect for both free choice and forced choice conditions. However, when shown various incorrect and the correct animation sequences in a forced choice situation, students invariably selected the correct ‘straight’ trajectory. Kaiser et al. explain this performance shift in terms of the animation temporally segregating the ball’s behaviour in the extended body system (whilst in the tube) from when it behaved as a point particle (after exiting the tube). It is this visible temporal parsing that is claimed to make the change in states more obvious to the subjects and allow them to make the correct inference. p. 19

The animation studies present a different form of the problems encountered in the static p. 19

diagram research. In particular they assume that running mental models and parsing external animations use the same structures and functional processes as when perceiving real-world dynamic systems. For example, in an earlier study Hegarty et al. (1988) proposed that people decided which attributes of a system were relevant to judging mechanical advantage on the basis of ‘causal models’ of mechanical systems arising from relevant (physical) experience with such systems. The appeal to an equivalence in processing, however, does not help us in understanding the merits of different forms of graphical representation in terms of how they are processed and interacted with for various tasks. A more pertinent question to address, then, is how do we understand and make connections between the static and animated forms that represent the dynamic processes of real systems such that we can make inferences about them both? p. 20

The same lack of specificity is seen in Kaiser et al.’s (1992) explanation of the superiority of animated over static forms in terms of the visible temporal parsing of the ball in the container system, making change in states more obvious. An alternative explanation for the difference in performance could be in terms of experience with the two representational formats, reflecting more a difficulty with interpreting the canonical forms in the diagram in relation to the problem that had to be solved rather than one of not being able to recognise the significance of the temporal parsing of the objects in different states. This objection reiterates our concern that there is a crucial role for expertise and practice that is not being recognised. Most importantly, whilst providing further support for the value of the explicitness inherent in animations, Kaiser et al.’s study offers no explanation of the cognitive mechanisms involved in learning and reasoning with animations. Pedagogically, too, it is unclear how animation can facilitate learning or problem-solving. In particular, we would emphasise the absence of an analysis of how a better level of understanding can result from seeing objects moving explicitly as opposed to having to imagine how they move. Indeed, Kaiser et al. comment on how subjects who had been shown the animation first and then a static diagram of the same problem performed no better than those who had p. 20

just been shown the static diagram. Here again, we have further evidence that the benefit of viewing an animation is transitory and not readily mapped onto the static representation with its more arbitrary conventions. p. 21

How do viewers identify the ‘key’ features and constraints of a graphical representation and then map them onto the relevant aspects of the problem to be solved? Further, there seems to be a real issue about what problem the subjects were ‘really’ solving and how far expertise and experience are central factors. It is often hard to separate general claims about graphical representations per se from factors that have to do with individual differences in ability in the subject or understanding of the domain-specific genre of the diagrams involved. In domains with highly evolved notations such as geometry or physics, diagrams are not ‘merely’ aids for solution but play an essential part in the process of knowledge acquisition and depiction. A circuit diagram, an architectural plan or a mathematical notation comprise a set of meaningless symbols to the uninitiated; they only take on their intended meaning through learning the conventions associated with them. In a real sense it is impossible to develop expertise in these subjects without the ability to both read and produce diagrams of a particular sort (cf. Anzai 1991). This strongly suggests that - in such domain-specific cases at least - diagrams can only trade on established domain knowledge to be effective. The point is well-put by Larkin and Simon (1987, p.71): “If students lack productions for making physics inferences from diagrams, they … will find them largely useless”. p. 21

two of our major themes, that there is a lack of an adequate cognitive processing model and that focusing on the externality of graphical representations to see p. 21

how they work is of crucial importance to a better understanding. p. 22

4.1 Processing and the resemblance fallacy? One problem with the Kaiser et al. (1992) study noted above was the lack of explanation of how subjects recognise the temporal segregation of the objects as being central to an understanding of mechanical systems. This seems to be because the external and internal representations are assumed simply to have the same characteristics. This is an example of what we shall call the ‘resemblance fallacy’, which has a much wider appearance in the graphical representation literature and may help to explain something of the apparent unwillingness to specify processing models. It is prevalent, we believe, because the structure of graphical representations, their spatial/iconic/figural qualities, promotes an intuition as to their value as an input for perception/cognition whereas the reality is that we have no well-articulated theory as to how such an advantage might work. Evidence for our over-reliance on such intuitions can be seen by examining the kinds of arguments that have been made for the links between graphical representations, perception and internal representations. The possibility of different representational formats - image/proposition/mental model - and their properties interacts with the issue of how graphical representation might work. p. 22

Consider first, work on conventional, static graphical representations. By design these are typically well-suited to the information pick-up capacities of the visual system - object perception, search, pattern-matching etc. Analysis of their syntactic properties in these terms, e.g. how elements are grouped, what affects discriminability etc., is therefore productive. However there is a problem in going beyond these data to identifying the form of internal representations that result. A particularly common argument has been that the quasi-pictorial qualities of images suggests a privileged link with pictorial input. For p. 22

example Winn (1987, p.159) summarises the relevance of work on imagery to graphical representation thus: “These studies exemplify a body of research that leads to the following conclusion: Graphic forms encourage students to create mental images that, in turn, make it easier for them to learn certain types of material”. And Reed (1993, p.299) claims a “substantial similarity between the functional equivalence of pictures and images”, stating that: “We would have a better understanding of how images aid problem solving if we had a better understanding of how pictures aid problem solving”. p. 23

while pictures can undoubtedly serve to stimulate imagery under certain circumstances (e.g. Finke, 1990) it is by no means clear that they are necessarily represented in this way. Halford (1993) points out that we do not have to accept any more than a mapping between relations for an external representation-internal representation pair. In addition there is some doubt about the extent to which imagery is computationally important and processing may be better explained in terms of other representational forms (e.g. Anderson, 1990; Molitor et al., 1989; Pylyshyn, 1973). In short the case for an intimate relationship between graphical representation and images may not be logically compelling and is currently heavily under-specified. p. 23

The prevalence of image-based explanations for graphical representation is, presumably, based on the premise that images facilitate cognitive operations on analogic representations. This has benefits for basic processes like optimising search and reducing the load on working memory (e.g. Larkin and Simon, 1987). However these same benefits have also been invoked for mental models and these, too, have been proposed as representations for graphical representation. Since the claims for images and mental models are close it is unsurprising to see some link proposed between the two, e.g. Hegarty (1992) for people working on pulley system problems: “… running a mental model involves transforming mental images… people infer motion of mechanical systems by transforming mental p. 23

images…” . Similarly Bauer and Johnson-Laird (1993) talk about the “manipulation of visual images”. Thus, while the mental model resulting from interaction with graphical representations need not be thought of as image-like (cf. Glenberg and Kruley, 1992), there is often an apparently close relationship posited between the two and, hence, the possibility of another intuitive and unsubstantiated link between picture input and representation type. Molitor et al. (1989, p.10) describe the situation thus: “In mental models reality is represented in an analogous, predominantly imaginative form”. However the same authors also note that the mental model construct is “very fuzzy and used in a different sense by each author” (p. 10), a conclusion echoed by Hong and O’ Neill (1992). As with images such a range of possibilities argues for caution in positing a necessary relationship between the pictorial nature of graphical representation and a particular representational format. At the very least, different graphical representations and/or tasks may engender different kinds of representations and this must remain an issue for future research. p. 24

there is an assumption that ‘adding’ animation to an equivalent static display will be advantageous. But why should this be so? In articles about animation we commonly find an intuition-led chain of assumptions, echoing the same causal chain of reasoning used to account for the efficacy of diagrams (external representation producing a mental image or mental model which in turn results in better learning or reasoning) that was criticised earlier. Here, an illustrative line of reasoning goes something like this: animations can show motion explicitly and ‘directly’ and hence provide more accurate information (Kaiser et al., 1992); this reduces processing demands on working memory allowing other tasks to be performed (Rieber and Kini, 1991) and enables more ‘useful’ mental models to be formed for solving problems (Park and Gittleman, 1992); these in turn facilitate learning or reasoning. Whilst the first part may be factually accurate, the rest does not logically follow. p. 24

As argued above in relation to diagrams, we cannot simply assume a privileged relationship between a graphical representation of a system - in this case an animation - and someone’s understanding or ability to reason about it, by virtue of its resemblance, albeit highly simplified and schematised, to the dynamic properties of a real-world system. As with diagrams used in specialised domains, e.g. physics or geometry, a person has to learn to ‘read’ and comprehend the significance of the content of the animations in relation to other information that is being presented verbally or as text and to assimilate this to their current understanding of the domain. This requires making multiple connections between what the animations are intending to convey and the abstract concepts that are being learned about. How students integrate information arising from different representations of knowledge is crucial (Laurillard, 1993). p. 25

Sounds like video would be the best? Or a perfect 3D representation, but of course sometimes diagrams should reduce the information to highlight specific aspects. Like layers on a map. p. 25

4.2 Processing and the internal/external The force of our comments, however, is not solely to do with being less intuitive in our accounts. Consider the claim by Larkin and Simon (1987, p. 97) that: “mental imagery - the uses of diagrams and other pictorial representations…held in human memory… play a role in problem solving quite analogous (to) external diagrams… and that this role is also played by the internal and the external in concert.” (our emphasis). How do the internal and the external act “in concert”? Clearly if we set up our processing model so that both representations have the same structure/characteristics the problem seems less apparent - and the solution becomes obvious too. But it seems clear that what is really needed is an integrated approach that analyses the dynamic cognitive processing of graphical representations; how this understanding is integrated with existing knowledge, what information gets lost, misinterpreted or correctly interpreted, and how is it subsequently re- represented, externally. p. 25

Larkin (1989) has tried to tackle the problem by outlining a computational model called DiBS, that represents information available in external displays as data structures that enable internal operators to be cued as to know what to do next. The model's central searching mechanism is based on the observation that 'each step requires only looking at the display, and doing what it suggests, without more effortful mental calculation or storage' (p319). DiBS, therefore, works largely by manipulating attributes of the external display. The examples that Larkin has chosen to represent in her model are well suited to the transformation of external data structures. They include simple everyday problems (e.g. brewing coffee) and textbook problems (e.g. linear equations) that once learned become highly routinised and error-free. Hence, for these kinds of tasks there is no need to activate any internal representations other than a very general mechanism that is characterised as knowing 'where an object wants to go' in each step of the task. As such, DiBs is more a model of how external representations cue a set of automatic procedural-based actions for solving clearly defined problems. In its current state, therefore, DiBs does not account for other kinds of learning and less well-defined problem-solving tasks, involving more complex cognitive processing, where combinations of interacting internal/external mechanisms are likely to come into play. Inspired by Larkin's (1989) ‘move to the external’, a number of empirical studies have been carried out since, in an attempt to analyse in more detail the properties of display-based problem-solving. For example, O'Malley and Draper (1992) differentiate between the knowledge users need to internalise when learning to use display-based word processors (e.g. MacWrite) with that which they can always depend upon being available in the external display. The tendency, therefore, appears to be for users to learn only what is necessary to enable them to find the information they require in the interface display. Information represented in such displays is viewed as an external memory aid “which 'fill the gaps' in users' internalised representations when they interact with the system” (p. 86). p. 26

How the gaps are filled and what cognitive mechanisms are involved, however, is not clear. This is also true of Zhang and Norman's (1994) recent analysis of distributed cognitive tasks (between internal and external representations), where they argue that an analysis of the relation between different external forms of representing the same abstract problem (in this case the Tower of Hanoi) is necessary before considering the processes that are activated when solving the problem. The study was designed so that in certain conditions subjects had to internalise several rules to carry out the task whilst in others the same rules were embedded in the external display. Their findings indicated that the fewer rules subjects had to internalise the easier it was for them to perform the problem-solving task. The implication is that external representations can significantly change the nature of a task through constraining the permissible moves allowed in solving the task. Furthermore, this form of ‘computational offloading’, i.e. implicitly embedding rules in the external representation as opposed to making subjects internalise them, is thought to reduce the load on internal working memory providing more ‘space’ for planning subsequent moves. Although Zhang and Norman did not investigate the processes involved in interacting with graphical representations in their experiment, they do suggest that the nature of the relationship is likely to be uni-directional: “perceptual processes are activated by external representations while cognitive processes are usually activated by internal representations” and that moreover “different processes are activated by different representations” (p118). We would argue, however, that the interplay between internal and external representations in problem-solving is likely to be more complex, involving cyclical, interacting processes, especially when considering how graphical representations are both perceived and acted upon. p. 27

the central concept of ‘immersion’ was an elusive one. One way of characterising it is in terms of realism. It is often suggested that a main advantage of virtual reality is that simulations can be constructed to have a higher level of fidelity with the objects they represent, compared with other kinds of external representations, e.g. static diagrams, in terms of having more functional, physical and spatial resemblance. The illusion afforded by the virtual reality technology can make virtual objects appear to be more realistic and to behave according to physical laws. For example, terrains developed for flight simulators can seem very life-like, giving a spatial awareness that can closely approximate that which would be obtained in the real world. virtual reality researchers are inclined to believe that learning and training applications, e.g. aviation and defence can be improved through having a greater fidelity with the represented world (e.g. Cover et al., 1993). One suggestion is that the transfer of training to the real world could be easier, with less errors. The push towards developing life-like simulations, however, raises the question of what are the cognitive benefits of representing the world at higher levels of realism than at higher levels of abstraction. The intuition that perceiving and acting in an artificial environment, that is designed to simulate a physical world as closely as possible, will provide a better learning or training experience through the development of better spatial cognition is reminiscent of the ‘resemblance fallacy’ discussed in the previous sections. Similarly, it p. 28

may prove to be as incorrect. For example, results from a recent study investigating transfer of training in virtual reality systems found that subjects learnt performance characteristics specific only to the virtual reality context, which were of no use when carrying out the same task in the real world (Kozak et al., 1993). These preliminary findings suggest, therefore, that the actual experience of being immersed in a virtual reality world is quite distinct from interacting with real world artefacts. The value of virtual reality, therefore, should not be assumed to come about through a structural and spatial equivalence between the virtual reality simulation and the real world. p. 29

Is this a general rule, or does it depend on the fidelity of the VR environment? p. 29

Another problem is the degree of interactivity within the virtual environment. Currently there exists largely indirect means of interacting with the virtual graphical representations - typically via selecting objects through moving customised joysticks and trackballs, gesturing through data gloves or more crudely through using a keyboard. This contrasts sharply with the high level of interactivity that exists with six degree full motion flight simulators. Not only do they have a real set of flight controls but they also have highly realistic feedback - all of which is well integrated with the animated graphical representations of the physical terrains that are flown over. p. 29

how best to constrain virtual reality simulations to provide external representations that are effective for training (cf. to the earlier idea about diagrams and graphical representations). In particular, the virtual environments need to be designed to guide the learner to the crucial aspects that are necessary for performing the appropriate activity for a given task at a given time. The issue then becomes one of determining what aspects of the represented world need to be included and how they should be represented, p. 29

what aspects should be omitted and what additional information needs to be represented that is not visible in the real world but would facilitate learning. From a cognitive perspective, it enables us to assess the benefits of virtual reality in terms of the processing mechanisms that operate at differing levels of abstraction of information. For example, we can analyse differences in task demands and performance characteristics for specific tasks, e.g. taking off or landing for different virtual reality simulations, ranging from presenting simple canonical structures (e.g. schematic outlines) to more fully rendered depictions of scenes. Hopefully, this way the pitfalls of the resemblance fallacy can be avoided. p. 30

Again, interesting to think about what a VR environment for dealing with abstract ideas would look like - where there is no "real physical reality" to compare with. p. 30

Another way in which the notion of virtual reality immersion has been characterised is in terms of ‘steering’ the interaction. Here, the intuition is that virtual reality simulations provide more opportunities to visualise and manipulate the behaviour of abstract data structures or processes which are not normally visible to the naked eye. For example, NASA have developed a Virtual Wind Tunnel, whereby a scientist (who is a computational fluid dynamicist) controls the computation of virtual smoke streams by using the finger tips (Gigante, 1993). Abstract equations for the computed airflow around a digital model of an aircraft are translated into visible smoke streams. By moving around the virtual aircraft in the virtual reality environment and visualizing the smoke streams, the fluid specialist is thought to be able to “discover areas of instability, separation of the flow from the aircraft’s surface, and other interesting phenomena” (Gigante, 1993, p 9). An assumption is that better mental models of the abstract processes will develop through making these kinds of processes more concrete (Gigante, 1993). It is also assumed that on-line problem-solving will be facilitated. Whilst not making any explicit links between the resulting mental structures and cognitive processes there is a sense that the two are connected. What we do not get a sense of, however, is how experts, who are highly familiar with abstract representations and have to interact with them in their work, are able to transfer between these forms of representation and the concretised visual representation of the same problem space in the virtual reality simulation. The value of being able to ‘steer’ a physical p. 30

simulation should be analysed, therefore, in relation to how it integrates with ways of interacting with other existing forms of external representations in professional practice. p. 31

two factors are clearly crucial: the kind of notation (e.g. the symbols of visual programming languages, universal icons) and the visual organisation that is used to structure them. The notation that is employed should be appropriate (easily understandable). This is true both for the reading and for the production of diagrams. Merrill and Reiser (1993, p. 12), describing students learning LISP, observe that “the requirement to translate into an external notation system that does not match well with the structure of the students’ plan imposes an additional working memory load of continually mapping between the two representations”. Secondly we should recognise the importance of the canonical forms of diagrams, e.g. recognising a diagram comprising of a set of images connected by arrows as a ‘cycle’ may be critical. The existence of such forms (capable of supporting many different kinds of content) has long been recognised by researchers (e.g. Amigues & Caillot, 1990; Anzai, 1991) and is obvious in the culture at large (cf. Lohse et al. 1991). One advantage of conventional 2D diagrams is that they can trade on such recognition by activating appropriate ‘readability rules’ (Sugiyama and Misue, 1991) and by cueing appropriate kinds of inferences in the reader. Cox and Brna (1993) noted that there was a strong correlation between a student’s ability to identify types of graphical representation (e.g. diagram, map, table) and their ability to use them to solve problems. Pedagogically, encouraging as wide an acquaintance with different forms of p. 32

Link to "visual literacy" p. 33

diagrams is clearly important for their effective use but it may also be that experience with static forms itself may be a useful precursor to the ability to read more dynamic ones. p. 33

Under such circumstances we need insight into how people read and interact with diagrams. The stress here is important for our ideas about developing good diagram skills. In the vast majority of studies and analyses of static diagrams the assumption has been that the subject does nothing to change the external form. This may well be true for cases such as library books or slides in a presentation but it may not be the optimal case. Koedinger and Anderson (1990) observe that high school students frequently use annotations to problem diagrams to hold together information needed for inferences. Such a strategy may also be performed mentally but, given previous arguments, is probably more efficient for learners when external. In fact the logical conclusion of these arguments is to maximise the load on the external representation. As has been observed in several cases making a ‘cognitive trace’ available for problem-solving is of great benefit (e.g. Merrill & Reiser, 1993). One lesson for diagram use might, therefore, be to promote opportunities for external manipulation (i.e. cognitive tracing) as well as encouraging production skills, as we noted above. p. 33

Good diagram design also has the crucial requirement that the degree of abstraction of material should be appropriate to the varying demands of the task and learner’s ability. Levonen and Lesgold (1993) describe SHERLOCK, a computer-based electronics coaching system, which has the facility for representing both realistic (picture-like) diagrams of the system and schematised expert representations of the same domain. Switching between the two enables a kind of apprenticeship learning. Consistent with this approach Cheng (1993) advocates the availability of multiple representations, from specific examples to overviews, which learners could choose to look at as they wished. While this raises issues about integration between different views it also emphasises the importance of learner control. There is no reason to doubt, for many situations, that multiple representations could be made available within a 2D diagram. p. 33

Finally getting to the idea of making lot's of different representations available. p. 33

We suggest that it is useful to begin to formalise them as a set of general conceptual design issues, akin to the set of cognitive dimensions of notations that Green (e.g. 1989, 1990) has advocated, for describing important features of the design of programming languages and software tools to support users’ tasks. p. 34

• Explicitness and Visibility Diagrams, animations and virtual reality can in their respective ways all make salient certain aspects of a display. A design objective, therefore, should be to facilitate perceptual parsing and inferencing, through directing attention to key components that are useful or essential for different stages of a problem-solving or a learning task. In addition, the various graphical representations can represent ‘hidden’ processes which underlie complex phenomena. The aim, here should be to facilitate higher level understanding, i.e. cognitive inferencing but also in relation to how this interacts with perceptual processing. As with ‘perceptual inferences’, the users may need much prior knowledge in knowing how to interpret what is shown. p. 34

Cognitive tracing and interactivity Diagrams that have been already constructed allow the user to leave cognitive traces, i.e. mark, update and highlight information. However this is a limited function. There is no p. 34

possibility of interaction or feedback - the user cannot test new configurations. In contrast, when interacting with animations and virtual reality objects there is more scope for providing feedback but less for leaving cognitive traces. For example, various parameters of a computer-based model can be set in a virtual reality or 3-D simulation (cf. microworlds) and the outcome directly observed. Graphical representations should be designed with a view towards how they support different kinds of cognitive tracing and levels of interactivity. p. 35

• Ease of production Related to the above issue is ease of production of a graphical representation. It appears, that diagram production and comprehension are intimately related. A history of being taught to draw diagrams makes for fewer problems with understanding new ones. This is particularly important for domains where evolved notations are crucial. However, where the possibility of acquiring expertise is limited, the demands of reading the diagram efficiently may be too great. Recent software developments now make it possible for users to select alternative or partially animated views of the same process, and to play (and replay) them at different speeds, thus enabling multiple abstractions to be interpreted. Furthermore, software is being developed that will allow novice users easily to construct their own animations through compiling components from a toolbox of animations or modify pre- designed animations. The hypothesis about diagram production and comprehension could be tested for these more interactive forms of animations: having a better understanding of how to create animations will enable people to have a better understanding of how they work and what they are trying to convey. p. 35

Creation of diagrams and visualizations is important for gaining visual literacy. p. 35

• Combining external representations The conventions of constructing 2D diagrams have largely evolved to be complementary to textual expositions. In some cases it may be that text is indispensable for understanding the function of a particular diagram. However fairly mundane factors such as spatial separation of text and diagram may significantly increase the computational load involved in comprehension (e.g. Sweller et al., 1990). In contrast, animations and virtual environments p. 35

have been designed to be largely graphical, although they may be accompanied by spoken narration or verbal text. Studies have shown that it can be more difficult to integrate written text with these kinds of graphical representation than with static diagrams. For example, response times from Rieber’s (1989) study of combining text with animations to represent Newton’s Laws of motion indicated that the subjects simply viewed the animations and then moved immediately onto the next screen of information without reading any of the accompanying text. Other studies which have combined spoken narration with animations, however, have fared better, showing that this combination is more effective. For example, Mayer and Anderson’s (1991) study of subjects’ understanding of the operation of a bicycle tyre pump, showed comprehension to be better when the information was depicted as an animation with concurrent narration, than when presented just as an animation. Having parallel auditory narration could also be effective for virtual reality to guide users in exploring and interacting with the environment. Hybrid graphical representations could also be developed that allow users to interact with static diagrams on a computer display by adding animations or conversely, allowing users immersed in a dynamic virtual reality environment to interact with static objects (e.g. jotting notes into a virtual notebook). The objective, here, would be to provide support for different kinds of interactivity. p. 36

• Distributed graphical representations While we have not commented on this aspect in the paper, diagrams offer the possibility of joint evolution of representations, e.g. in idea-sketching, where planning can be facilitated using notations in any framework that suits the task. Here temporary conventions can be set up, which also has the drawback of being potentially unintelligible at some later date or to others. Various shared drawing tools also have been developed to support collaborative sketching and designing (e.g. Ishii and Kobayashi, 1992; Scrivener et al., 1992). Collaborative design sessions can be recorded which when played back re-construct the collaborative drawings as animations. virtual reality environments can also provide opportunities for virtual construction of graphical representations for users in geographically dispersed locations. However, the value of enabling collaborative construction and editing p. 36

of graphical representations in terms of enhancing task performance is only beginning to be researched. p. 37

we promote an alternative approach that analyses how different graphical representations work in terms of core ‘external cognition’ processes and properties of the graphical representation, e.g. computational offloading, re-representation and graphical constraining. We believe that such an enterprise is central to evolving a more adequate account of the cognitive benefits and mechanisms involved. p. 38

Related to this is a further, critical and under-acknowledged theme, that of interactivity. Specifying how people interact with graphical representations, when learning, solving problems and making inferences, is complex since it will involve not only a specification of the cognitive mechanisms alluded to above but also some sense of the behavioural aspects. For example the fact that students prefer to mark diagrams as they work, the established value of cognitive traces and the dialectic between graphical representation production and use all point to a need to conceptualise graphical representations as more than passively observed, with obvious implications for design and innovation. In turn the potential significance of such activity will be a function of variables such as the level of experience with the graphical representation and knowledge domain, type of task and abstractness of information being represented. Many of the presumed benefits of good-old fashioned graphical representations (i.e. static diagrams) were considered to be due to years of practice of perceptual processing of visual stimuli and the learning of graphical conventions. This may help us to understand why advanced graphical technologies (e.g. animations and virtual reality) have not, as yet, been able to demonstrate comparable performance or learning benefits. Similarly we have even less understanding of how (and if) computational offloading works in such dynamic environments. There may also be contingent problems such as the fact that the temporal constraints of existing kinds of ‘passive’ animations and ‘immersed’ virtual reality may only allow for shallow processing of information (e.g. p. 38

Palmiter et al., 1991; Philips, 1986), thereby preventing them from having the equivalent computational benefits that static diagrams offer. p. 39

Bauer, M. I., & Johnson-Laird, P. N. (1993) How diagrams can improve reasoning. p. 39

Cox, R. & Brna, P. (1993) Analytical reasoning with external representations. In: R. Cox, M. Petre, P. Brna & J. Lee, Eds. Proceedings of the AI-ED93 Workshop on Graphical Representations, Reasoning and Communication.August, Edinburgh. pp. 33-36. Cox, R. & Brna, P. (1995). Supporting the use of external representations in problem solving: The need for flexible learning environments. Journal of Artificial Intelligence in Education, 6 (in press). p. 40

Laurillard, D. (1993) Rethinking University Teaching. London: Routledge. p. 42

Merrill, D. & Reiser, B. J. (1993) Scaffolding the acquisition of complex skills with reasoning-congruent learning environments. In: R. Cox, M. Petre, P. Brna & J. Lee, Eds. Proceedings of the AI-ED93 Workshop on Graphical Representations, Reasoning p. 42

Norman, D. (1993) Cognition in the head and in the world. Cognitive Science, 17 (1), 16. p. 43

Stenning, K. & Oberlander, J. (1995) A cognitive theory of graphical and linguistic reasoning: logic and implementation. Cognitive Science, 19, 97-140. Stenning, K. and Tobin, R. (1995) Assigning information to modalities: comparing graphical treatments of the syllogism. In: E. Ejerhed & S. Lindstrom, Eds. Action, Language and Cognition. Proceedings of International Conference on Dynamic Semantics, Umea: Sweden. September 1993. Kluwer. (in press) p. 44

Winn, W. (1987) Charts, Graphs, and Diagrams in Educational Materials. In: D. M. Willows & H. A. Houghton Eds. The Psychology of Illustration. Volume 1 Basic Research. New York: Springer-Verlag. pp. 152-198. p. 44

Winn, W. (1993) An account of how readers search for information in diagrams. Contemporary Educational Psychology, 18, (2), 162-185. Zhang, J. & Norman, D.A. (1994) Representations in distributed cognitive tasks. Cognitive Science, 18, 87-122 p. 45