Challenges in representing information with augmented reality to support manual procedural tasks

1.   Introduction

With the development of Industry 4.0, the machines used are becoming more and more complex. This causes the non-automated, manually performed tasks such as repair or maintenance, which are generally called procedural tasks ^[38], to become increasingly demanding. Augmented reality has shown its potential to support such tasks by providing spatially referenced information in situ ^[68]. This ability allows a faster and less error prone completion compared to traditional instructional media like paper manuals or digital instructions on computer screens. However, this way of presenting information to a user is very invasive in the visual field. Every virtual element that is introduced diminishes a part of the physical environment. This may lead to unnecessary cognitive and sensory effort, i.e. cognitive and sensory activity which is not directed at completing a task but at perceiving and processing information from virtual as well as physical sources. Examples are virtual objects in front of similarly colored backgrounds that take time to recognize, covered information that needs to be cognitively reconstructed or an unclear relationship of virtual and physical objects. Designers of augmented reality applications need to be supported in their task of presenting virtual information such that these problems are minimized.

During the last decades many aspects of human perception like color ^[62,63] or depth perception ^[28,64] have been thoroughly researched for augmented reality. Also some visualization methods for typical problems in augmented reality have been developed. Examples are the virtual window ^[28] for improved depth estimation or focus and context visualization ^[54] for better integration of virtual and physical objects. Further guidelines for traditional HMIs ^[20] or for VR have been transferred to augmented reality or new guidelines for specific use cases like training ^[103] have been developed. What is missing so far is an analysis of possible sources for unnecessary sensory and cognitive effort that stem from inappropriate use of augmented reality to support manually performed procedural tasks.

In the following, a conceptual framework for modeling information representation in augmented reality is described. Is is not concerned with the actual presentation or visualization of information but instead is concerned with its representation. This can be seen as focusing on the containers of information which are included into an augmented reality scene ^[99]. The intention is to explore possible representation challenges when creating augmented reality applications that support manual execution of procedural tasks and understand why they arise. The basic idea behind the framework is that physical and virtual objects exist which are relevant for a user when working on a procedural task. Information representation must react to the need to clearly see all of them and their relations. Such objects can be divided into multiple classes that have different characteristics concerning their relation to the physical environment as well as their manipulability by software. This classification can then be used to identify sources of unnecessary sensory and cognitive effort and describe those in form of challenges.

The framework is not concerned with user interaction. While this is an important part of most augmented reality applications, it introduces at lot of new aspects that go beyond the scope of this paper. Instead, a well chosen information representation should reduce the needed user interaction and support a user being able to concentrate on the actual task. It is also not concerned with a specific device type. Instead the focus is on modelling information representation with augmented reality and identifying general representation challenges. However, actual solutions for these challenges can and probably will depend on the type of device.

Since its first draft was published ^[71], this framework has been fundamentally revised. The modeling of information representation in augmented reality and the classification scheme for information objects have been refined. The challenges were concretized and are now derived and defined on the basis of the modeling. The preliminary literature research has been replaced by a more detailed one.

This paper is organized as follows: First an overview of related work is given. Then the components of this framework are presented and their characteristics are discussed. Based on those, the five challenges clarity, consistency, visibility, orientability and information linking are identified. For each of them a description, a derivation and a literature overview is given. At the end a summary is made and approaches for future research are outlined.

2.   Related work

Augmented reality was defined by Azuma ^[4] as having the three following characteristics:

● Combines real and virtual

● Interactive in real time

● Registered in 3-D

Especially the first characteristic is very relevant, because it leads to the questions how humans perceive this combination and which issues and new challenges for information representation arise from it. Kruijff et al. ^[56] identified three categories of problems that affect perception and as a consequence understanding of information presented in augmented reality as well as five possible sources of them. For each combination of problem category and source they list which problems affecting perception may occur and what their cause is. Their approach is very broad and tries to capture many aspects of augmented reality including ergonomic factors such as device properties. Aspects relevant for information representation are only a small part and are not systematically approached.

Drascic and Milgram ^[22] and Livingston at al. ^[62] also created comprehensive overviews of perception in augmented reality but did not apply a comparably structured approach. Instead they identified problem areas, described known perceptual issues and, where applicable, possible solutions for them. Again, information representation was not in the focus of their work.

Several detailed studies that cover specific aspects of perception in augmented reality have previously been published. Focus of research have been topics like color perception, e.g. ^[42,62], text legibility, e.g. ^[40,41] or depth perception, e.g. ^[28,64]. Their findings are an important input to overcome the challenges which arise for information representation and will later be included when the challenges are discussed.

Guidelines for 2D user interface design already exist since many decades and at first glance it seems obvious to transfer those to augmented reality. However, augmented reality user interfaces are inherently different from those for traditional computers or mobile devices and even though it is possible, the resulting recommendations are rather general ^[20]. A field which is more closely related to augmented reality is virtual reality. Both share user interfaces that have a 3D structure and many common issues like depth perception have already been addressed for virtual reality ^[9,12]. However, the combination of physical and virtual world that is specific for augmented reality, is not part of it. Thus guidelines for virtual reality can give valuable input to information representation with augmented reality but are missing this crucial aspect. Augmented virtuality is the umbrella term for systems that integrate parts from the physical world into a virtual environment. Research concerning perception in this area is in many cases transferable to augmented reality and thus not discussed separately ^[22]. Some researchers simply use the broader term mixed reality to avoid this distinction. In general, augmented virtuality has seen less research than augmented reality ^[87]. Also guidelines for augmented reality user interface design have been created, e.g. for augmented reality training applications ^[36,103,104]. Such guidelines usually focus on one specific use case for augmented reality. To the best of the author's knowledge, none of them focuses on information representation in augmented reality to support for procedural tasks.

A taxonomy for annotations used in augmented reality systems was created by Wither et al. ^[105]. While their focus is outdoor augmented reality, the overall structure and the used dimensions are general enough to be applicable to many different use cases. They define an augmented reality annotation to be any virtual object that extends a physical object with additional information. Each annotation consists of two parts, a spatially dependent component that creates the connection to the physical world, and an spatially independent component that is the carrier of the actual information. To classify existing annotations they use the following dimensions:

● Location complexity: how spatially complex the location of an annotation is.

● Location movement: if the location of an annotation moves and if it does, how it moves.

● Semantic relevance: how semantically relevant an annotation for the annotated object is.

● Content complexity: how complex the content of the annotation is.

● Interactivity: whether it is possible to interact with an annotation and how complex the possible interactions are.

● Annotation permanence: how permanently an annotation exist and what the limiting factors are.

They use this taxonomy to classify existing forms of augmented reality annotations from scientific literature. Their approach is very broad, so that special properties of procedural tasks are not considered in particular. Even though it may seem possible at first glance, their approach cannot be applied directly to the problem of modeling information representation to identify sources of unnecessary sensory and cognitive effort. It is limited to annotations and only models how these are integrated into augmented reality environments. What is not included, though, are the reasons that have led to the various ways of integrating annotations. However, their dimensions can provide valuable input to the analysis of the properties of information representation with augmented reality.

Further conceptualization was carried out by Tönnis at al. ^[99] by creating a taxonomy to classify information representations in augmented reality systems. Their goals were to facilitate the analysis of augmented reality applications, better investigate their understanding by humans and identify new research fields. The taxonomy contains the following dimensions:

● Temporality: the permanence of a virtual object's existence. It can either be permanently (continuous) or only for limited amount of time (discrete).

● Dimensionality: the spatial dimensions of an object. It can either be 2D or 3D.

● Viewpoint reference frame: the viewpoint that virtual objects are presented from. It can either be egocentric, egomotion or exocentric.

● Mounting: the mounting point of a virtual object. It can either be human, environment, world or multiple mountings.

● Type of reference: the extend to which a virtual object refers to a physical object or a location. It can either reference visible objects inside the field of view (direct overlays and references), invisible objects inside the field of view (indirect overlays and references) or objects outside the field of view (pure references).

In addition to a theoretical consideration, the authors also explore design challenges that arise for different representations but due to their focus, they do not go into depth with regard to human factors. While their work is an important basis to analyse the characteristics of information representation in augmented reality, their approach cannot be be directly adopted. They focus on the full range of augmented reality applications, so that they cannot specifically address characteristics of procedural task. For example, for procedural tasks their temporality dimension is no longer a dimension but is always discrete. Furthermore, they only consider virtual objects, but not physical objects, which are very important for the execution of procedural tasks ^[100]. In addition, they only roughly differentiate between different gradations of relation to physical space, but it is precisely this gradation that increases the difficulty of representing information.

Vincent et al. ^[101] suggest to model hand held augmented reality as a set of the three categories physical world, representation of the physical world and digital augmentation. These are connected by the two spatial relationships physical world to its representation and representation to its augmentation. The most rigid connection is conformal, which is a precise alignment of categories, while the least rigid one is none when no connection exists anymore. In between them are the relaxed connections. They are no longer strict alignments but a spatial connection is still clearly recognizable for them. This idea to model augmented reality as independent categories which are connected with each other can also be adapted to model information representation.

3.   Modeling information representation

In order to perform procedural tasks, it is necessary for humans to interact with objects ^[38]. While some of these objects have to be manipulated, others provide information. Instructions may be the most obvious example for the second kind, but there are also those which give information on object states, object positions etc. In many cases objects that must be manipulated also provide information on themselves. An example is a bolt which is mounted at some place. Simply by being visible it indicates its mounting position or what kind of wrench is needed to unbolt it. It is particularly important that these objects and their relations can be easily seen without interference. Through augmented reality further objects with more information are integrated into the physical world. An example is a label that describes how to remove the bolt from the previous example. For this kind of information providing objects a conceptual framework is developed that defines a common vocabulary to describe them. allows to classify them and facilitates analyzing which characteristics they have. This later helps to derive challenges that occur when representing information to support performing procedural tasks.

In the context of this framework the previously described objects are called information objects because they convey the relevant information for completing a task. While some information objects are physical, most parts of the physical world do not provide relevant information for a specific task. Examples are a coffee mug placed next to the work space or parts of a machine that do not have any influence on an ongoing repair. These parts form what is called the environment. A combination of environment and information objects is called a scene. Besides being bound to a spatial position in the scene, an information object can also be connected to other information objects. An example is a label which contains further information on a physical object. The other information objects, points or areas that an information object is bound or connected to, are called its anchors. A user has a viewpoint from which she or he looks at a scene and what she or he sees is called a combined view, which is usually limited to a view frustum.

3.1.   Five classes of information objects

As already becomes clear from the previous simple examples with the bolt and the label, information objects can be quite diverse. A key factor is how tightly coupled an information object is with the physical world. Thus the following categorization is made here based on the gradation of spatial connection that information objects can have:

● Direct physical information objects: information objects from the physical world that a user directly sees without any mediation from a device. Examples are tools, bolts, wires etc.

● Indirect Physical Information Objects: information objects from the physical world that a user perceives in a mediated way, e.g. via the screen of a hand held device. Examples are infrared images of a physical object, landmarks that are needed to correlate a video image with the physical world etc. Simple video reproductions of direct physical information objects are included here.

● Spatial virtual information objects: information objects that are virtual and can be positioned in the scene such that a meaningful spatial context is created. This means that their position and size must give additional information to a user beyond the content of the information object and the connections to its anchors. Examples are 3D models of construction parts, virtual wires overlaid on walls, animation of tools to show their use etc.

● Spatially referenced virtual information objects: information objects that are virtual and are only anchored in the physical world to a point, an area or another information object. Their relation to the physical world does not allow to create meaningful spatial context. Examples are labels, POI markers etc.

● Detached virtual information objects: information objects that do not have a spatial connection. Examples are general descriptions of a work step, progress indicators etc. While strictly speaking they are not even part of augmented reality, they have to be considered here because they also convey part of the information need by a user while completing a procedural task.

This classification is of course only possible within a usage context. In one case an information object may be classified as a spatial virtual information object while in another case it may be classified as a spatially referenced virtual information object. The difference is that in the first case the pose and size of the virtual object would give information to a user while this is not the case in the second. Coming back to the the example with the bolt, it could mean the in the first case the bolt had to be inserted somewhere and displaying the bolt at this position would actually give a user more information than just simply displaying it at an arbitrary position. In the second case it might just be a generic 3D icon which could be displayed at any position as long as the connection to its anchor is visualized. In a third case it may not even be an information object because it is of no relevance for the respective task. This breakdown of the augmented workspace into objects obviously requires that they are known and that a scene is not just a combination of an environment with a not further divided augmentation. Necessary documentation models have already been described for augmented reality that can provide the needed granularity ^[57,69]. However, this does not mean that such objects have to exist as high quality 3D models. Instead it is important to be able to address them individually.

Indirect physical information objects, spatial virtual information objects and spatially referenced virtual information objects can have one or more anchors. In case of indirect physical information objects, this is always the part of the environment or the direct physical information object of which they are a reproductions. For the other two classes of information objects these can be points, areas or other information objects. While the anchors define where objects are connected with, they are not necessarily the actual positions at which they are placed. Wither et al. ^[105] define an augmented reality annotation as having a spatially dependent and a spatially independent component. Their spatially dependent component is a special case of an anchor where a spatial or spatially registered virtual information object has one anchor that is a physical object.

The classification of information objects by spatial relation shares similarities with the dimension location complexity by Wither et al. ^[105]. However, there are several major differences. While physical objects are also considered here, this is not the case in their taxonomy. In regard to virtual objects, the two schemata classify different, albeit very similar, properties of objects. While Wither et al. classify the strength of spatial embedding chosen for the inclusion of an object, here the extent to which an object is a carrier of spatial information is classified. Finally, the differentiation of the individual classes from each other is more distinct here.

A distinction from the dimension dimensionality of from Tönnis et al. ^[99] is very similar. The two classes spatial virtual information objects and spatially referenced virtual information object seem to be identical to their classes 3D and 2D at first glance. However, while a 3D icon for a point of interest is in the class 3D, it is missing any meaningful spatial context in the physical world and thus would be classified as spatially referenced virtual information object here. A virtual model of a bolt instead can have this meaningful spatial context because it can have exactly one size and one position where it fits a specific nut like a physical bolt would.

The way information objects are divided into multiple classes is similar to the breakdown of augmented reality user interface into the three categories physical world, its reproduction and a virtual augmentation by Vincent et al. ^[101]. However, there are several significant differences. First, they do not consider individual objects as carrier of information but tread their categories as not further subdivided. They are also missing a further breakdown of the augmentation into multiple classes of objects which have different properties concerning their spatial relation. Their category representation of the physical world might seem equivalent to the class indirect physical information objects but they do not consider any modified reproduction of physical objects and instead regard this as part of the augmentation.

3.2.   Example classifications

To illustrate what information objects are and how they are classified, three example images from scientific publications are included here. Each of them is a snapshot from an augmented reality system for which the information objects are identified and their classification is explained as far as this is possible based on the given information.

In Figure 1a an instruction can be seen, which tells a user to turn a part of a combustion chamber from a jet engine towards the right until it reaches a correct position. Without seeing the part that must be moved, the whole work step is impossible to complete, which makes it an information object. In this case optical see-through glasses are used and the chamber that can be seen here is not a video reproduction on a screen but the actual object. Thus it is a direct physical information object. The position of the arrow has a specific meaning and it can only be moved in a very constrained way without loosing information conveyed by its spatial context. So it is a spatial virtual information object. The blue and green labels refer to relevant parts of the combustion chamber. However, their actual positions do not matter but only where they refer to. Thus they are spatially referenced virtual information objects. The descriptive text does not have a spatial connection to anywhere and is a detached virtual information object.

Figure 1.  Information objects used in augmented reality systems to support procedural tasks.

DownLoad: Full-Size Img PowerPoint

Figure 1b depicts two different visualizations and explanations for the same work step to remove a sensor. First the upper picture is analyzed. The 3D model of the sensor can only have one position and one size where it can be placed in a meaningful context. Otherwise it would not match the mounting position of an actual sensor. Thus this model is a spatial virtual information object. However, from the description of the work step it is known that a physical sensor is mounted as well and is just covered by its virtual counterpart. Thus it must be included in the list of information objects as a direct physical information object. For the arrow the same argument as for the arrow in Figure 1a can be made and it is classified as a spatial virtual information object. The information on the right side is general information on the work step without a spatial connection and is classified as a detached virtual information object. The lower image, however, is more complicated because there is only a diffuse circle that highlights the sensor. But is this circle even an information object on its own? Instead one can argue that it becomes part of a video reproduction of the sensor, which then (including the ring) is classified as one indirect physical information object.

Figure 1c shows an augmented reality system that supports car maintenance and is currently at a work step which requires to open the bonnet. The maybe most noticeable part is that the car's bonnet is colored green in the augmented reality presentation. Thus the physical bonnet can be classified as a direct physical information object while the virtual presentation is an indirect physical information object. The 3D model of the tool needed for this work step is included in a way that shows where it must be placed for this task. Because this placement already gives information to a user, it is classified as a spatial virtual information object. All other virtual elements, as can be judged from the publication, are detached virtual information objects.

3.3.   Characteristics of information objects

To be able to derive representation challenges from this model, it is necessary to explore specific characteristics of information objects. These will later be used to identify sources of unnecessary sensory or cognitive effort which can be caused by inapt information representation. Four relevant characteristics of information objects are described in the following:

Spatial Relation: Information objects are spatially related to and located in the physical world. For direct physical information objects this is trivial. Indirect physical information objects are (altered) reproductions of physical objects and are directly related to their position. For spatial virtual information objects or spatially referenced virtual information objects this characteristic directly follows from the definition by Azuma ^[4] because it requires that objects are registered in 3D and that real, i.e. physical, and virtual objects are combined into one space. Tönnis et al. ^[99] describe this with two dimensions in their taxonomy. One is mounting which is used to differentiate where objects are attached to and together with which other objects they move. The other is type of reference, which is used to differentiate from what perspective the objects are seen. In the context of information linking Müller and Dauenhauer ^[67] described two classes of reference systems that objects can be located in. Those are world or spectator coordinate systems (WCS resp. SCS) which may be independently applied to the position or the orientation of objects.

Connectedness: Information objects are connected with each other. While all all of them are at least spatially connected, there are also semantic connections between information objects from the same as well als different classes ^[67,105]. The reason for this is that objects in the different classes often represent various types of information concerning the same matter. This can be interpreted as annotations to places and objects from the physical world ^[105]. Indirect physical information objects, for example, can be altered reproductions of direct physical information objects on some sort of display. Another example are labels, a type of spatially referenced virtual information objects, which annotate objects in the physical world, a type of direct physical information objects. Yet another example are CAD models that are included to display details of a machine and thus have physical counterparts. These examples all have in common that augmented reality is used to add more information to existing objects or places, which in this framework are modeled as anchors of information objects. Detached virtual information objects do not have this characteristic, because they have no meaningful spatial connection to the physical world. When they are included as part of a scene, their position does not convey any information relevant for a task nor do they provide any information on an anchor. Connectedness is not the same as the dimension mounting from Tönnis et al. ^[99]. It describes a semantic connection that can be expressed by a spatial connection, while mounting describes the reference that system an object is situated in. Wither et al. ^[105] showed that the here discussed connections are especially relevant when supporting maintenance, inspection, construction and assembly tasks with augmented reality.

Discrete Change: Information objects are subject to abrupt and discrete changes over time. The main reason is that procedural tasks, especially in the case of repairs ^[96], are divided into discrete steps. With each step the relevant information objects can and in many cases will change. This leads to a temporal sequencing concerning the presence of information objects. But even within a work step information objects may change, e.g. due some event that makes the display of new information objects necessary. Tönnis at al. ^[99] classify this behavior as discrete with their dimension temporality, i.e. information that is only occasionally part of the augmentation. Wither et al. ^[105] classify this behavior as temporal in their dimension annotation permanence and were able to find it especially in augmented reality applications that support construction and assembly tasks. This characteristic does not imply that there is no information object which is part of multiple steps or even all steps but instead that the overall set of information objects will change during the task. Direct physical information objects do not have this characteristic because physical objects usually do not just appear, disappear or rapidly change their appearance. a Manipulability: The decreasing spatial relation of information objects across the five classes is accompanied by a gain in flexibility to manipulate them by software. While direct physical information objects cannot be changed by software, this is already possible to a limited extent with indirect physical information objects, for example by displaying a changed representation of a direct physical information object on a screen. Spatial virtual information objects can in principle be manipulated as desired, e.g. moved, enlarged or deformed. However, this is accompanied by a loss of information on spatial context. In the next class, spatially referenced virtual information objects, the only relevant spatial information are the references to one or more anchors that must be preserved. Detached virtual information objects do not have spatial connection, so that these objects can be manipulated without loss of spatial information.

3.4.   Characteristics of a combined view

Besides the previously described characteristics of information objects, the characteristics of their combination into a combined view are also relevant. Three of them are described in the following:

Combination: As already shortly explained, combined view is the view that a user has of information objects and environment from the viewpoint. It stems from the basic definition of augmented reality by Azuma ^[4]. For many application scenarios in the field of procedural task support like repair or maintenance of machinery, the viewpoint cannot or can only partly be controlled through software. Instead it depends on a user's movements ^[4]. Thus it must be regarded as neither exactly predictable nor controllable. The same can be true for the environment, based on the use case, and with a decreasing degree, based on their class, for information objects. Depending on the environment and the information objects a slight change, e.g. caused by a change of the viewpoint or a change of one of the information objects, can cause a very different perception. An example is the perception of color which significantly changes for different background colors ^[62]. Unfortunate combinations may even render an augmented reality system unusable. Another consequence is that the intensification of one information object, e.g. due to increased size or luminance, may easily diminish other. Combination is limited to a view frustum which theoretically may span the whole field of view but is usually much smaller.

Fluctuation: The combination of the information objects, environment and view point into a combined view as seen by a user is influenced by non controllable fluctuations ^[96]. These are mostly caused by problems to precisely and quickly align reference systems with each other, e.g. those of a physical object and its virtual counterpart. A typical example are tracking errors, that cause spatially referenced or spatial virtual objects not to be aligned with their intended position ^[82]. However, the problem is not limited to this single cause, but extends to all variables influencing the mapping like bad calibration of HMDs, inaccurate 3D models, low frame rates etc. Vincent et al. ^[101] model this behavior by defining two spatial mappings that link the physical world with its representation and the representation of the physical world with the augmentation.

Reference Systems: When combining information objects into one scene, they can be positioned and oriented in different reference systems which can move independently from each other. These may be e.g. the screen of a handheld device, a user's head or a reference system attached to a physical object. This binding can be done independently from any anchor. A spatial virtual information object could be attached to a user's head, while the anchor is a direct physical information object located elsewhere. Also orientation and positioning do not have to be identical. Billboards are positioned at a fixed point in the physical world but oriented towards the user. Two basic kinds of reference systems can be can be distinguished: world coordinate systems and spectator coordinate systems ^[67]. The first kind contains all the reference systems that are independent from a user while the second kind contains reference systems that are bound to a user. A special case is direct spatial mapping ^[67]. It means that a spatial virtual information object is WCS positioned / WCS oriented and is positioned and sized such that a meaningful spatial context is created. An example for this is a virtual bolt which is positioned such that it matches the position and size of a hidden physical bolt.

4.   Representation challenges

One goal when designing augmented reality applications for procedural task support must be to minimize a user's sensory and cognitive effort to perceive relevant information. Otherwise the user will spend unnecessary time with this instead of the actual task and will experience increased cognitive load as well as to faster fatigue ^[20]. Sensory and cognitive effort are not further distinguished here but treated as one. Reasons are that they are very difficult to differentiate and are often closely interwoven, especially in the case of visual perception ^[8].

Based on the previously identified characteristics of the classes of information object and the characteristics of combination, five distinct representation challenges can be identified. These describe possible sources of unnecessary sensory and cognitive effort that need to be addressed when designing augmented reality user interfaces to support procedural tasks. For each of these challenges it is explained what the challenge is, how it can be derived from the previously explained characteristics and a list of literature with further information and possible solutions is given. While most of the given literature is specifically on augmented reality or a related field, some references offer fundamental insights from another discipline, so that these are also listed here.

Many, if not all of the here presented challenges could also be addressed by providing adequate user interaction techniques, e.g. by enabling a user to manipulate virtual objects as long as she or he is satisfied with the combined view. However, this would create additional effort which is not directed at solving an actual task. Thus the goal must be to provide an information representation which minimizes this kind of interaction.

4.1.   Clarity challenge

Regardless of the current combination of information objects and environment, information objects must be individually recognizable as good as possible. Thus the challenge is to present them in a clear and concise way that does only afford minimal cognitive and sensory effort. Examples are a text which must be easily readable or a model of a mechanical component that must be effortlessly identifiable as such. The particular challenge is that every change of viewpoint, environment or information objects changes the perception of individual information objects. One example is that a color change from one object also influences how colors of other objects are perceived ^[29]. A user's perception of this is difficult or even impossible to control. This challenge is only concerned with information objects that are visible. Occluded objects or objects outside of the view frustum are addressed in other challenges.

Figure 2 gives examples of effects that affect the clarity challenge. Figure 2a shows how objects can become very difficult to recognize due to color distortion. This effect becomes even more intense in front of colored backgrounds. Figure 2b shows how the interaction of background and text drawing style changes the legibility. To a certain degree both effects can be compensated by humans but the price for this is an increase of sensory and cognitive effort.

Figure 2.  Examples of effects that affect the clarity challenge.

DownLoad: Full-Size Img PowerPoint

This challenge is a consequence of a combination of two characteristics. The first characteristic spatial relation implies that information objects may not be in an optimal position for viewing them because of their position or orientation. They may get too small due to distance or the viewing angle may become disadvantageous. The second characteristic combination increases this problem because the background, which consists of other information objects or the environment, behind the information object may distort it. The result is that a user must spend sensory and cognitive resources to recognize information objects before he or she can understand them.

The clarity challenge can be further divided intro three aspects. Those are visual acuity, i.e. how well the structures of an object are visible, color perception, i.e. how color is perceived in combined views, and text legibility, i.e. how well text can be read from virtual objects. For references see Table 1.

Table 1.  References for the clarity challenge.

Aspect	References
Visual acuity	DiVerdi, Hollerer and Schreyer [21]; Livingston, Ai and Decker [58]; Livingston, Barrow and Sibley [59]; Livingston, Gabbard, Swan, Sibley and Barrow [62]; Livingston [63]
Color perception	Darken, Sullivan and Lennerton [24]; Fairchild [29]; Flatla and Gutwin [33]; Gabbard, Swan, Zedlitz and Winchester [42]; Itoh, Dzitsiuk, Amano and Klinker [48]; Livingston, Gabbard, Swan, Sibley and Barrow [62]; Livingston [63]; Smallman and Boynton[84]
Text legibility	Debernardis, Fiorentino, Gattullo, Monno and Uva [19]; Fiorentino, Debernardis, Uva and Monno [31]; Gabbard, Swan, Hix, Schulman, Lucas and Gupta [39]; Gabbard, Swan and Hix [40]; Gabbard, Swan, Hix, Kim and Fitch [41]; Leykin and Tuceryan [65]

---

 | Show Table

DownLoad: CSV

4.2.   Consistency challenge

While the physical world is inherently consistent, this is not automatically true for augmented reality. Examples are a faulty tracking, that causes virtual objects to be incorrectly placed ^[82], insufficient visualizations of depth cues ^[56] or a slow system that causes latency ^[26]. However, to keep the user from unnecessary sensory and cognitive effort, she or he must perceive a consistent state at all times, i.e. no contradictory information should be displayed. For example, depth must be visualized, a delay may not exceed a certain time span etc. Other than in virtual environments or traditional user interfaces, this is more complex for augmented reality because, as described, not all classes of information objects are fully controllable by software. Instead, an augmented reality system must react to changes of the physical world, which are immediate and may happen abruptly.

Figure 3 contains examples of effects that affect the consistency challenge. The green square in Figure 3a is positioned behind the wall but there are no visual cues to support this. Only when moving the viewpoint, a user will experience this inconsistency because of its relative movement and the perceived position on the wall. The shadow in Figure 3b integrates the right cup better into the scene and makes it look less like a foreign body. This causes the scene to be more consistent and be less distracting for a user.

Figure 3.  Examples of effects that affect the consistency challenge.

DownLoad: Full-Size Img PowerPoint

This challenge is caused by a combination of multiple characteristics. Combination implies that all information objects and the environment are combined into one view and cannot be seen separately. When fluctuation gets too strong, e.g. because movements of virtual information objects are delayed compared to physical information objects, a user perceives an inconsistent state. The spatial relation of information objects complicates this because depth cues may as well be inconsistent. This leads to effects like virtual objects that seem to float in front of physical objects while they are actually positioned be behind then. Also manipulability restricts the way a consistent state can be enforced because not all information objects are fully controllable and an application must react to changes of those which are less controllable to recover a consistent state.

The consistency challenge can be further divided intro three aspects. Those are correctly regarded depth perception and correctly used depth visualization, visual integration of virtual information objects into the scene and timing factors like latency. For references see Table 2.

Table 2.  References for the consistency challenge.

Aspect	References
Depth perception and visualization	Avery, Sandor and Thomas [3]; Bane and Höllerer [6]; Bichlmeier and Navab [10]; Cutting [17]; Cutting and Vishton [18]; Drascic and Milgram [22]; Elmqvist, Assarsson and Tsigas [25]; Franklin [35]; Furmanski, Azuma and Daily [28]; Jerome and Witmer [53]; Jones, Swan, Singh, Kolstad and Ellis[52]; Kalkofen, Mendez and Schmalstieg [54]; Lerotic, Chung, Mylonas and Yang [60]; Livingston, Ai and Decker [58]; Livingston, Garrett, Hirota, Whitton, Pisano, Fuchs and others [61]; Livingston, Swan, Gabbard, Höllerer, Hix, Julier, Baillot and Brown [64]; Milgram, Zhai, Drascic and Grodski [72]; Peterson, Axholt, Cooperand Ellis [75]; Robertson, MacIntyre and Walker [82]; Sandor, Cunningham, Dey and Mattila [86]; Sielhorst, Bichlmeier, Heining and Navab [85]; Sugano, Kato and Tachibana [94]; Swan, Jones, Kolstad, Livingston and Smallman [92]
Visual integration	Feng [32]; Fischer, Bartz and Straber [30]; Franklin [35]; Haller [43]; Kalkofen, Mendez and Schmalstieg [54]; MacIntyre and Coelho [66]; Robertson, MacIntyre and Walker [82]; Sugano, Kato and Tachibana [94]; [50]; Vincent, Nigay and Kurata [101]; Yeh and Wickens [106]
Timing and latency	Ellis, Breant, Manges, Jacoby and Adelstein [26]; Ellis, Wolfram, and Adelstein [27]; Franklin [35]; Hirsh and Sherrick [47]; Holloway [46]; Ng, Lepinski, Wigdor, Sanders and Dietz [73]; Pöppel [80]

---

 | Show Table

DownLoad: CSV

4.3.   Visibility challenge

To optimally support a user in acquiring and understanding relevant information for a work step, all information objects must be visible to her or him. If some of them are covered by others, it will likely take extra effort to cope with this, e.g. by moving to better view point. Covering important information objects may even become a safety hazard when dangerous parts like sharp edges or moving parts cannot be seen. Because visibility is caused by an interaction of virtual information objects, which can be controlled via a program, and and physical information objects, which cannot be controlled, the degree to which this is possible depends on the individual usage situation. This challenge is concerned with information objects inside the view frustum, not those that are invisible because they are outside of it.

Figure 4 shows an example for the visibility challenge where a direct physical information object (the physical building block) is covered by a spatial virtual information object (the virtual building block). This makes it difficult to interact with it because necessary visual feedback is missing.

Figure 4.  The virtual building block makes the physical block nearly invisible and hard to interact with (image taken from ^[89]).

DownLoad: Full-Size Img PowerPoint

The visibility challenge is caused by a combination of three characteristics. The physical as well the virtual objects are organized in a spatial structure as established by spatial relation. Since virtual objects are not subject to any physical laws, it is even possible that they are located at the same location as other objects. When they are combined as described by combination, they may overlap from a user's perspective. Manipulability is also relevant in this case because it limits how the challenge can be solved by moving or changing objects. Direct physical information objects cannot be manipulated, e.g. moved, by an augmented reality system. For indirect physical objects as well as all virtual objects this is may come with a loss of spatial context depending on their spatial connection.

In the most trivial case, the visibility challenge can be solved by moving the viewpoint such that information objects do not overlap anymore from a user's perspective. However, this causes additional effort by forcing a user to actively interact or may not always feasible. Possible reasons are that movement of the viewpoint causes other objects to overlap, that a movement is not possible or that objects overlap from too many viewpoints.

The visibility challenge can be further divided into two aspects. Those are resolving overlapping information objects spatially, i.e. moving them to other positions, and superposition of information objects with partial transparency. For references see Table 3.

Table 3.  References for the visibility challenge.

Aspect	References
Spatial resolution	Azuma and Furmanski [2]; Bell, Feiner and Höllerer [5]; Grasset, Langlotz, Kalkofen, Tatzgern and Schmalstieg [37]; Leykin and Tuceryan [65]; Peterson, Axholt, Cooper and Ellis [75]; Peterson, Axholt and Ellis [76]; Peterson, Axholt and Ellis [77] Rosten, Reitmayr and Drummond [83]; Shibata, Nakamoto, Sasaki, Kimura and Tamura [95]; Thanedar and Höllerer [97]
Resolution by transparen-cy	Elmqvist, Assarsson, Tsigas [25]; Johnson, Edwards and Hawkes [49]; Kalkofen, Mendez and Schmalstieg [54]; King, Piekarski and Thomas[55]; Livingston, Swan, Gabbard, Höllerer, Hix, Julier, Baillot and Brown [64]; Robertson, MacIntyre and Walker[82]; Sandor, Cunningham, Dey and Mattila [86];

---

 | Show Table

DownLoad: CSV

4.4.   Orientability challenge

At any time a user must be supported in knowing at which position she or he is in a scene. When a user is wearing a head mounted display the surroundings around her or him might be hardly visible, especially when the display is cluttered with virtual information objects. For hand held devices clutter can make it hard to relate the display content with the physical world. This gets even harder when the display shows an altered version of the surroundings, e.g. an infrared image. Also the user must be supported in knowing where relevant information objects are. Even when the visibility challenge is adequately addressed, this may still be problem for information objects which are inside the view frustum, when they are not sufficiently salient. Information objects outside of the view frustum are not visible and may not be known to the user. They may also have disappeared or moved or new ones may have been added. Getting an overview of those and maintaining it, causes sensory and cognitive effort for a user. A factor which complicates this challenge is that users tend to develop tunnel vision when using augmented reality systems ^[98].

As an example, Figure 5a shows how arrows can be used to point at objects outside the field of view and support a user to find them. This allows orientating her- or himself more quickly and to minimize the effort of searching for objects. In Figure 5b an example can be seen of how information objects are placed incoherently over time. A user will have to search for the objects again each time a relevant position change occurs.

Figure 5.  Examples for the consistency challenge.

DownLoad: Full-Size Img PowerPoint

The orientation challenge is caused by a combination of four characteristics. As described by spatial relation, physical as well as virtual objects are organized in a spatial structure. A user has to obtain orientation, where in this structure she or he is and which information object is at what place. As described by discrete change, an abrupt change in the information objects can occur which forces a user to reorient every time this happens. The characteristic reference systems is also relevant, because objects may move independently from each other in different reference systems which causes the spatial arrangement to change while the set of information objects stays the same. An augmentation, as described by the characteristic combination, is limited in its extend to the view frustum and information objects, except for direct spatial information objects, outside of it cannot be seen. This causes effort for a user to remember or (re)discover them.

The orientability challenge can be further divded intro three aspects. Those are orientability of a user in her or his spatial surrounding, object discovery outside or inside the view frustum and supporting a user to keep track of known objects and their positions. For references see Table 4.

Table 4.  References for the orientability challenge.

Aspect	References
Orientability	Julier, Lanzagorta, Baillot, Rosenblum, Feiner, Höllerer and Sestito [51]; Kalkofen, Mendez and Schmalstieg [54]; Kruijff, Swan and Feiner [56]
Object discovery	Bane and Höllerer [6]; Bell, Höllerer and Feiner [7]; Bork, Schnelzer, Eck and Navab[13]; Biocca, Owen, Tang and Bohil [11]; Biocca, Tang, Owen and Xiao [14]; Feiner, MacIntyre, Höllerer and Webster [34]; Henderson and Feiner [45]; Henderson and Feiner[44]; Sandor, Dey, Cunningham, Barbier, Eck, Urquhart, Marner, Jarvis, and Rhee [88]; Schinke, Henze and Boll [91]; Schwerdtfeger, Klinker [93]
Keeping track	Azuma Furmanski [2]; Grasset, Langlotz, Kalkofen, Tatzgern and Schmalstieg [37]; Madsen, Tatzqern, Madsen, Schmalstieg and Kalkofen [70]; Neumann and Majoros [74]; Polys, Bowman, and North [78]

---

 | Show Table

DownLoad: CSV

4.5.   Information linking challenge

The ability to spatially register information makes augmented reality a very effective medium to show objects as connected by spatial assignment. However, it may not always be appropriate to position every information object at its respective anchor, e.g. because objects would overlap and thus reduce their visibility. Sometimes information objects may also have multiple anchors which makes it impossible to show every connection by spatial assignment. In any case, it must be possible for a user to recognize connections with minimal sensory and cognitive effort. Insufficient tracking precision and tracking errors further complicate this challenge because a user must cope with these disturbances. Figure 6 contains examples for visualizations that show connections of information objects to their anchors.

Figure 6.  Possible ways to solve the information linking challenge and display links of information objects to their anchors.

DownLoad: Full-Size Img PowerPoint

The challenge information linking arises from a combination of three characteristics. Connectedness is the basis because connections of information objects with their anchors must be made clear to the user. When objects are moved away from their anchor or, in case of spatial virtual information objects, from a meaningful spatial context to another position, e.g. as part of handling the visibility or clarity challenge, it may no longer be possible to recognize these connections. Spatial relation implies that these connections have to be visualized in such a way that they are unambiguously recognizable in a 3D space. The fluctuation characteristic can increase difficulty to recognize the connections because of effects like latency, low frame rates, insufficient tracking etc.

This challenge can be further divided into three aspects. Those are linking information objects by the used reference system, linking information objects by different visual connections and linking information objects by graphical context. For references see Table 5.

Table 5.  References for the information linking challenge.

Aspect	References
Reference system	Chen, Pyla and Bowman [16]; Coelho, MacIntyre and Julier [15]; Madsen, Tatzqern, Madsen, Schmalstieg and Kalkofen [70]; Müller and Dauenhauer [67]; Polys, Bowman and North [78]; Polys, Kim and Bowman[79]; Robertson, MacIntyre and Walker [81]; Robertson, MacIntyre and Walker [82]
Visual connection	Dauenhauer and Müller[23]; Grasset, Langlotz, Kalkofen, Tatzgern and Schmalstieg [37]; Müller and Dauenhauer [67]
Graphical context	Robertson, MacIntyre and Walker [82]; Robertson, MacIntyre and Walker [81]

---

 | Show Table

DownLoad: CSV

4.6.   Conflicting resolution measures

Measures to overcome the previously outlined challenges are not always complementary but instead may even be contradictory. Take for example multiple virtual information objects which all share a common anchor. When all of them are placed at their anchors' positions to support information linking ^[67], at least some of them cannot be fully visible. To solve the visibility challenge, however, this must be achieved. One possible solution is to make them partly transparent. As a result, the clarity challenge cannot be completely solved because the information objects interfere with each other by overlapping or shining through ^[64]. So an alternative is to move them such that there no longer any overlap ^[5]. In this case, the information linking challenge is not solved completely because any distance between an information object and its anchor causes additional effort for a user to recognize the connection ^[67]. This simple example makes it apparent that it is not possible to generally solve the challenges but instead for each use case it has to be individually weighted which measures should be used.

5.   Conclusion

Representing information with augmented reality in a way that optimally supports execution of procedural tasks like repair or maintenance remains challenging. Unsuitable representation leads to increased sensory and cognitive effort for users and distracts from the task at hand. Here a contribution is made by developing a framework for modeling information representation for such use cases and identifying representation challenges that need to be addressed in order to minimize sensory and cognitive effort. As a basis five distinct classes of information objects are established which are distinguished by their gradation of spatial relation to the physical world. For them and their combination into one view specific characteristics are identified. When those are combined, they form possible sources of unnecessary sensory or cognitive effort. Thus the representation challenges clarity, consistency, visibility, orientability and information linking are formulated, which must be taken into account when creating augmented reality systems to support users in executing procedural tasks. They all have in common that not properly solving them will cause a user to spend additional effort to see and understand presented information. In addition, literature that provides indications on how to solve the challenges is collected. However, it becomes clear that not all challenges can be perfectly solved at the same time, since some solutions are contradictory. Examples are visibility and information linking for information objects that share the same anchor. For the visibility challenge, objects should be moved away from each other in order to avoid overlapping while for the information linking challenge, these objects should be left as close as possible to the position of their anchor.

Possible next steps can go into three directions. One direction is to identify existing gaps for the challenges. Visibility, for example, is still missing a technique to handle multiple spatial virtual information objects which are located at the same position. A further example is the visual integration aspect from the consistency challenge. Even though different factors are known, it is not clear how these interact and how much integration is actually needed. Another direction is the identification of new representation challenges which have not been addressed here but also fit into this model. It would be interesting to extend the presented work, in order to give hints on how to handle further sources of unnecessary sensory and cognitive effort. Restricting the type of application, e. g. to only one type of device, would probably allow to be more specific in this respect. A third direction is the transfer of this framework to other application areas of augmented reality. For example, in an augmented reality training system cognitive effort does not always need to be minimized but instead can be part of the learning process.

Conflict of interest

The author declares no conflict of interest.