Figure 6-1. Illustration Process with User-Navigation.Shaded area shows the modifications to the architecture.
From this point on the illustrations we will describe truly differ from those generated by human illustrators. In the past chapters we described how our methodology is applied to generate static presentations. While it is useful to generate these illustrations quickly and efficiently, we should not ignore the capabilities afforded us by current computer technology. The same process that enables our system to generate static illustrations can be used to generate and present dynamic (or changing) illustrations.
We define an interactive intent-based illustration as an illustration that remains bound to (possibly changing) communicative intent and which allows access to any element in the multi-level representation of the illustration. The multi-level representation consists of the communicative intent that specified the illustration initially, its designs, its style strategies, its styles, and the data held in the illustration itself. Thus, interaction falls into two categories: modification and analysis. The system can interact with other modules that modify the values that describe the world, the user’s view of the world, as well as the very intent of the illustration. The system can also interact with modules that request different types of analysis of the illustration and the system’s view of the world. A fully interactive system enables modification at any level of the illustration, illustration process, and world, and could analyze the relationship between the various objects to the same degree.
As we described in Chapter 3, our architecture is based on a multi-level representation of an illustration and the dynamic relationship of the three components that manipulate it. This multi-level representation describes not only the results of the decisions made during the illustration process, but also describes the decisions themselves, and how illustration-objects correspond to the objects in the world that is being depicted. The evaluators percolate up to detect goal conflicts to select alternative plans, while the methods percolate down to activate new subgoals. The relationships between the various components need not terminate once a satisfactory solution is achieved. Instead, they may be used to evaluate, analyze, and modify the illustration continuously.
In this thesis we address four different types of interaction: user-navigation, changing worlds, changing goals, and self-evaluation. The user can navigate in the illustrated world by changing the values that define the view specification. Thus the user interacts with the system. The world itself can change; that is, other modules may report new values for the position and states of objects in the world. Thus, the world interacts with the system. In both cases, the system continuously evaluates and adjusts the visual representation of the world so that the communicative intent is satisfied despite these changes. Other modules can modify the communicative intent, by modifying current goals as well as introducing new ones. Thus, other modules interact with the system, causing the illustrations to change. The system can also be requested by external modules to evaluate its illustrations using the same primitives and rules that define the visual language as well as the criteria used in the illustration process. For example, other modules can request the system to evaluate goals that are not associated with the illustration. In all cases, the interaction is supported by 1) enabling access to previously inaccessible parts of the system and by 2) introducing specialized meta-rules.
We do not wish to limit the user to viewing a static presentation. The user may want to see an object from a different angle, or look at other objects in the scene. The user may simply not wish to be limited to one presentation, when the hardware clearly can support many. Enabling the user to navigate may in fact enhance the user’s understanding of the depicted world. For example, the spatial relationship between the different objects may be more clearly understood if seen while navigating, or the shape of an object may be more clearly understood if seen from different views. Unfortunately, user-navigation may just as easily confuse the user as he or she navigates and can longer find objects he or she wants to see. We will show how IBIS redesigns the illustration during the course of user-navigation to avoid confusion.
User-navigation in an intent-based illustrated 3D world differs sharply from conventional navigation in 3D graphics systems. The illustrated 3D environment presented to the user as he or she navigates remains continuously bound to the same communicative intent. As the user changes the values that define the view specification, the system is directed to detect goal conflicts associated with the communicative intent and, when it determines it is necessary, to modify the 3D illustrated environment accordingly. As shown in Figure 6-1, a module to change the view specification is added to the system. The view specification is thereby constrained by the values provided by the user. Meta-rules for user navigation are used to guide the system to detect goal conflicts and handle them in a fashion that favors speed over accuracy.
Figure 6-1. Illustration Process with User-Navigation.Meta-rules are used during user-navigation to activate evaluations to detect when goals are violated by the change in the view specification These meta-rules are used to increase speed and avoid unnecessary changes to the illustration. Figure 6-2 shows the meta-rules for user-navigation. In the following section we describe how all these meta-rules are applied using two examples.
1. do not apply methods that change the view specification
2. reorder style strategy rules so that:
all visibility goals are handled first
all recognizability goals are handled second
all other style strategy goals are handled next
3. if the view specification is changed
then reevaluate asserted style strategy goals
4. if a communicative-goal cannot be satisfied
then generate an inset illustration for it
5. if an object that must be visible is obscured
then apply the visibility method by-cutaway-view
6. if a style strategy goal is evaluated to fail
then try alternative design evaluators for the
communicative goal for which the style strategy was applied
7. if all alternative design-evaluators for a
communicative goal have been tried
(by applying meta-rule 6) and none succeeded
then let the drafter apply alternative methods for
the failed style strategy goal
8. if an inset was generated (by applying rule 4)
but the communicative goal is evaluated as
successful in the main illustration
then remove the inset
9. if an object that must be visible is visible
and it is the subject of a cutaway view
then remove the cutaway view
10. use smooth transitions
In Chapter 3 we showed that the order in which rules are applied by the illustrators and drafters is determined by an ordered list. Meta-rules 1 through 5 in Figure 5-2 affect both the order and content of these lists, and thus determine the order in which both components will apply methods and evaluators during user-navigation. Meta-rule 1 states that all methods that alter the view specification should not be attempted, since the view specification is constrained. Note, however, that evaluators that depend on the view specification remain active.
In Chapter 3 we also stated that unless specified otherwise, goals were handled in the order in which they were input. The illustrator assigns the drafter style strategies in the order specified by the design method. Meta-rule 2 specifies the drafter should ignore this ordering and instead handle all visibility and recognizability goals first. These goals are more sensitive to the view specification, thus they are more likely to necessitate changes to satisfy them. In some cases, this ordering minimizes the changes made to the illustration. For example, suppose an object is associated with both highlight and visible goals. During user-navigation, both goals are violated. The visibility goal is handled first, and a cutaway view is generated. The highlighting goal is automatically satisfied because, as we saw in Chapter 4, one of the evaluators for highlighting specifies that if an object is subject to a unique graphical device, then the object is evaluated as highlighted.
Although a module is changing the value of the view specification, a rule is needed to alert the drafter to this fact so that it reevaluates the style strategies and calls on illustration procedures to analyze the current state of the illustration. Meta-rule 3 specifies that whenever the user changes the values that define the view specification, the asserted style strategies goals should be reevaluated.
During user-navigation it would be awkward to change continuously the styles applied to satisfy violated goals. Instead, when it is determined that a communicative goal is simply not satisfied in the active illustration, a composite illustration is generated. Meta-rule 4 specifies that if all methods fail for a communicative goal, then an inset illustration should be generated for that goal.
If an object that must remain visible is obscured by other objects, we know that we will have to resort to one of the style methods that alters the appearance of objects that block it. The cutaway view is favored (it requires less processing than the ghosting technique and introduces fewer interpretative problems as described in Chapter 4). However, the rule can be altered so that ghosting could be used instead. Meta-rule 5 specifies that if an object that must be visible is obscured, then a cutaway-view is to be used.
Meta-rules 6 and 7 determine whether or not a component applies evaluators or methods, as well as which component, the illustrator or drafter, is active.
Although one particular method was applied to accomplish a goal, other methods may be successfully achieved in the same illustration as a side-effect. Suppose that during user-navigation, the method applied is evaluated to fail. To avoid unnecessary changes to the illustration, evaluators associated with other methods for the same goal then should be applied. If one of these methods is deemed successful, then no modification to the illustration is warranted. A style strategy goal is asserted to apply a design method for a particular communicative goal. As we have shown in previous chapters, other design methods may be evaluated as successful so no modifications to the illustration are needed to achieve that communicative goal. Meta-rule 6 specifies that when a style strategy is evaluated as unsuccessful, design evaluators for the same communicative goal for which the style strategy goal was asserted are activated. Thus, if the communicative goal is satisfied using another method, no changes are made to the illustration. In Chapter 3 we described that the illustrator waited for the drafter to return evaluations to resume work. In this special case, the illustrator takes over, while the drafter waits to see if it has to handle the failed style strategy. Once the illustrator determines that no other methods for the violated communicative goal are currently satisfied, the illustrator instructs the drafter to handle the style strategy. Meta-rule 7 specifies that if all design-evaluators have failed, then the drafter should find an alternative method to satisfy the violated style strategy.
If a change was introduced to handle a violated goal, then it is no longer needed when the active illustration is evaluated to satisfy that goal. Meta-rule 8 specifies that if an inset illustration was generated to solve a goal that is no longer violated, then it should be removed. Similarly, meta-rule 9 specifies that if an object is no longer obscured then no cutaway view is necessary.
During user-navigation it is important not to confuse the user with abrupt changes. Transformations made to objects’ appearances should appear gradually. Meta-rule 10 specify visual effects be applied so that the transition is smooth. For example, rather than just apply a cutaway view (or blast a hole in the object), a cutaway appears and disappears with increasing and decreasing levels of transparency. These rules may compromise the accuracy of each frame during user-navigation in favor of an aesthetic. In the case of the cutaway view, the obscured object may not be visible for, at most, a few frames.
In this example we will show how the meta-rules are applied to determine when visibility and recognizability goals are violated during the course of user-navigation. Figure 6-3 shows an illustration designed to show the location of the radio’s MODE dial with highest priority. It has been specified with the communicative intent shown in Figure 6-4. Figure 6-5 shows the design method that was applied to satisfy the location goal.
Figure 6-3. Communicative intent: show location of MODE dial.
User navigation: initial illustration.
(location mode-dial radio highest)
(?goal = location)
?object = function-dial
?context = radio
(threshold = highest)
The method states that in order for the location of an object to be shown in a certain context that object must be included. An illustration-object is generated and populates the illustration-object list. Next, the MODE dial should be recognizable. The constraints stored with the function-dial constrain the view specification. Next, the context object must be included and an illustration object for the radio is generated. The illustration, therefore, consists of two illustration-objects, the MODE dial and radio. The method then specifies that the MODE dial be both visible and highlighted. The MODE dial is highlighted using the methods described in Chapter 3. Additional lights are shone on the MODE dial while the lights shone on all other objects are dimmed. Finally, the radio itself must be recognizable, so the recognizability constraints associated with the radio are applied and a view specification is selected.
(method-1-for-showing-location-with-highest-priority
;; ASSIGNED TO ILLUSTRATOR
(location ?object ?context)
=>
;; ASSIGN TO DRAFTER
(include ?object highest) ;; 1
(recognizable ?object high ;; 2
(include ?context highest) ;; 3
(visible ?object high) ;; 4
(highlight ?object high) ;; 5
(recognizable ?context highest) ;; 6
)
(evaluator-2-that-location-is-shown-with-highest-rating
;; FROM DRAFTER
(evaluated include ?object ?val1:>=highest) ;; 1
(evaluated recognizable ?object ?val2:>=highest) ;; 2
(evaluated include ?context ?val3:>=high) ;; 3
(evaluated visible ?object ?val4:>=high) ;; 4
(evaluated highlight ?object ?val5:>=high) ;; 5
;; RETRIEVE FROM OBJECT REPRESENTATION
(landmark ?context ?landmark)
;; FROM DRAFTER
(evaluated recognizable ?landmark ?val6:>=high) ;; 6
(evaluated visible ?landmark ?val7::>=high) ;; 7
=>
(evaluated location ?object ?context highest)
)
Figure 6-7. Communicative intent: show location of MODE dial. User navigation: user is zooming in.
The system is notified that the user will be navigating. This enables the meta-rules shown in Figure 6-2. The user begins to zoom in. This triggers meta-rule 3 so that style strategies will be reevaluated. Methods for changing the view are deactivated, and rules are reordered by meta-rules 1 and 2. The visibility goals are handled first by the drafter. The MODE dial is evaluated to be visible, but the visibility evaluator for the radio returns failure because the radio is clipped and does not appear in its entirety. Meta-rule 6 specifies that alternative design evaluators be applied. The visibility goal is associated with the location goal, so the illustrator applies other design evaluators for location. Figure 6-6 shows the design-evaluator the illustrator selected. It specifies that a landmark object for the context object should be shown. Figure 6-7 shows the state of the illustration when this evaluator succeeds. The keypad buttons are landmark objects, so the evaluation is successful. The system thus determines that the illustration continues to satisfy the communicative intent and no modifications to the illustration are needed.
The user continues to zoom in. The illustration no longer satisfies the location goal because none of the radio’s landmark objects are visible. Meta-rule 3 is applied and an inset is generated to handle the location goal. As described in Chapter 5, a new illustrator is created and assigned the violated goals with priority and a copy of the original illustration with the view specification unspecified. The illustrator also inherits the illustrative style as well as a complete history of the illustration process used to generate the original illustration. This illustrator satisfies the visibility goal for the radio by applying the original view specification. The appearance of the radio is consistent in the two illustrations; the MODE dial is highlighted in both illustrations. A rule for positioning the inset specifies that the inset be placed near the MODE dial in the original illustration. Figure 6-8 shows the composite illustration with the inset.
Figure 6-8. Communicative intent: show location of MODE dial. User navigation: user is zooming in. Automatically generated inset illustration.
Similarly, when the user begins to zoom out, and the visibility goal succeeds in the original illustration, the inset is removed by applying meta-rule 7. As the user continues to zoom out, the recognizability goal for the MODE dial fails because the dial is no longer legible. An inset illustration is generated, by applying meta-rule 3, since all design methods for the location goal specify that the MODE dial should be recognizable and the recognizability constraints associated with the MODE dial are violated.
Now the user starts to change the angle of the view. The radio’s antenna obscures the MODE dial and the visibility goal fails. Meta-rule 5 specifies that a cutaway view should be generated. The part of the antenna that obscures the MODE dial fade away as specified by meta-rule 10. Figure 6-9 shows the illustration with the cutaway view. The user continues to move, and the antenna no longer obscures the MODE dial. Meta-rule 9 specifies that the cutaway view be removed and the transparent part of the antenna fades in as specified by meta-rule 10.
Figure 6-9. Cutaway mask for the MODE dial during user-navigation.
The second type of interaction we will describe is made possible by the system’s facilities for self-evaluation and query handling. Corresponding to the three levels of representation for illustration, we classify three levels of queries that provide information that can be used by other modules. This information can consist of the results of the same evaluators that are used during the design process, the results of the various methods used, as well as the relationship between illustration-objects and physical objects.
Illustration-queries concern the appearance of the illustration. This information can be used by other modules to refer to both visual and semantic aspects of the illustrations. For example, in COMET, cross-references are generated based on this information [McKeown et al. 92]. Types of cross-referencing range from referring to the top or inset illustration; the object that is near the bottom, the object shown in the cutaway view, the highlighted object, or statements like, “. . . as shown in this illustration, the button pad is in the middle of the radio.” Illustration queries can also report an illustration’s resolution, style, viewing specification, and lighting. This information enables a text generator to generate references such as “this illustration is a bird’s eye view of . . .”
Additionally, illustration queries can be used to determine the relationship between the appearance of the object in the real world and its appearance in the illustration. Although an object may have a bright red surface in the real world, it may appear gray in the illustration. This information may be used by a media coordinator to explicitly assign the goal to show the object’s material to one of the media generators. This information may be used by the other generators. The text generator could plan to describe the object as a “bright red” object but adjust its plan to integrate the knowledge that the object appears grayish, and instead generate something like the “the grayish object is actually bright red.” Using all the knowledge available, the text could be modified to say, “the barely visible object in the back of the room is painted in bright red.”
Style-queries concern the methods used to achieve visual
effects. The system can report the number of visual cues used,
how successful they are, and why they were applied. Not only
can other generators refer to the specific visual cues but also
to their intended use. Instead of simply referring to the “highlighted
object” a text generator could refer to “the object
that has been highlighted to show you where it
is.” Style strategy evaluators can be called on to report
on goals not currently associated with the illustration. For
instance, the system may query how recognizable an object is in
the current illustration. This information may be used to determine
what information has been presented to the user (and to determine
what should be presented next).
Figure 6-10. Two cross-references generated by COMET.
Communicative goal-queries concern how well communicative goals are achieved. This type of knowledge is useful for a number of applications. What is more, the mechanisms for self-evaluation can be called upon to determine the success of goals that are not currently associated with the illustration. For instance, although an illustration is designed to show how an object is manipulated, an illustrator can becalled to evaluate how well that object’s location is conveyed. Goal evaluations can be used in a multimedia system to negotiate the assignment of goals. The media coordinator’s task is to ensure that the entire communication is successful. If a generator does not satisfactorily achieve all the goals it was assigned, then those failed goals can be assigned to other generators. We believe that all generators should be able to evaluate how well goals have been achieved and should make this information available externally. Figure 6-10 shows two cross-references COMET generated using information provided by IBIS [McKeown et al. 92]. The cover plate is referred to as “highlighted” and the illustration is referred to as “on the right.” Figure 6-11 summarizes the types of information that can be obtained.
Communicative goals:
what goals are successful, what goals are unsuccessful
given a particular goal, how well it is achieved and how
what goals are associated with a particular object
Designs:
what designs are used and why
Style Strategy Goals
what goals are successful, what goals are unsuccessful
given a particular goal, how well it is achieved and how
what goals are associated with a particular object
Styles:
what styles are used and why
Illustration Procedures:
what illustration procedures were called and why
what procedures were called relating to a particular object
Illustration:
what are the illustration’s data objects contained:
list of relations, illustration-objects, meta-objects
view specification
lighting
special effects
size, location, position and type of an illustration
As time passes, due to the user’s actions (or lack of action) another module (and as we will see in Chapter 7 the illustration system itself) can determine that the communicative goals should change. There are several reasons why goals may be changed. A user may simply not understand the communicative material he or she is being presented with. The user can indicate this directly, by asking for further explanation, or the user’s actions may be interpreted to mean that further explanation is needed. In Chapter 7 we will show how a user’s actions in the real world are monitored to determine when to change goals. In one example we will show how the system decomposed a task into subtasks; in another example we will show how the communicative intent is updated to reflect the next steps in a task. In this section we describe how COMET changes goals to respond to a user’s query.
Without modification to the architecture, our system supports the introduction of new goals. When the system is given a communicative intent to satisfy, a set of communicative goals is assigned in sequence to the illustrator. New goals can simply be added to this sequence. Similarly, our system easily supports the modification of existing goals by changing their thresholds. However, if the importance of a goal is lowered, then the system will not alter the illustration it is already satisfied. Thus, the illustration may emphasize this goal to a greater degree than desired. Meta-rules are necessary to handle these goal changes more effectively, depending upon the particular situation.
Figure 6-12 shows the meta-rules for goal changes applied when the goal is changed as a result of the user’s query. It will be shown in the next sections how these goals are applied.
1. reorder methods so that
methods that alter the view specification are tried last
2. if a new goal is introduced
then choose a method that alters the current illustration,
preferably a method currently used in the illustration,
and use smooth gradual transitions
3. if the importance of an existing goal is increased
then generate an inset for that goal
that applies a different method to achieve that goal
When a user is viewing an illustration, the change in view would be abrupt and disturbing. It may introduce confusion by causing the user to lose his or her sense of location and position. When addressing user queries, meta-rule 1 specifies that methods that alter the view specification be tried last.
If the user is viewing an illustration and asks a question, this question should be answered in context of the current illustration. Meta-rule 2 specifies that methods that alter the current illustration be used and that they be applied gradually, to draw attention to them. Meta-rule 2 also specifies that the method applied should be one with which the user is familiar, or in this case, one that is currently being used.
If the user is viewing an illustration that currently is evaluated to satisfy a particular goal, and the user requests that this goal be satisfied more, an obvious visual cue is warranted. In this case, an entirely new method should be applied to achieve this goal, and a separate illustration is generated to address that goal. Meta-rule 3 specifies that an inset be generated and that a new method be used to satisfy the goal.
User queries are fielded by COMET’s user-interface and handled by the content-planner [McKeown et al. 92]. A COMET user can query “where” an object is located. The content-planner sends IBIS a location goal for that object. If a goal for showing the object’s location is already associated with the illustration, then a goal to increase its importance is sent.
Figure 6-13. COMET explanation. Cutaway view is generated in response to a user’s query.
Suppose COMET’s user is presented with an illustration in which the holding battery is partially occluded by other objects. The user uses COMET’s user-interface to ask, “Where is the holding battery?” and a location goal is assigned to IBIS. Meta-rules 1 and 2 are activated. The location method specifies that the battery must be visible. The illustration procedure detects that the battery is occluded by several objects. The drafter selects to generate a cutaway view to achieve the visibility goal now associated with the battery. The resultant cutaway view is shown in Figure 6-13 along with the text that is generated by COMET’s text generator using the information provided by IBIS about the styles used.
Suppose that COMET’s user is presented with the display shown in Figure 6-14. The illustration has been designed to show how the MODE dial is set [McKeown et al. 92]. The communicative intent included a lower priority location goal for the MODE dial, which has been evaluated as successful. However, the user repeatedly queries “Where is the MODE dial?” and COMET sends a location goal to IBIS. Meta-rule 3 is applied since the location goal is currently satisfied, so an inset is generated for the location goal. An illustrator is created to handle the high-priority location goal. It inherits the current specification of the illustration and selects the location design method shown in Figure 6-5. The inset is positioned near the dial and the resultant illustration is shown in Figure 6-15.
Figure 6-14. COMET explanation.
Figure 6-15. Response to user query in COMET. Increase importance of communicative goal: location of MODE dial.
The objects that are depicted in an illustration may be moving or changing state. The illustration system can receive information from another module that reports new position information relating to the objects in the world. This information can originate from sensors placed on objects in the real world, or can be generated by a simulator. Thus the system is required to update the illustration continuously to reflect the objects movements. However, once again the illustration remains bound to the communicative intent, so as objects move, the illustration is redesigned if goals are violated.
The same mechanisms that are used to maintain the current state of an object (so that previous states can be illustrated in the same illustration) is enhanced to support changing worlds. During the course of a COMET session different tasks are described to the user. Each task potentially changes the state of the radio. These changes are recorded so that subsequent illustrations are consistent with the current state of the world during the various stages of a procedure. For example, it is explained that the knob is turned to a particular setting. All subsequent illustrations depicting the radio in this session should reflect this change, showing the knob turned to the new setting. The system thus maintains a record of the current and past states of the world.
This representation of the world can be changed by information provided by external modules. As shown in Figure 6-16, a module is introduced to alter the values that describe the objects. As described in Chapter 3, the relationship between the objects that appear in an illustration and the objects in the world is maintained. These relations are used to update the illustration-objects as well as to alert the drafter that reevaluation is warranted. In Chapter 7 we will show how this mechanism is used to handle tracked objects. However, this mechanism can also be used in simulations.
The object-representation describes the different states of an
object. This information can be used to detect when objects reach
a certain state. For example, the dice shown in the illustrations
in Chapter 1, are at state “rest” when they are no longer
moving.
Figure 6-16. Illustration process with changing world. Shaded area indicates changes to the architecture.
The rule for showing the value of the “roll,” depends upon this knowledge, so that the rule will be activated only when the dice are no longer moving. We will show in Chapter 7 how this mechanism can also be used to change goals and detect task completion.
We have described some types of interaction to enhance communication with the user. Our methodology is based on a dynamic system of methods and evaluators. These methods and evaluators allow our system to support the four types of interaction described in this chapter: user-navigation, changing worlds, goal changes, and self-evaluation. User-navigation enables the user to interact directly with the illustration, and thus to change the view specification to see the illustrated 3D environment differently. We have demonstrated that an intent-based approach to user-navigation that ensures that communicative goals are continuously satisfied. We have also shown that this illustration system need not be limited to static worlds. The object representation scheme and illustration representation presented in Chapter 3 easily supports changing worlds, because the relationships between illustration-objects and objects in the world is always maintained. In Chapter 7 we will show how these mechanisms enhance communication by enabling the system to determine what the user is doing.
The mechanisms for changing goals and self-evaluation allow other modules to determine, using a greater knowledge, what communicative intent to specify. Thus, interactive intent-based illustrations enable the user to engage in a dialogue, in albeit a limited fashion, with the illustration system, either directly by user-navigation, or through the intervention of other modules. The mechanisms for goal changes enable the system to respond to the user. This is accomplished indirectly through other modules that determine the communicative intent. In Chapter 7 we will show how this is accomplished directly. The modes of interaction described here have been designed to support the illustration system’s participation in a multimedia system. The mechanisms for self-evaluation enhance multimedia communications by providing knowledge that can be used in media coordination, cross-referencing, and goal negotiation.