Figure 4-1. The Visual Language.
In this chapter we describe the rules and primitives that make up the visual language employed by our system. Our approach is based on the notion of the separation of design and style (modeled after the separation of content planning and surface generation in natural language generation). As shown in Figure 4-1, an illustration is represented by three different levels of primitives: communicative goals represent the illustration's communicative intent; style strategies represent the illustration's design; the values set by the illustration represent the illustration's styles. Two sets of rules define the relationship between each pair of adjacent levels: design rules map communicative goals to style strategy goals, and style rules map style strategy goals to illustration procedures. There are two types of rules, methods and evaluators. Methods represent how to accomplish goals, and evaluators represent how to determine how successfully a goal has been achieved. Thus, design methods identify the sets of visual effects needed to achieve communicative goals, and style methods identify the sets of illustration procedures used to create these visual effects. Style evaluators represent how well a visual effect is accomplished given the current state of the illustration by calling illustration procedures that examine the illustration. Design evaluators represent how well a communicative goal is accomplished based on how well certain visual effects are achieved.
Figure 4-1. The Visual Language.
Although people communicate using visual material in their daily lives, rarely were they formally trained to do so in elementary school. Lacking a grammar book or dictionary describing the syntax and semantics of the visual language, we were unable to apply well-known or indisputable rules of usage. Consequently, we turned to works describing various aspects of visual communication to develop the visual language presented in this thesis.
[Gombrich 72], [Gombrich et al. 72], [Gombrich 84a], [Gombrich 84b], [Arnheim 79], [Arnheim 86], [Wollheim 87], and [Arnheim 88] describe how styles are recognized in visual material. [Wittkower 77] describe how symbols are used in context of culture. [Freedburg 89] presents a theory of response to images, and [Crary 90] provides an historical analysis of vision which describe how people look at visual material. [Wittgenstein 58] and [Wittgenstein 82] examine how aspects (as, for example, depth, shapes, and organization) are conveyed in pictures and describe how the elements of pictures refer to things. [Goodman 76], [Saville-Troike 89], and [Searle 89a, Searle 89b] describe models of communication, and [Pierce 58]. [Eco 76], [Mukarosky 76], [Clarke, Jr. 87], and the articles in [Bryson 83] apply semiotic theory to visual material and art which helped us develop our model of communicative intent: the relationship between the objects, their representation, and how they are interpreted. [Baxandall 85] describes how artists' intentions are realized in art. [Hagen 80] [Hagen 86], [Rock 87], [Pinker 85], and [MacLeod and Pick 74] explain issues in visual cognition and perception. In order to determine how a visual language might be formalized, we turned to works that describe some of the basic primitives of a visual language and what they are used to express in very general terms (such as how certain colors express strong emotion or how certain spatial arrangements express harmony) [Albers 63], [Taylor 64], [Dondis 73], [Itten 73], [Kobayashi 90], and [Davidoff 91]. [Nodine and Fisher 79] provides an anthology of works in pictorial representation and [Dreyfuss 72] and [Frutiger 89] describe specific visual symbols and signs. [Magnum 70], [Guptill 76], [Addington 86], [Martin 89], and [Kelvin 92] each describe technical illustration and rendering techniques. [Bertin 77], [Tufte 83], [Tufte 90], [Holmes 91] and [Robin 92] describe how information is communicated with visual material. Finally, [Meyer 87] inspired us to develop a hierarchical visual language in his discussion of musical styles which he describes as rules, strategies, and laws.
The visual language presented was initially designed to meet the needs of the COMET multimedia explanation system. In later chapters we will show how the language is augmented to support different applications. The visual language is designed to represent the conventions used to illustrate how objects with a specific functionality work, are used and fixed, and how a user interacts with them. Thus, the illustrations are task-oriented and explanation-oriented. We have not attempted to formalize a complete visual language that could, for example, illustrate children's books or convey emotion (although we believe that the primitives presented here are basic and can be applied along with new ones to generate different types of illustrations).
To our knowledge there is no available work that defines a visual language for 3D illustration applying a framework similar to ours as shown in Figure 4-1. The bi-directional mapping of goals (represented by the methods and evaluators) clearly represents how to achieve goals as well as how to determine how well these goals are accomplished in an illustration. Additionally, we have been unable to find any work that describes, explicitly, how to generate and examine all values that define a 3D illustration (view specification, lighting, list of objects and specific rendering instructions) stating both how to achieve a set of communicative goals and how to evaluate those goals' success rates.
We will contrast the structure of the visual language we present by comparing it to another form of communication with which we are all familiar-how we communicate through our personal appearance. This parallel is presented to provide context for the classes of primitives and rules and to demonstrate the generality of the notion of separation of design from style.
Communicative intent describes what it is that the illustration should accomplish. For example, the intent may be to explain something to the user, to get the user to perform some task, or to convey the value of some abstract property. Communicative intent is represented by a set of prioritized communicative goals.
Consider communicative goals in our own appearance and the way we decorate ourselves. We communicate a lot about ourselves by the way we appear or by the costumes we wear [Bojatyrev 76]. We can, for instance, choose to communicate visually that we are married. The notion of being married verges on the universal; yet the manner in which it is conveyed as well as the reasons for which it might be conveyed vary greatly from one culture to the next, and from one person to another. We may therefore say that showing our marital status is a communicative goal, and that this goal can be accomplished by altering our appearance. It is something we may or may not choose to do when we present ourselves to others. Thus, while the communicative goal (showing we are married) is universal, the way we accomplish this goal (e.g., wearing special jewelry) is culturally bound and follows traditional conventions.
We have identified some of the communicative goals to specify the intent in our domain of explanation for maintenance and repair. These communicative goals are each meant to be universal-that is, a picture in any culture could be designed to achieve any one of these goals. But again, what these goals mean and how they are satisfied depends on all aspects of the situation; ranging from cultural differences to the current set of objects being referred to. At the same time, we can look at a picture and based on what we know of the visual cues being used, interpret it to accomplish some of these goals, just as we may look at a person, and based on what we see and what we know of that person's traditions, we can interpret some part of the person's appearance to be a symbol for marriage.
Style strategies describe the visual effects used to satisfy communicative goals. In our example of communicating through our personal appearance, style strategies would include the types of clothing we wear and the jewelry with which we adorn ourselves. The style strategy itself need not specify exactly what pair of pants we are wearing, or even if the jewelry we are wearing is of gold or silver. The style strategy does not specify the purpose of the item of clothing. All the ways we adorn ourselves can be represented as different style strategies. Thus, a general style strategy that specifies that the head be covered can be satisfied in a great number of ways.
Similarly, there are a number of things we can do in a picture. Style strategies describe the types of visual cues and effects in a picture. For example, we can include an object in a scene. This is a fairly basic property of a picture. The style strategy for including an object does not specify why an object would be included, nor does it specify how that object is to be included.
We said that showing that one is married is a communicative goal. Among the style strategies we can apply in our appearance is to wear a ring. A design method for showing a married status, appropriate for some cultures, would specify that a ring be worn on a particular finger. This design method may also specify that the ring be of a certain shape and material. The design method would also specify that in order to show marital status that ring should be visible to the person to whom the status is to be communicated. A corresponding design evaluator would determine how well the marriage status is conveyed based on several criteria, including the visibility of the hand, ring, and recognizability of the ring as a special type of ring. Similarly, a design method in natural language generation would specify a particular sentence structure or paragraph style.
Style methods specify all the different ways to accomplish style strategy goals. Thus, the list of style methods for wedding bands corresponds to a catalogue of available rings. Similarly, style methods in natural language generation would correspond to the different words or phrases to choose from. Style evaluators would be used to determine the ring's visibility and recognizability. Therefore, the communicative intent to show marital status would be evaluated as unsuccessful if the wedding band were hidden or worn on the wrong finger. Similarly, it may be evaluated with a poor success rating if the ring were of an unusual style that is not easily identifiable as of the type "wedding" band.
Thus, the model for illustration that defines communicative intent, design and style can be generalized to represent different sorts of communication. This model provides many advantages. First, it enables us to formalize illustration as a generate-and-test system. Because we have described that all constraints are global, it would be very difficult to implement a system that never made a wrong choice. Second, it provides a clear division of labor for the three components described in Chapter 3. Third, the division of labor reflects a division in knowledge. The system is easily extensible as we will show in later chapters. Since each component and body of knowledge is independent, modifications to one component need not necessitate modifications to other parts of the system. Finally, this model is general so that it can be used by other media generators in a multimedia system and thus provide better interaction and coordination.
As we described in Chapter 3, communicative intent is specified by a set of ordered and prioritized communicative goals. Figure 4-2 shows the communicative goals we have implemented for the COMET system. These communicative goals have been used to generate illustrations explaining the maintenance and repair of an army radio. Figure 4-3 shows the style strategies implemented for COMET. Again, this list does not constitute a complete set of visual effects that can be achieved in pictures, but has been designed to accomplish the communicative goals listed in Figure 4-2.
show show to the object
property show the value of an object's property
state show the object in a particular state
location show an object's location in some context
reference refer to the object
change show the change in an object's state
relative-location show two or more objects' relative
position
In this section all of the communicative goals listed in Figure 4-2 will be shown. Examples in the following chapters will complete the demonstration of each of the style strategies.
IBIS designed the illustration shown in Figure 4-4 to show the radio and to depict it so that its material is shown. The communicative intent used to specify this illustration is:
(show radio highest)
(property material radio high)
As we will see in the following sections, this communicative intent is achieved by creating an illustration object for the radio, by selecting a view specification such that the radio is unclipped, not occluded, and seen from an angle that shows it face and appears in the center of the illustration, and by rendering the radio using full shading.
Figure 4-4.
Communicative intent:show the radio, show the radio's material property
The show goal is used to familiarize a user with a particular object or to introduce a new object. Typically an object is shown along with a specific property. The show goal requires that the object be depicted so that the representation of that object is both visible and recognizable. The design method IBIS applied is shown in Figure 4-5. It requires that the radio be included, visible to a high degree, and somewhat recognizable, and finally that it be the focus of attention.
(method-for-show-with-highest-priority
;; ASSIGNED TO ILLUSTRATOR
(show ?object highest)
=>
;; ASSIGN TO DRAFTER
(include ?object highest) ;; 1
(visible ?object high) ;; 2
(recognizable ?object medium) ;; 3
(focus ?object high) ;; 4
)
As described in Chapter 3, an object is included by generating an illustration-object to represent it. Ordinarily, an illustration-object is generated with the same properties associated with the object in the knowledge-base: its geometry, material and surface information, and position. If the object has subparts, they are included in the same illustration-object unless there exists specialized knowledge that specifies otherwise.
The include goal does not guarantee that the object will appear in the illustration; other goals are used to ensure an object's presence in an illustration, through the selection of a view specification and rendering styles. Evaluating whether or not an object is included entails examining the list of illustration-objects and determining if there are illustrations-objects corresponding to the object, its subparts, and associated and captive objects.
The illustration-object corresponding to the radio in the object list for the illustration shown in Figure 4-4 consists of all the positions of radio's descendent objects.
One of the most important style strategies is visible. An object's visibility depends upon the view specification and the all the objects included in the scene. In our current implementation, the visibility test is performed once a view specification has been specified. (However, we have developed view independent methods to determine the relationship between objects in the scene [Feiner and Seligmann 92].)
Two main tests are performed. First, it is determined if the object appears (partially or completely) within the view volume or in the user's field of view. Second, it is determined which objects occlude (partially or completely) the object. To avoid unnecessary processing, these two tests are used to classify all the objects in the scene with respect to the objects with visible goals.
The current implementation uses bounding boxes to approximate the volumes and areas occupied by different objects in the illustration. While the use of bounding boxes increases the speed to perform the tests, enabling real and near-real time performance, it unfortunately produces inaccurate results when we test for clipping. We have developed different algorithms to determine the set of objects occluding another object [Feiner and Seligmann 92], one of which is based on z-buffer hardware and display-list picking.
The most straightforward manner to make an object visible is to select a view specification in which the object appears in unclipped in the illustration and in which no objects occlude the object. In Figure 4-3, the view specification is constrained so that the radio is not clipped. There are no other objects in the scene so there is no occlusion.
As described in Chapter 3, each object representation includes recognizability constraints. These constraints are designed to ensure that the visual representation enables the user to recognize the object. The recognizable goal is to depict an object so that certain characteristics and/or properties are shown in a way such that the user can recognize the object. This goal applies the recognizability constraints associated with the object. As we described in Chapter 3, the recognizable goal affects how an object is depicted, what features are depicted, from what angle the object is viewed, and at what resolution the object is rendered. Recognizability is evaluated by applying methods that determine whether or not the constraints are met.
The recognizability constraint applied in Figure 4-4 specifies that the radio be depicted with its front face showing with predominance and so that two other faces be shown. The first angle tried is chosen.
The focus goal is to draw the user's attention to an object [McKeown 85]. This mechanism is used so that the user understands that the focused object is an important part of the communication. The method employed in this illustration is to center on the focused object in the illustration.
Focus is evaluated differently. One evaluator determines if the object is the only object that appears in unclipped in the illustration. Other evaluators determine if the object is centered on. Another evaluation checks if other techniques are being used to draw the user's attention to that object, such as highlighting.
When we describe objects to others we say something about the value of particular properties. These properties may be visual or non-visual. For example, we refer to "the yellow pencil," "the wooden pencil," "the #2 pencil," or "the Eagle pencil." The property goal is to show the value of an object's specific property, such as its color or shape. Figure 4-6 shows the method IBIS applied to show the radio's material in the illustration in Figure 4-4. This method specifies that the object be included and that the visual-property be depicted.
(method-for-property-with-highest-priority
;; ASSIGNED TO ILLUSTRATOR
(property ?property ?object highest)
=>
;; ASSIGN TO DRAFTER
(include ?object ?property highest) ;; 1
(visual-property ?property ?object medium) ;; 2
)
Within the context of a multimedia system, the property goal is useful when an object's property is referred to by another component. In this case, the visual representation of the object should be consistent with its description. The property goal ensures that the specific property be depicted and at the same time disqualifies methods that alter the value of that property.
The visual-property goal is accomplished by selecting rendering techniques that show the value of the property. In Figure 4-4, the radio is shown using full shading and the material definition of the radio's subparts has not been altered.
Among the tasks explained by the COMET system is to replace the holding-battery. Now that the user is familiarized with the radio (determined by the user-model maintained by COMET), COMET's content planner determines that the graphics generator should show where the old-battery is located, to show the radio on its top side, and to refer to the new battery. IBIS generated the illustration shown in Figure 4-7 to achieve the following communicative intent:
(location old-battery high)
(state radio top-side highest)
(reference new-battery highest)
As we will see, a cutaway view is generated to reveal the old-battery's location while the view specification is selected to both depict the entire radio and the new-battery that stands to the left of the radio.
Our explanations depend on the user's ability to find objects or understand where they are. The location goal is to show where an object is. Of course, the value of an object's location depends on the context in which it is to be depicted. For example, the keyboard on which these words are being typed is on a specific desk, in a specific room, in a specific building, block, city, country and so on. The location goal is to show an object's location in a particular context object that may be determined by the system using context information, or it may be specified with the goal. As described in Chapter 3, the object representation can specify different context objects that are used to show location. If this information is not provided, then the object's semantic-parent is used as the context object.
IBIS applied the design method shown in Figure 4-8 for showing location. It requires that the object and context object be included. Both the object and context object must be visible, the object must be recognizable, the context object must be recognizable to a lesser degree, and the object must be highlighted. The evaluator for this method checks that these style strategies are successful.
Figure 4-7.
Communicative intent:location of the old-battery, state of the radio upside-down, reference the new battery
(method-for-location-with-highest-priority
;; ASSIGNED TO ILLUSTRATOR
(location ?object highest)
;; RETRIEVED FROM OBJECT REPRESENTATION
(semantic-parent ?object ?context-object)
=>
;; ASSIGN TO DRAFTER
(include ?object highest) ;; 1
(recognizable ?object high) ;; 2
(include ?context-object highest) ;; 3
(visible ?object high) ;; 4
(highlight ?object medium) ;; 5
(recognizable ?context-object high) ;; 6
(visible ?context-object medium-high) ;; 7
)
We described in an earlier section that all the objects in the object-list are classified as occluding and non-occluding. When no view specification can be found to eliminate occluding objects an alternative method is tried.
One solution is to simply remove occluding objects from the scene. However, in Chapter 1 we described the problem of false implicature and in our example with the cat on the mat demonstrated how the exclusion of objects may be misleading to the user. While we do not address the problem of false implicature directly, our methods are designed to avoid introducing false implicatures. Given that every object that is included has been included to satisfy one or more goals, then removing any of these objects threatens to violate these goals. The occluding objects can only be removed if and only if the goals associated with the illustration are satisfied using methods that do not require that they be included. Thus, the bed, mat, and other objects in the room would have been included in the illustration-object for the context-object, the room.
A second method is to render all occluding objects with transparency. This method may introduce another kind of false implicature. Objects that should not be seen may now be revealed, thus altering the communicative value of the illustration by falsely implying that these objects are important to the communication.
To avoid these problems, we have implemented two methods modeled after common illustrative techniques. One is to generate a cutaway view which gives the appearance of a hole having been cut though the object to reveal the object that must remain visible. The second method is to ghost away only those portions of the occluding objects that occlude the object that must be visible. These methods have the advantage of altering only the portions of the occluding objects that actually block the object.
Consider the old-battery which is located in the interior of the radio. No view exists, from outside the radio in which the old battery is not occluded. In this case, portions of the radio occlude the old-battery. In Figure 4-7, a hard-edged cutaway mask has been generated to make the battery visible. Figure 4-9 shows a illustration with the ghosting technique used instead.
The highlight goal is to draw attention to an object by using an obvious visual cue to distinguish it. The appearance of the object is modified so that the user understands that an illustrative technique is being applied to emphasize that object. If the user is aware of the visual techniques that are being used to communicate, then the user is more likely to correctly interpret the intended purpose of the visual cues. In Figure 4-7, because the battery is the central object in the only cutaway view, the highlight evaluator asserts that the battery is successfully highlighted. In Chapter 3 we described other techniques to achieve highlighting.
Explanations that concern how an object works require that the user be able to recognize the object in different states and distinguish these states. For example, an instruction may be to turn off the radio if the power is on. The user needs to be able to determine if the radio is on to determine if this step is necessary. The state goal is meant to depict the distinguishing features that define a particular state. In Chapter 3 we described how state information is encodes as part of the object representation.
Figure 4-9. Using ghosting to handle visibility.
The state goal is to depict the values that identify the object's state. To show the state of the knob, the representation must show the dial from an angle such that the marking and labels are seen. To show the state of the switch, its color value must be shown. To show the state of the lid, it must be shown at an angle that shows how it was rotated. To show the state of the LED display, the message must be legible.
(method-for-state-with-highest-priority
;; ASSIGNED TO ILLUSTRATOR
(state ?object ?state highest)
=>
;; ASSIGN TO DRAFTER
(include ?object ?state highest) ;; 1
(visible ?object high) ;; 2
(recognizable ?object ?state medium) ;; 3
)
Figure 4-10 shows the design method used to show the object's state. It specifies that the object should be included (using the state representation), that it should be visible, and that it should moderately recognizable. The radio object is included in its entirety, the view specification is constrained so that it is not clipped, and the recognizability constraints further constrain the view specification.
The reference goal is used to ensure that auxiliary objects are present in the illustration, thus providing a mechanism to provide coordinated and cohesive presentations. For instance, in the case of a multimedia communication, the text may refer to several objects. This rule ensures their presence in the illustration.
Figure 4-11 shows the design method that is applied in Figure 4-7. It specifies that the object should be included and visible. The view specification is selected so that the new battery (on the left of the radio) is not clipped out of the illustration.
(method-for-reference-with-highest-priority
;; ASSIGNED TO ILLUSTRATOR
(reference ?object highest)
=>
;; ASSIGN TO DRAFTER
(include ?object highest) ;; 1
(visible ?object high) ;; 2
)
The next step in replacing the holding-battery is to remove the captive screws that fasten the cover over the battery's compartment. Again, COMET's content planner determined that the graphics generator show how the captive-screws are loosened as well as their location. IBIS designed the illustration shown in Figure 4-12 to achieve the following communicative intent:
(change captive-screw1 loosened high)
(change captive-screw2 loosened high)
(relative-location captive-screw1 captive-screw2 high)
As we will see, the change in state of the captive-screws is depicted by special arrows designed to show the loosening action. The view specification is chosen so that the radio is shown with a greater degree of recognizability so that the relative-location of each screw, with regard to the radio is depicted.
Figure 4-12. Communicative intent: change of the captive-screws andrelative-location of the captive-screws
The change goal is to depict how an object changes state (or has changed state) as well as the difference between the changed states. Objects change state under different conditions. More commonly, in our domains, a user is instructed how to manipulate an object so that the object reaches a specified state, but objects can change due to internal workings of the object, to other conditions such as time, or as a side-effect of other changes. If the change is due to user intervention, then the user is shown how one object changes from one state to another so that he or she may perform the action required to initiate that change. In the example of turning off the radio if it is on, a change goal is used to show how, specifically, to turn the power off. In Chapter 3 we showed how our system designed an illustration to show the change in state of a radio's dial.
(method-for-change-with-highest-priority
;; ASSIGNED TO ILLUSTRATOR
(change ?object loosen highest)
;; RETRIEVED FROM OBJECT REPRESENTATION
(shape ?object screw)
=>
;; ASSIGN TO DRAFTER
(include ?object loosened highest) ;; 1
(context ?object loosened high) ;; 2
(visible ?object high) ;; 3
(recognizable ?object medium) ;; 4
(meta-object ?object loosen-screw medium) ;; 5
)
Figure 4-13 shows the design method that was applied in Figure 4-12. This method is specific to objects of type screw and for the action loosening. The method specifies that the object be included, that a context object be included as well, that the object be visible, recognizable, and a meta-object designed especially for showing loosening be added.
The methods for showing an object's context generates an illustration-object to represent the context-object, appropriate associated objects, and appropriate captive, and captor objects.
The context goal is achieved by including context objects or by generating at least one illustration-object corresponding to an ancestor of the object and all of its subparts. As described in Chapter 3, the object-representation can specify the identify of the context object, otherwise the object's semantic-parent is used. In Figure 4-12, the radio is used as the context object for each captive-screw.
Technical illustrations typically include special objects that have been added to provide further information. Meta-objects are objects that do not exist in the real world, but are generated to help solve communicative goals. Arrows are used to show different types of object movement and direction such as twisting, turning, opening, pulling and so on, leader lines map labels to objects, align lines show along what axis two or more objects are aligned and a label displays text or icons.
Each meta-object is generated based on the concept that is being conveyed. For example the arrow representing how a screw is loosened is different in appearance from an arrow showing how an object is turned. [Feiner 85] addressed issues for the automatic generation of arrows to show along what path an object moves. We have instead created a library of arrow types that are associated with actions and or objects.
In our current implementation, methods for generating arrows rely on "to" and "from" state representations and the type of action as "turning" or "pressing." For example, the arrow that is generated to show how the channel knob is turned in Chapter 3 was generated with the tail end beginning at the "from" state and the arrow tip ending at the "to" state. Since it is a single-axis "rotating" arrow, a curved arrow style is used.
The arrows that are generated to show the loosening of the captive-screws in Figure 4-12 are created using a circular arrow shape that is scaled and positioned based on the dimensions and position of each captive-screw.
If there are several similar objects or configurations in the world, it may be important to show the spatial relationship between these objects in order to distinguish them. For example, there may be two radios on a table, but the communicative intent concerns how one is manipulated. The scene is depicted in a manner to show both radios. The relative location goal is to show a group of objects in a common context. The context may be provided with the goal or determined by the system. This goal is different from, for example, a set of separate location goals for the individual objects. If the goal is successful, the user should be able to discern the relationship between the objects, that is if they are, for instance, above, below, or aligned with each other. These relationships are semantic in nature and in many cases better conveyed using both text and graphics. This goal may be further specified to emphasize one of these objects.
(method-for-relative-location-with-highest-priority
;; ASSIGNED TO ILLUSTRATOR
(relative-location ?object1 ?object2 highest)
;; RETRIEVE FROM OBJECT REPRESENTATION
(context-object ?object1 ?context-object)
(context-object ?object2 ?context-object)
=>
;; ASSIGN TO DRAFTER
(include ?object1 highest) ;; 1
(include ?object2 highest) ;; 2
(include ?context-object highest) ;; 3
(recognizable ?context-object highest) ;; 4
(visible ?context-object high) ;; 5
(visible ?object1 highest) ;; 6
(visible ?object2 highest) ;; 7
(recognizable ?object1 medium) ;; 8
(recognizable ?object2 medium) ;; 9
)
Figure 4-14 shows the design method applied in Figure 4-12. It specifies that both objects be included as well as the common context-object. It is the context-object that has the highest recognizability goal.
Like natural language generation, illustration can be decomposed into a multi-layered task. We have presented a visual language that consists of three different types of primitives: communicative goals, style strategies and illustration procedures. Two types of rules are used to map communicative goals to a set of illustration procedures. Design rules map communicative goals to style strategies, and style rules map style strategies to illustration procedures. Different types of medium-specific communication can be modeled using a similar scheme. Natural language generation separates content planning from surface generation, corresponding to what we call design and style. We have also shown how the same scheme can be applied to model how people communicate by their personal appearance. The set of primitives we have described are sufficient to generate the illustrations for the COMET domain. In the next chapters we will show how this visual language is applied to accommodate new and different capabilities.