Temporal Patterns in Complex Interaction

David England

School of Computing and Mathematical Sciences
Liverpool John Moores University
Byrom St
Liverpool L3 3AF



Example Pattern

Name:  Avatar joining or the slow appearance of objects in virtual reality
Context:. Networked virtual worlds
Forces:  The influence of network delays on user interaction and perception of shared objects, plus the influence of graphical rendering delays and complexity
Solution: The rendering of an intermediate object until the full object can be displayed

Position Statement

Our area of interest brings together two strands. Firstly that of design patterns for object oriented design, and secondly that of temporal notations for specifying complex interaction in human-computer interfaces.

Design Patterns initially arose from the work of Alexander (1997) in the field of architecture. Alexander began to recognise that successful architectural designs were based of the combination of already successful individual components. He and his colleagues formulated a design notation and design method that allowed architects and residents to work together to formulate workable designs from a complex mixture of design criteria and compromises. This work found echoes in software design (Gamma et al 1995), and in particular in object oriented design. For example, the Model View Controller (MVC) architecture of Smalltalk 80 (Krasner and Pope 1988) has been recognised as a standard design pattern for Graphical User Interfaces. It is found repeated in Apple’s MacApp Framework, Microsoft’s MFC Document/View and also in Sun’s Java Swing toolkit. However, the design influence of MVC is not limited to the graphical interfaces, it also provides a general basis for considering the creation, composition and communication of objects and classes in a design. As yet, however, it has not been extended to multimedia systems nor to virtual reality systems. The latter in particular is still looking for a common design framework.

Our second thread, of temporal notations, comes from the growing view that human-computer interaction is becoming more complex. Designers are faced with systems that are; distributed, use multiple sources of media, systems with multiple input devices, and applications that use internal concurrency. In earlier work (England et al 1992) we developed a notation, XUAN, eXecutable User Action Notation (Gray 1994) aimed at capturing complex descriptions of interaction. For example, we included elements of temporal relations and temporal constraints to capture descriptions of user interface with time-varying media. A simulator was built for XUAN which allowed our psychology colleagues to experiment with different temporal interaction scenarios.

Coming forward to the present we can use notations like XUAN (or Petri nets, or Statecharts) to help us deal with designs that involve notions of currency and time. However, at the same time, the influence of MVC has been to produce rapid application tools for the production of graphical user interfaces. This has led to a design gap. User interface design usually begins with some task or process model of what the end user wishes to achieve. Rapid application tools begin with a component view of what the user can achieve with a particular interface. The difficulty comes in matching up the required task or process with the appropriate components. One way forward would to produce design patterns that deal with the concurrent and temporal properties of tasks and processes. Currently few such patterns exist and it will be the main task of our future work to develop such patterns. We have made a start on this work with (England and Gray 1998) where we looked at the application of XUAN to shared virtual worlds. We identified some common interaction scenarios. For example, a new user joining a 3d world over a slow network. Our work with XUAN has shown some of its limitations, for example, in repeatingly applying descriptions to common situations. Now this is an area in which pattern notations are ideal so we would want to move XUAN to a more pattern-style approach, perhaps a PUAN, or  Pattern User Action Notation. We would also want to reuse the lessons learnt in building a simulator for XUAN by providing a simulator for PUAN which would enable the developer to specify temporal relations and constraints amongst user and system actions. For example, in our scenario of a user joining a virtual world, we might specify that if they are on a slow link, a temporary avatar might appear in their position, until such time as the geometry of the full avatar arrived.

There is a wealth of material from which we can draw further scenarios. There is our own earlier work on building and using shared virtual worlds (Broll and England 1995). There are a variety of available virtual worlds with which we can experiment such as DIVE (Carlsson Hagsand 1993). More recently we have been working with David Roberts, of Reading University on specifying distributed interactions, for example, playing football in a distributed virtual reality. Our work also links in with the work of Dave Manley (PhD student CMS) and Tony Burrows (PhD Student CMS). Manley is looking at holophonic sound as a representation medium in virtual reality. This has many interesting temporal aspects, such as the temporal nature of sound itself and the sequencing of sound with the visual display and user interaction. Burrows is looking at specifying the behaviour of objects and scenes in virtual worlds. Again we can apply the pattern approach to identifying common themes related to temporal relations and constraints in virtual world


Alexander C, (1977), Ishikawa S, Silverstein S, “A Pattern Language : Towns, Buildings, Construction”, Oxford University Press

Broll W, England D (1995), "Bringing Worlds Together: Adding Multi-User Support to VRML", Proceedings of VRML 95 Symposium, ACM, New York, 1995.

Carlsson C and Hagsand O,  (1993), "DIVE - A Platform for Multi-User Virtual Environments", Computers & Graphics, pp. 663-669, Vol. 17, No. 6

England D (1998), Gray P D, “"Temporal aspects of interaction in shared virtual worlds", Interacting with Computers, Vol. 11, pp 87-105.

England D, Gray P, Draper SW, O’Donnel P, (1992), “Temporal Aspects of Usability”, MRC/ESRC/SERC project No. 9201233.

Gamma E (1995), Helm R, Johnson R, Vlissides J, “Design Patterns: Elements of Reusable Object-Oriented Software, Addison_Wesley.

Gray P, England D, McGowan S, "XUAN: Enhancing the UAN to capture temporal relationships among actions", in Proceedings of BCS HCI '94, Cambridge University Press, September, 1994

Krasner G E (1988), Pope S T, “A cookbook for using the model-view-controller user interface paradigm in Smalltalk-80”, Journal of Object Oriented Programming, 1(3), 26-49, August/September

A position paper prepared for the Usability Pattern Language Workshop at Interact '99