Communication in Virtual Environments Establishing Common Ground for a Collaborative Spatia

Communication in Virtual Environments:Establishing Common Ground for a Collaborative Spatial TaskAnn-Sofie Axelsson1, Åsa Abelin2, Ralph Schroeder11 Dept. of Technology and Society, Chalmers University of Technology, Gothenburg, Sweden2Dept. of Linguistics, Göteborg University, Gothenburg, SwedenEmail: asax@mot.chalmers.se; abelin@ling.gu.se; ralsch@mot.chalmers.seCorresponding author: asax@mot.chalmers.seAbstractThe aim of this study was to find out how the communication process in a shared virtual environment (SVE) varies depending on which type of technical system is used. Of specific interest in the study was how common ground, a fundamental part of the communication act, is established in various computer-mediated collaborative situations.Participants in this study carried out a collaborative task solving a Rubiks-cube type puzzle in three different conditions: (1) in a SVE where one person used an immersive projection technology (IPT) system and the other person used an ordinary desktop system, (2) in a SVE where both persons used the same type of IPT system (though they were in different countries), and (3) in a SVE where both persons used ordinary desktop systems.By varying the technologies in the SVE conditions we wanted to investigate how the grounding process is affected by the different technologies used as well as by the combination of same and different VR systems.By analyzing audio recordings of the communication between the partners in the three conditions we found that the grounding techniques vary between the different conditions due to the characteristics of the technologies used. Most important to the grounding process we found were the technical possibilities of:- hearing the partner- seeing the partner and partner’s objects- sharing the view of the environment with the partner- understanding how (with what devices) the partner interacts with the SVEHere we discuss how different VR technologies affect the grounding process. In the full version of the paper we will go into further depth about this, and also make suggestions of how to facilitate networked collaboration in the future.KeywordsNetworked Virtual Reality, Computer-Mediated-Communication, Social Interaction, Presence, CAVE.1. IntroductionAs in all Virtual Reality (VR) the visual part is central also in Shared Virtual Environments (SVE’s), since the environment itself, objects in the environment, as well as the users, are graphically depicted. However, since it is a collaborative situation in a SVE where users are involved in a joint project of some kind, such that putting together a machine, viewing a shared document, or hunting down a monster, people need not only to see the environment and each other, but also to communicate in order to convey their own intentions, confirm the understanding of each others’ intentions, and coordinate their joint actions. The communication in SVE’s, as in all human interaction, is a mix of explicit (written or spoken) and implicit (e.g. hand gestures, eye movements, facial expressions) components. However, as we shall see, communication is not as easily carried out successfully in SVE’s as in face-to-face situations, since there are certain characteristics of the technology used (input and output) which makes the communication if not more problematic, somewhat different from face-to-face communication. However, as we also shall see, people are generally, if needed, very capable of adapting to new conditions, i.e. achieving their goals but with other means and techniques than usual. This certainly applies to changed conditions for communication. Since communication is essential to most activities, people try hard to make it work. And if it does not work out with old techniques, people create new ones.This paper describes how the communication process in a collaborative situation varies depending on which type of technical system is used. Of specific interest in the study was how common ground, a fundamental part of the communication act, is established in different mediated collaborative situations.2. Method2.1 General Study DescriptionThe present study is based on data collected within a larger experimental study conducted during 1999-2001, which has previously been extensively reported on elsewhere (e.g. Axelsson, 2001; Schroeder, 2001). This larger study investigated, in general terms, how carrying out a collaborative task in pairs in a SVE, as opposed to face-to-face, is perceived by the involved collaborators. The previous papers are mainly based on self-reports through questionnaires, whereas the present study is based on audio recordings of the communication between the collaborators while carrying out the task in pairs.2.2 ParticipantsThe present study reports on the communication between 18 voluntary participants divided into pairs, who carried out a collaborative task in one out of three different conditions: (1) in a collaborative virtual environment (SVE) where one personused an immersive projection technology (IPT) system and the other person used an ordinary desktop system, (2) in a SVE where both persons used the same type of IPT system (though they were in different countries), and (3) in a SVE where both persons used ordinary desktop systems.The audio recordings that are studied here were arbitrarily chosen, such that there was no significant difference between the recorded groups regarding age, gender, or computer experience. There is also a mix within each condition regarding how successful the groups were to solve the task.2.3 Technical DescriptionIn the condition where the collaborators used the same type of immersive VR systems (immersive projection technology, IPT) (hence ‘all-immersive’), one participant (at Chalmers University) used a system that was a 3x3x3 meter TAN VR-CUBE with stereo projection on five walls (no ceiling). The application was run on a Silicon Graphics Onyx2 Infinity Reality with 14 250MHz R10000 MIPS processors, 2GB RAM and 3 Infinite Reality2 graphics pipes. The participant wore CrystalEyes shutter glasses and used a 3-D wand for navigation. A Polhemus magnetic tracking device tracked both the glasses and the hand.The other participant in this condition (at University College London) used a system that was a ReaCTor made by Trimension. It consists of three 3mx2.2m solid acrylic walls and a 3m square floor. It was powered by a Silicon Graphics Onyx2 with 8 300MHz R12000 MIPS processors, 8GB ram and 4 Infinite Reality2 graphics pipes. The participant wore CrystalEyes stereo glasses which were tracked by an Intersense IS900 system. The participant held a device with 4 buttons and an analogue joystick that was similarly tracked.In the condition where the collaborators used the same types of non- immersive VR systems (hence ‘all-non-immersive’), both participants used a desktop system at Chalmers University consisting of a Silicon Graphics O2 with one MIPS R10000 processor and 256MB RAM and a 19-inch screen display. In this case a dVise 6.0 software was used.In the condition where the collaborators used different types of VR systems (hence ‘mixed-immersion’), one participant used the IPT system at Chalmers University (see description above) while the other participant used a desktop system, also at Chalmers University (see description above).With the IPT systems, the participants could move the blocks or cubes by putting their hand into the virtual cube and pressing the button of the 3-D wand. On the desktop system, participants could navigate by moving the middle mouse button and select the cubes by clicking on the cube with the left mouse button. To move the cubes, they had to keep the right mouse button pressed and move the mouse in the desired direction. They could also rotate the cube by pressing the right mouse button combined with the shift key. With the IPT systems, the participants could point at objects quite intuitively, by using their physical hand holding the 3D wand/joystick, something which was not possible with the desktop system. However, both IPT and desktop systems allowed the participants to ‘mark’ objects by selecting them and thereby making the outlines of the selected cube dashed.When it comes to audio communication, the participants in the mixed types of VR systems condition as well as the all-desktop condition used a telephone connection and headsets to communicate, whereas in the all-IPT condition, the Robust Audio Toolkit (RAT) and headsets were used.2.4 ProcedureThe task in all three conditions was to solve a puzzle involving 8 blocks with different colours on different sides and to rearrange the blocks such that each side of the finished cube would display a single colour. The colours on the sides of the 8 blocks were the following: red, blue, green, orange, yellow, white, and black. The participants were given a maximum amount of 20 minutes to solve the task. In all conditions the trial was preceded by 5-10 minutes of instructions to the task and training to handle the equipment.2.5 DataWe recorded the verbal communication between the participants while carrying out the task. The audio recordings between the participating pairs were then transcribed in a fairly straightforward way, such that we did not notate intonations, prolonged sounds, overlapping speech, pauses, or other linguistic phenomena of the more specific kind, since we were only interested in the semantic contributions, meaning and utterances, i.e. more of what was said than how it was said.Although the verbal communication in all three conditions was most likely accompanied by nonverbal communication such as object manipulation, gestures, facial expressions, and body/avatar positioning; we did not record the movements of the participants or, in the SVE’s, of the avatars. However, which will be seen in the results section below, we have not completely ignored nonverbal communication in our analyses, but taken it into account only in those obvious examples when the participants undoubtedly have used a gesture or the like to complement the verbal communication in the grounding process.2.6 AnalysesThe transcriptions of the audio recordings have been systematically examined and carefully analysed in light of the theories about grounding in communication (section 4.). What has been of specific interest are passages in the transcriptions where the participants try to establish common ground by the use of different techniques. In the results section (section 5.) a number of examples will be given which show how different communication techniques are employed in order to create common ground. The grounding techniques vary between the different conditions due to the characteristics of the technologies used. 3. Previous StudiesIf one considers the importance of communication for any kind of activity, one finds that surprisingly little research has been done on communication in SVE’s. Going back to the period before graphical VE’s however, when email, text chats, and MUD’s (Multi User Dungeons) were the only CMC’s (Computer Mediated Communication) available, there werecontinuously new research concerning the differences between communication face-to-face and using different CMC’s Among those, there were only a few studies that took the problem of creating common ground into consideration, and then discussed it more as a problem of lack of cues, shared social context, and anonymity (for an overview, see for example Herring 1996). Since most CMC’s were asynchronous, the problem, of course, was of a different nature than it is when the communicators are simultaneously in the communication space, trying to understand each other. It was more a question of not being aware of what the communicating partner was up to, due to the fact that the partners did not share time and space, than of the more sophisticated problem that has arised with SVE’s, which is sharing the same virtual space, as being almost face-to-face, but still having trouble with technology.A few studies on SVE’s have been done though, however with little ambition to link technology with social interaction. Most studies have only made the simple conclusion that certain devices (e.g. pointing devices, computer screens, system setups etc.) are better and some worse for achieving a well-functioning collaborative situation without any further discussion of how exactly technology affects social interaction or, even less, of how people using technology for communication are able to adapt to the situation and, by using different techniques, overcome these problems without any changes to the technology whatsoever. There are, however, also good examples of studies on SVE’s and social interaction (see for example Churchill, Snowdon, and Munro 2001; Schroeder 2002).4. Grounding in Face-to-Face Communication and in SVE’sEvery day we meet and communicate with people. About the weather, world politics, and every day issues like work, home, and children. When thinking about this, communication and understanding seems like a simple thing. However, when thinking about it closer, one can see that the communication process is rather complicated and that there are a number of things that can go wrong on the way to joint understanding. Clark and Brennan (1993), and later Clark (1996), who’s theories about communication are the basis for the analysis of the data in this study, discuss these problems as well as the techniques to circumvene the upcoming problems.The most basic problem is if the listener does not at all take notice of the speaker. But even if the listener notice that something has been said, s/he does not necessarily hear what has been said. And last, even if the listener can hear exactly what is said, it is not certain that one understands it. In this study we found that the characteristics of the SVE (e.g. impaired visibility and audibility) hinder the development of understanding and make certain techniques necessary to overcome these obstacles.5. ResultsBelow the main problems occurring in networked collaborative situations will be presented, as well as the specific techniques applied in each case to solve these problems. The examples will be discussed in the final section of this paper (in the full version of the paper we will be able to present more examples as well as discuss these in even greater depth).We found that only in the all-immersive condition, where the audio link was somewhat unstable due to network problems, the collaborators seemed to have problems hearing each other, which consequently made them put effort into using techniques for making that first connection with their partner. When an utterance was made either the speaker had to repeat his/her utterance several times, or the listener (who had, after all, noticed that something was said, but definitely not what was said) asks the speaker to repeat the utterance. When these problems occur, common ground is far out of reach.A problem which is more complex and therefore more difficult to solve, is when the communication process has come as far as to understanding, i.e. the utterance has been correctly recognized by the listener, but not understood. This problem occurred in all three conditions, but, which will be discussed more below, due to different reasons. As are the reasons different, as are the solutions. In the following a few examples from the collaborative trials will be given, demonstrating some of the problems that occurred as well as the solutions to these problems.Since a main part of the communication during the collaborative trials was spent talking about the objects (the eight cubes or blocks making up the puzzle) - their colours, their current position, as well as their potential fit with other cubes – the problems of understanding that occurred did, without exception, relate to the problem of referring correctly to an object. Example 1. Alternative descriptionIn order to obtain a joint understanding of which object is referred to, it is common that the speaker, as well as the listener, gives an ‘alternative description’ of which object is referred to. When the speaker does this, s/he usually does this by expanding the initial utterance. In this study we found that in the all-immersive sítuation, only the speaker expanded his/her utterances, while in the other conditions both speaker and listener gave alternative descriptions.Cave: It has to go on the, the black has to go on the inside of the cube, okay(Subject in the all-immersive condition) Example 2. Indicative gestureOne common referring technique in face-to-face communication is so-called ‘indicative gestures’, i.e. using gestures to point out an object to the listener. In SVE’s however, non-verbal language is restricted, usually due to limited gesture input in the system. Although this was also the case in the present study, the collaborators seemed to find ways of getting around these obstacles, using new types of indicative gestures, adapted to the technical characteristics of each system. While in the all-immersive condition it was possible to use the hand-held joystick to point out objects this was not possible in the non-immersive conditions. However, in all conditions the subjects could click on an object and thereby ‘mark’ it - an indication which was also visible to the partner, and therefore very much used by all subjects. However, the difference we found, was that in the all-immersive condition people referred to objects that they were handling themselves, while the non-immersed subjects in the mixed condition referred to objects that were handled by their immersed partners.Cave: Yes, wait a minute there is one more in the back as wellDesktop: Yes, you are getting that one now…Cave: YesExample 3. Trial referenceOne referring technique commonly used by the speaker in face-to-face communication is so-called ‘trial references’, which is a way of referring to something (or someone) you are not completely sure about. The trial reference is usually made complete by the listener who ‘fills out’ the missing information.In the present study this seemed to be a common technique in order to accomplish common understanding. It was most common in the all-immersive condition but did also occur in the other conditions as is seen in the example.Cave: It’s…Desktop: … sort of black on the outsideExample 4 and 5. The body as a point of referenceA referring technique which is not so common in face-to-face interaction but which was particularly common in the SVE’s in the present study, is referring to an object by using the own or the partner’s body as a point of reference. The reason why this technique is not so common in face-to-face interaction, is, of course, that face-to-face we can use other, more subtle techniques like eye gazes and nods instead of explicit and verbal references. The difference we found between the conditions in the present study, was that in the all-immersive condition the subjects to a higher degree used his/her own body as a point of reference, wanting the listener to come and share his/her view, while in the conditions involving non-immersive systems, the subjects used the partner’s body as a referential point for further information about directions (left, right, behind etc.).Example 4.B: I think I did something a little bit eh…, no there… no, oh no… it’s not going very well for us, come, come over here,to where I’m standing, in the corner, [name of partner](Subject in the all-immersive condition) Example 5.Desktop: Yes, because that has the one with red edges or the one furthest out to the right tooCave: The one in your… over here? This one?Desktop: No in the other directionCave: Your, mine… You mean to your right?6. Discussion and ConclusionThe examples presented above show that different grounding techniques are used under different technical conditions. Why is that so? In the following we will discuss possible explanations of what the results showed.The reason that alternative descriptions given by the speaker (and not by the listener) are more common in the all-immersive condition might be that the unstable audio-link in this condition made it less expensive (time and effort taking many turns in order to hear each other) if the speaker would give a well-formulated description than hoping for the listener to fill out what is lacking. The unstable audio-link we also see as a possible explanation to the fact that trial references are more common in the all-immersive condition compared to the other conditions. Trying out a reference seems safer (and again, less expensive) than giving full instructions at once in case the partner might not be listening at all.The almost face-to-face like conditions when using the immersive systems might be a possible explanation to why the immersive subjects tended to make indications more to themselves, while the non-immersed subjects made more references to the (immersed) partner. The immersed subjects had a first-person perspective, were physically in the VE (CAVE), and had intuitive interaction techniques with the environment, while the non-immersed subjects had a more ‘detached’ view of the environment (third-person perspective) and a much more difficult input technique. These technical characteristics might also explain why the collaborators referred to their own (virtual) body in such different ways. Our other reports from these trials (Axelsson et al. 2001 and Schroeder et al. 2001) report that the immersed subjects had a higher sense of presence than did the non-immersed subjects, something which undoubtedly affects the way the subjects refer to their own and other bodies.The few examples in this short version of the paper show that establishing common ground in a SVE is not an easy thing. Each technical system used, with it’s input and output possibilities, allow certain communication techniques, while in connection with other systems (usually with other possibilities), completely new possibilities (and constraints) arise.To conclude, we found the following four technical possibilities most important to the grounding process:- hearing the partner- seeing the partner and partner’s objects- sharing the view of the environment with the partner- understanding how (with what devices) the partner interacts with the SVEIn the full version of the paper we will go into further depth about this, and also make suggestions of how to facilitate networked collaboration in the future.ReferencesAxelsson, AS., Å. Abelin, I. Heldal, R. Schroeder, and J. Wideström (2001). Cubes in the cube: A comparison of a puzzle-solving task in a virtual and a real environment. CyberPsychology & Behavior 4(2 April): 279-286.Clark, H. H. and S. E. Brennan (1993). Grounding in Communication. Perspectives on socially shared cognition. L. B. Resnick, J. M. Levine and S. D. Teasley. Washington, American Psychological Association.Clark, H. H. (1996). Using Language. Cambridge, Cambridge Univ. Press.Churchill, E. F., D. N. Snowdon, and A. J. Munro, (Eds.) (2001). Collaborative virtual environments : digital places and spaces for interaction.. London: Springer.Herring, S. (1996) (ed.). Computer-mediated communication : linguistic, social, and cross-cultural perspectives. Amsterdam; Philadelphia : J. Benjamins.Schroeder, R., A. Steed, AS. Axelsson, I. Heldal, Å. Abelin, J. Wideström, A. Nilsson, and M. Slater (2001). Collaborating in networked immersive spaces: as good as being there together? Computers & Graphics(25): 781-788.Schroeder, R. (Ed.) (2002). The Social Life of Avatars : Presence and Interaction in Shared Virtual Environments. London: Springer.。