VIRTUAL REALITY AND PROTOTYPING June 1999, Laval (France) A Theoretical Approach of the Des

A Theoretical Approach of the Design and Evaluationof a Virtual Reality DeviceP. Fuchs, F. Nashashibi, D. LourdeauxRobotics Center - ENSMP, Paris, Francee-mail: fuchs@caor.ensmp.fr, fawzi@caor.ensmp.f r, lourdeau@caor.ensmp.frAbstractThe majority of virtual reality researches mainly tackle the problems of modeling and digitalization of the virtual world. On interfacing man and the virtual world, few studies analyze theoretically these problems, with introduction of new concepts and new methods: behavioral interfaces, three levels of immersion and interaction (sensorimotor, mental and functional), the mental immersion based on the schemas, the behavioral software assistance In this article, we propose a theoretical and pragmatic approach for the design and the evaluation of a virtual reality device, with two examples: the virtual reality training on infrastructures railway and the study of the behavior of consumer in a virtual store.1 Theoretical approach of virtual reality1.1. Bases of reality virtualWe expose different levels of definitions which enable to clarify the field of virtual reality, by separating its finality from its functional and technical definitions.1.1.1. Definitions of virtual realityThe finality of virtual reality is to allow one or several people to do sensorimotor, and thus mental, experiments in an artificial world, which is either imaginary, or a simulation of some aspects of the real world. The fields of the real world, which must be simulated, are to be determined during the stage of design. This initial stage is fundamental and must be explicitly analyzed. The principal error is that the designer seeks the greatest "degree of realism" (limited in practice by technical and economic constraints) without worrying to know precisely the fields of the reality which the application requires. It is absurd to wish the behavior of the virtual world entirely identical of the real world is absurd. If we want to make a " virtual " reality, it is with an aim of modifying some fields of the " true " reality. For example:- training of people in a virtual world in order to avoid real risks to them,- studying a future product which is thus not materialized yet, etc.But if a simulation represents only part of reality, virtual reality also enables to simulate differently some physical phenomena, " the realism " going beyond " true " reality:- physical, invisible phenomena by the human senses (radioactivity, infra-reds, etc.) can be virtually represented.- the images of the objects in boundary representation can be more comprehensible. Virtual Reality (VR) can be also exploited to represent concepts, structures and symbolic entities, associated with the simulated real world. Those allow the user to have a better mental representation of its environment.The functional definition enables us to specify the field of the VR applications: with his own perception of reality, man discerns the concepts of time and space on which hecan interact only according to immutable physical laws1. The VR will enable him to be extracted from physical reality to change virtually the time, the space and the type of interaction: interaction, with an environment simulating reality, or interaction, with an imaginary or symbolic world. These three functions allow, by their combinations, a functional taxonomy of all the VR applications [1].The technical definition:to provide these three functions, the VR techniques are founded on the interaction2 in time real (1) with a world virtual (2), using behavioral interface (3) allowing the natural immersion (4) of the user. We will specify the terms " interface behavioral" and the concept of natural immersion.1.1.2. The behavioral interfacesThe techniques of VR enable person to act via its muscles on a virtual environment which reacts in feedback on the user’s senses. We prefer to employ the term of Behavioral Interfaces rather than interface device or man-machine interfaces in order to differentiate the BI : it is a device which exploits a human, natural behavior and without (or with very little) preliminary training. We will specify these criteria further. The "sensorial interfaces" are designed to transmit sensorial stimuli from the virtual world towards the man and conversely the " motor interfaces " to transmit motor responses of the man towards the virtual world.1.2. Hierarchical structure based on immersion and interactionIn any VR device, man is in the center of the system because the virtual application is dedicated to him. On the basis of this report, we have an anthropocentric not a technocentric approach. We thus have the anthropocentric diagram of the immersion of man, as it is perceived by a user (the left part of the diagram). But for the designer, this “user-diagram” is to be supplemented. We schematize the interfacing at the physical level (first level). We speak about sensorimotor immersion, because the computer isBut this diagram is restrictive because it represents only partially the problems and the solutions which must study any designer. The behavioral interfacing implies similar problems of the operator-machine interfacing. In this last case, it is not only a matter of thinking the physical level interfacing, we should also understand from which mental models the person thinks and acts. As Rabardel[2] described it, the interfacing is presented in a triadic diagram. We actually propose the equivalent for the behavioral 1 According to Piaget (we will refer to later), the sensorimotor intelligence organizes reality by building the categories of the action which are the schemas of the permanent object, space, time and causality.2 We think navigation is only one particular interaction.interfacing in VR: the designer schematizes the interfacing by the links from the person to the material interface and from this interface to the computer. The user must have a direct behavior in the virtual world, the BI’s have to be transparent for him. The direct link of the diagram represents this desired behavior: "desired motor function" and "desired perception". The link between the user and the interface device represents effective motor function and "effective perception". The BI’s is based materially on artifacts and are used according to mental user's models.In order to use easily the BI’s, we can provide Behavioral Software, implying the terms : "programmed motor function" and "programmed perception". For example, we can add specific constraints in the movements of a object in order to help the user (an object will be put on a table when a face of this object will be parallel and close to this table) [3]. Effective perception is a sensory perception imposed to the user; it is partial, natural as much as possible and more or less coherent. For example, the perception of the movement utilizes in natural perception: vision, sensitivity to the efforts, the vestibular apparatus and touch. According to the selected BI, some of the sensorial receivers will not be stimulated, others will be stimulated with different characteristics compared with real world. The effective stimulation will perhaps imply inconsistencies which the brain will have to manage for fear of failure (similar remarks for effective motor function). This problem of the inconsistencies is inherent in the techniques of the RV and is often complex to solve. Can the brain adapt itself ? will it refuse them? Knowledge and experiences of the specialists in cognitive sciences can help us to validate our designs.2 Behavioral Interfaces, schemas and metaphors2.1. Concept of schemaWe specified that we seek to propose to the user a natural immersion. At first, we can affirm what is performed naturally is made unconsciously and without mental effort. For example: in a virtual store, the consumer wishes to move between the corridors. We propose to him as BI: a carriage that he will push and orient with respect to a screen. The consumer will unconsciously and naturally exploit the knowledge he acquired. He thus uses a assimilated schema. The concept of schema was proposed by the psychologist Piaget [4]. According to him who analyzed the birth of the child's intelligence, in particular at his sensorimotor dimension, the schemas constitute man's means with the help of which it can assimilate the situations and the objects. We dare todo a parallel between the comprehension of a virtual world by a user and the comprehension of our real world by the child. For Piaget, a schema is a mental structure which enables to repeat the same action or to apply it to new actions (like the consumer handling its carriage under neighboring conditions compared with the real world). It is on this concept that we base our design to obtain really behavioral interfaces, proposing a natural immersion and interactivity. The behavioral interface is thus a mixed entity including an artifact (material device) and therefore a schema. Let us note that :- the schema is reproducible: it contains the conditions of recognition of the situations on which it is applicable.- the schema is accommodated: it applies to new situations- the schema is used and assimilated unconsciously.To specify these concepts, let us take the example of the object gripping. The associated schema is not the particular course of the movements and perceptions. It is the general groundwork enabling to reproduce and to propose varied achievements in different circumstances. The arm is more or less stretched out or opened according to the distance and the size of the object. Whatever the object, we always must act with the same schema of gripping, only sensorimotor parameters vary. A schema , exploited in virtual environment, is often modified compared with the schema in real environment. The requested sensorimotor functions can be partly different from those of the real world. For example, the gripping and the handling of the objects in the virtual store imply modified sensorimotor functions: thanks to a tracker with six degrees of freedom (dof), the user can take and handle the object virtually. But its glance is directed towards the screen, and not towards its hands, and the grasp of the object is achieved without the real closing of the hands.In conclusion, as Piaget had mentioned, the sensorimotor intelligence manages to solve a group of problem of actions, with building a complex system of schema and to organize the reality according to a group of space-time and causal rules. We affirm that the person exploits the same process to organize the virtual world according to a group of space-time and causal rules. It is the fundamental postulate of our VR design.2.2. Use of schemas or metaphors in virtual realityIn the case of technical, economic or theoretical difficulties not making it possible to exploit a schema, we can avoid these difficulties by employing a "virtual" metaphor. Instead of exploiting the natural behavior of the person, one in general proposes a symbolic image of wished motor function or wished perception. For example, one could have proposed to the consumer to validate the purchase of a product performing a click on an icon representing a check-out. This action becomes symbolic and it is no more representative of sensorimotor action in a real store. The use of metaphor requires more concentration and cognitive effort than the exploitation of a schema. The exploitation of metaphor moves us away from the finality of VR. One will thus use a metaphor instead of a schema in case of theoretical, economic or technical impossibility.3 Methodology of the design and the evaluation of a VR device3.1. The diagram of reference in RVThere is a great discussion concerning the concept of immersion or presence in a virtual environment [5]. We propose a general diagram for the VR which defines three levels ofimmersion and interaction with their particular characteristics. We already clarified we speak about the physical level "sensorimotor immersion and interaction", because physically the computer is connected to the man by his senses and its motor responses. This level of immersion and interaction is quantifiable compared to the characteristics of exploited senses and motor answers. As clarified in the triadic diagram on the interfacing, the user must immerse himself mentally in the virtual world, the lower level of sensorimotor immersion and interaction having to be mentally invisible for him. Here, we speak about the second level "mental immersion and interaction". On a third level concerning the application of VR, the objective is to achieve an immersion of the man with the aim of realizing a process or a function. It is the case of a "functional immersion and interaction" (I²=”Immersion and Interaction”). We can compare this three levels diagram with the close representations of M. Fréjus (sensory, cognitive and operative models)[6] or P. David (sensorial, cognitive and behavioral immersions) [7]. In parallel to the two levels of sensorimotor and mental I² for the person, the virtual world has two software levels. Symmetrically in I² sensorimotor, the computer must manage the real time software ( real time kernel and drivers for material interfaces), allowing physical realization of the virtual world. Vis-a-vis I² mental, the specific VR software must manage behavioral modeling of the virtual world. This software part must provide Behavioral Software Assistance (BSA) of " programmed motor function and perception ", associated with the exploited schemas, in order to facilitate I² mental of the person, example: paragraph 1.2. [3]. In all the applications, behavioral modeling is decomposable in some traditional basic behaviors which we will call the "Behavioral Primitives in VR" (BPVR): navigation, object handling, visual observation, orientation, locomotion, etc. We obtain the general VR diagram, which will exploitedWith this multi-field diagram3, we clarify the concepts of immersion and interaction. Instead of seeking to reproduce most accurately reality, we are interested now with the optimum of functional I² for the application and, by deduction, with the optima of mental and sensorimotor I². It is this innovative step we present in the following paragraph.3.2. step of designFor the design of a VR device, we make the assumption that it be preferable to reach the level of sensorimotor intelligence of the user to facilitate its I² in the world virtual, rather than to exploit its level of semiotic intelligence. Therefore for a given BPVR, with some required characteristics, we will rather privilege the choice of the schema (with its corresponding artifact) than exploit a metaphor. The chronological step is the following :Stage 1: determination of the desired functional I²,Stage 2: determination of the necessary BPVR deduced from functional I²,Stage 3: design of the BI deduced from the BPVR (Schemas or metaphors, artifacts, sense and motor answers)Stage 4: design of the BSA from the BIStage 5: design of RV software (BSA program, BI drivers, etc.)The choice of the schemas is thus very significant in the phase of design and must be studied carefully. The schemas used in the virtual must be assimilated and familiar to the concerned population (children, professionals, etc...). In all the cases, they must annihilate the sensorimotor inconsistencies. These ones are important obstacles for sensorimotor and mental I². The study of the sensorimotor inconsistencies and the adaptation of the brain is an opened problem, that the specialists in neurophysiology and cognitive sciences should study further.This general step of design must be validated with user's experiences. A BI has two functionings : the first envisaged by the designer and the second used by the operator. The designer has a technical and internal representation of the BI and the user will have only an external representation. For example, the virtual displacement of the user on a ground can be done by a manual command (with dataglove or 6 dof tracker), if one does not want or can exploit a travelator. But there is a risk that the user could interpret his interface commands differently from those envisaged by the designer. The user 's difficulty is the auto-representation of his body: where am I, where am I going? and what is the reference mark which I am manipulating? (is it located at my current point of view, in the direction of my trajectory, or in the center of my body?).3.3. Step of the evaluationThe evaluation of a VR device must be based on the same groundwork of our general diagram. It should not only relate to sensorimotor I², like often observed. One should not seek to evaluate the degree of realism but one must refer to the only required BPVR and their corresponding schemas to obtain correct functional I². At the level of 3 With each block, we associate the physicist (for the B.I.), the neurophysiologist (sense and motor answers), the data processing specialist (software), the psychologist (mental processes) and the ergonomics specialist (wished perception and motor function).sensorimotor I², the "metrological" characteristics of the artifact (the material interface) must theoretically be values identical to the psychophysical characteristics of the senses and motor answers. If they are lower, one can quantitatively determine the relative sensorimotor deficit. At the level of mental I²,the evaluation is more difficult to quantify. After the choice of a new schema, tests of validation must confirm the choice. The more BI training time is weak, the more effective the schema is. To evaluate concretely the performance of a natural behavioral interfacing, we propose to make traditional tests of ergonomics and tests "without assistance": the testers will try to use the interface without explanations before use. If the interface exploit a easy and little skewed schema, the user must interact very naturally, after a little training time. At the level of functional I², we can give only one general rule of evaluation. The function of VR device must be compared with the similar function in a real world: for example, does the consumer have a similar behavior in a virtual store compared to a real store?3.4. Simple example: opening of a doorWe take the case where the user must cross a door. Instead of using a metaphor (click on icon of a half-opened door), the person must move a handle naturally. The geometrical shape of the handle can indicate the gesture which must be achieved, unconsciously thanks to the corresponding schema of opening: rotation, translation or mixed movements. Some BSA can facilitate handling: proximity attractions, kinematics constraints, etc.3.5. Virtual store of commercial experimentation3.5.1. IntroductionIn Vivo, a French company, performs marketing studies while observing consumers during their purchase. Their behaviors are analyzed. In Vivo, the Robotics Center and SIM TEAM company work together to achieve a virtual store.3.5.2. The analysis of the problem based on our general VR diagramAt the level of functional I²,the device must make it possible to a person to make naturally its purchase, but in a virtual store. The consumer must be able to move in the corridor, stop in front of a shelf and choose a product. The products must be displayed with a resolution of great quality, because it is a matter of esthetic in product packaging. At the level of mental I², we can classify the various BPVR :1 The two most significant BPVR are the observation of a product and its2 The third BPVR is the observation of the products on the shelves.3 The other BPVR are secondary, to insure the conditions of purchase: displacement and orientation of the customer in the corridors, purchase a product.At the level of sensorimotor I²,it is necessary that the selected schemas and the artifacts are employed as natural as possible to induce a minimal time of adaptation. (1 to 3’, weak compared to total time: 20 ’). Under these conditions, the immersion of the consumer will be successful, he forgets that he is vis-a-vis a .We could validate these results with real test where, over 24 people, all could make their purchase, after 3 to 5 minutes of explanations.3.5.2.1. Visual immersionIn order to obtain the visual immersion most natural and most relevant, it is necessary to consider a projection of scale 1. The consumer is thus in front of a large screen. Theproduct must be displayed with a high resolution corresponding to the visual capacities of the user (visual acuteness of 1' of angle). The field of vision must be wide (130 °). Following these psychophysical characteristics, the solution is a group of 6 XGA videoprojectors, providing a 4m x 2m image with 3072x1536 pixels, displayed with 6 graphic cards put into 3 PC computers.3.5.2.2 The natural 3D handling of the products with 6 dofWe refer on the handling schema of an object in the real world which we clarified in 2.1. The handling gesture being identical to those of the real world, we could check that the consumers do not have any difficulty to handle the product virtually. To facilitate the grasp of the product on the shelves and put down in a carriage, we added BSA (proximity attraction). Results of the tests: 22 out of 24 people could observe without difficulty the 6 faces of a package and read the texts there, after 3' of training.Handling of a product with a tracker (6ddl), included in a ball3.5.2.3 navigation in the storeIn the corridors, the trajectories are predetermined to simplify the commands of displacements (BSA). Moving the carriage orders the displacement (see 2.1). The only skew, compared to the schema in reality, resides in the amplitudes of the movements. The design and the realization of this experimentation store were based on our general VR diagram and on the search for schemas. We tested the validity of our patented design, with the following principal conclusion: the customer makes his purchases in a virtual store and is not in front of a screen.3.6 Virtual training on railways infrastructuresThe training on real railways is often complex to implement and do not offer all the teaching possibilities which allow virtual simulations. The French railway society SNCFcarries out a project with our Center on a " virtual training " of its agents for the intervention on the railways.3.6.1. The analysis of the problem based on our general VR diagramAt the level of functional I², VR devices must enable train drivers to be trained for a specified mission on railway infrastructures. For example, the train drivers of TGV (high speed train) should be trained to control and to handle the on line switching at high speed, in the event of abnormal operation of switching command. For the trainer, the device is a teaching tool which must enable him to propose various teaching strategies. At the level of mental I² : the three significant BPVR are the orientation on the railway, displacement 2D along this one (length 300m) and the three-dimensional handling of objects (3ddl). Another BPVR is the visual observation in all the directions. At the level of sensorimotor I²,it is necessary that the selected schemas and the artifacts are of a natural employment to induce a minimal training time. (10’ to 20', weak compared to few hours of training). The immersion of the driver must also be successful. Under these conditions, he forgets that he is vis-a-vis a system and he concentrates on his training.3.6.2 2D displacement on the railwaysA priori, the schema to be exploited allowing a natural interaction is obviously the walk. The basic technical problem is the difficulty to achieve a travelator (2D), authorizing the walk in two directions. Prototypes were produced, but they are difficult to use. Even a one direction travelator (1D), imposes a sensorimotor inconsistency, which must be assimilable: the vestibular system of the user indicates an absence of real movement. It is necessary to use another schema enabling to turn: the schema of the change of direction by a bike handlebars. Motricity BSA will be associated with the exploited artifacts, to facilitate mental I² of the person.3.6.3 The orientation on the railwaysThe orientation poses a delicate problem. Experiments [8] showed that the real rotation of the body on itself was relatively significant for the orientation (contrary to the translations). Will the user be able to be always well oriented if a 1D-travelator is exploited? Tests are envisaged to analyze the solutions.3.6.4 The visual immersionThe training user must able to dialogue with the trainer beside, we thus do not use Head Mounted Displays. For natural actions, the virtual environment must be represented on scale 1. The user must have a large field of vision to see the approach of the trains and to direct itself towards the ways. The display must thus be in back projection by one or more projectors, according to whether a 1D or 2D travelator is exploited.3.6.5 Natural handling of objects in 3D with 3ddl:Compared to the handling of products with 6 dof in the virtual store, the handled objects should not be oriented in all directions. The handling of a tracker with two hands is thus not obligatory in this case. One can thus use a dataglove to handle the telephone or whatever with one hand only. The dataglove has the advantage of transmitting also binary signals by the movement of the fingers (grasp and pose an object, etc.) and to exploit a very natural schema. The principal lack of the schema is in the lack of force and tactile feedback. This lack does not make it possible to reproduce the difficulty of areal handling. BSA are added to this artifact to facilitate handling: attraction (when the glove is near an object, they attract each other) and cinematic restraints. A "virtual pocket" (metaphor) is displayed to virtually move with objects without holding them.4 Prospects and conclusionsWe are prospecting to make a taxonomy of the principal BPVR and an inventory of their solutions, tested and validated by the scientific community. They could be clarified starting from the concepts of schemas and to be improved by adequate BSA. From our experiments, we conclude on the validity of our approach of design and evaluation. We propose this approach to any designer, which will make it possible to validate it and improve it. The accumulation of these experiments must gradually enable to have sufficient knowledge on the effective realization of the principal BPRV.REFERENCES[1] P. Fuchs, Book "Les interfaces de la réalité virtuelle", AJIIMD, 1996, ISBN 2-9509954-0-3, les Presses de l'Ecole des Mines de Paris.[2] P. Rabardel, Book "Les hommes et les technologies, approche cognitive des instruments contemporains", Armand Colin, 1995, ISBN, 2-200-21569-X.[3] D. Mamam, Thèse 1998, Ecole des Mines de Paris, 60 Bd St Michel.[4] J. Piaget, B. Inhelder, Book "La psychologie de l'enfant", Que sais-je, 369, PUF, 96.[5] Livre "Nouvelle Interfaces Homme-Machine", série ARAGO, Observatoire Français des Techniques Avancées, 1996.[6] M. Frejus, A. Drouin, réalité virtuelle et processus cognitifs, EDFF-DER, UniversitéParis 5, René Descartes, 1996.[7] Ph. David, A. Guidon, J.N. Temem & A. Gazet. Using virtual reality fortesting a station signing scheme. WCCRR'97, Florence, 1997.[8] S. Chance, F. Gaunet, A. Beall, Locomotion Mode Affects the Updating of Objects Encountered During travel, Presence, vol 7, N° 2, MIT, 168-178, 1998.。