Co-operative Evaluation of a Desktop Virtual Reality System, Workshop on User Centered Desi

Co-operative Evaluation of a DesktopVirtual Reality SystemTim Marsh and Peter WrightHCI GroupDepartment of Computer ScienceUniversity of YorkYO10 5DD{tmarsh, pcw}@AbstractA summative usability evaluation of a desktop virtual reality (VR) system wasdeveloped and a preliminary study then conducted. The purpose of the study was twofold. Firstly, to test whether the traditional evaluation technique, co-operative evaluation, is effective in the evaluation of desktop VR systems. Co-operative evaluation is a variation on a ‘think-aloud’ verbal protocol, whereby, in addition to concurrently 'thinking-aloud', users are encouraged to ask any questions about an evaluation, relating to the computer-based system, the application, or the tasks that they are required to perform during the evaluation. As well as this, the evaluator may ask questions of the user at any time during the evaluation. Results from the study indicate that this additional probing technique enables an evaluator to elicit further usability problems that may not have otherwise been exteriorized by the user. Additionally, a method is developed which attempts to turn round the qualitative 'think-aloud' type data into quantitative data. This provides a way of evaluating empirical 'think-aloud' evaluation methods and will be useful for comparing their effectiveness to evaluate 3D virtual reality systems. Secondly, attempts were made to capture a user’s awareness of the spatial layout of virtual environment (VE) and the objects and their positions within the VE. However, problems with the methods that were employed in the evaluation are identified, and suggestions that may inform future evaluations are made in an attempt to capture more directly a user's awareness of the spatial layout of a VE.1 The StudyA preliminary study was conducted with 4 users employed in a summative evaluation of a desktop VR system. The following are the research questions that the study attempts to answer:i.how effective is co-operative evaluation in evaluating the usability of desktopVR?ii.is co-operative evaluation an appropriate technique for capturing the spatial features or qualities of a 3D virtual reality system?The 4 users volunteered to participate in the study. They consisted of 2 males and 2 females. All used computers in their daily work and two users had home computers.Their daily computer usage ranged from approximately 1 hour to 9 hours per day. Only one user had experience of VR for entertainment purposes.2 Desktop VRThe test environment for the study was a desktop VR system called the Virtual Production Planner © [VPP97]. Virtual Production Planner is a desktop VR based training system. End users are intended to be student and professional television camera operators, producers, set designers and directors. Its purpose is to provide a teaching and development environment for a range of studio television production techniques from, storyboarding, camera operation and planning, development and creation of virtual productions, to set design and layout, and directing virtual productions. The sets provided are 3D computer generated graphical representations of two British Broadcasting Corporation (BBC) television programmes, the soap opera EastEnders™, and the news and current affairs programme, Newsnight TM The range of features and functions that can be performed in the Newsnight TM set are representative of that provided in the Virtual Production Planner system and therefore, this was used in the evaluation study.3 MethodThree standard tasks were devised to test the usability, functionality and effectiveness of the Virtual Production Planner:Task 1. Identify / Operate Cameras:Users are asked to identify the 5 cameras in the studio. 4 conventional studio cameras that move around the floor of the studio, and 1 free or flying camera, and perform the following: Select cameras, Move cameras, Control the camera views.Task 2. Orientation / Familiarization:The second task consisted of 3 phases. Firstly, by viewing through camera 1, navigate around the Newsnight studio, and draw a plan view of the studio. Secondly, using the flying / free camera navigate around Newsnight studio and again draw a plan view of the studio. Finally, users were shown how to obtain the plan view mode of the studio and asked to move the cameras around the studio from the plan view. Following task 2, subjects were then asked to complete a questionnaire.Task 3. Set-Up / Line-up Cameras: Without using the plan view, subjects were asked to position camera 1 to show a close-up of the presenter, and position camera 2 to show a shot of the presenter over the guest’s left shoulder. Using the plan view, subjects were asked to position camera 3 to show a straight-on shot of the desk, with the presenter and guest seated at either end and the monitor over the desk, and finally, position camera 4 to show a close-up of the guest with the studio monitor displaying the BBC logo to the left of the screen.©BBC/Colt Virtual Reality Ltd. 1997.™EastEnders and Newsnight are trademarks of the British Broadcasting Corporation (BBC).To obtain information about how well VPP supports the tasks, users were asked to adopt a concurrent ‘think-aloud’ verbal protocol. Additionally, users were encouraged to ask the evaluator any questions about the desktop VR system and the tasks that they were required to perform, and they were informed that the evaluator may ask questions of the user at any time during the evaluation. This method of evaluation is co-operative evaluation, as previously described. For an in-depth treatment of co-operative evaluation the reader is referred to Wright and Monk (1989, 1992), and Monk, Wright, Haber, and Davenport (1993). To provide a way to assess the co-operative evaluation technique, a method is developed which attempts to turn round the qualitative 'think-aloud' type data into quantitative data. Based on a similar technique proposed by Hackman and Beirs (1992), empirical evaluations are assessed according to their quantity and quality of the users' 'think-aloud' verbalizations. The quantity is attained by counting all verbalizations; scoring a single utterance, statement, or sentence, or group of these relating to the same issue, as one. The quality of each of these is then judged as low or high, by two evaluators: the study evaluator and an independent evaluator. The criteria by which this will be judged is as follows: Low Quality usability problems are judged to be of low importance or low quality to designers of virtual environments, are more than likely to be user errors in operation, and users will have no problems overcoming these with more usage and time with the virtual environment. High Quality usability problems are judged to be of high importance or high quality to designers of virtual environments, they are not the criteria for low quality usability problems, they did occur frequently, or usability problems that may not likely be overcome with more usage and time, and may possibly shock or startle a user, for example, moving through or colliding with objects. Essentially, this provides a method of evaluating the evaluation technique. Finally, as suggested in task 2, a user's awareness of the layout of the studio and the objects and their positions within the studio is tested. This is carried out by getting the users to draw plan views (birds eye view) of the studio layout and the positions of objects on grid paper. Two plan drawings will be made. One using the conventional studio floor camera (moving along the floor of the studio) and the other using the flying camera (flying through the space of the studio) to navigate around the studio. During each navigation, users are asked to draw a plan view of the studio layout.4 ResultsUsers’ times to complete the study ranged from 15 to 43 minutes (26.5 minutes average). Refer to table 1. The evaluation produced a total of 130 verbalizations. Of these, 30 were usability-related verbalizations. A full list of usability-related verbalizations exteriorized during the evaluation study is given in the Appendix. 10 of these (33%) were exteriorized following evaluator questions to the user.User:1234Total‘think-aloud’ verbalizations (in-total) (average time for each verbalization)14(184s)66(28s)31(36s)19(47s)130Usability-related verbalizations (quantity in total) (percentage of total verbalizations)9(64%)12(18%)3(9%)6(32%)30Evaluator questions to user24146347Prompting usability-related verbalizations (Percentage of usability-related verbalizations)5(55%)3(25%)1(33%)1(17%)10User questions to evaluator823518 Reminded to talk-aloud70018 Study time (minutes)43311915Table 1. ‘Think-aloud’ verbalization made during the evaluation of a desktop VR systemEvaluator 1LOW HIGH TOTALLOW10515 Evaluator 2HIGH01515TOTAL102030Table 2. Confusion MatrixThe usability problems were judged to be of low or high quality to designers of VEs by the two evaluators (see Appendix). A confusion matrix to show the level of confidence and not chance judgements of low or high quality usability problems, is shown below in table 2. From the confusion matrix, we have a coding schedule of 2 different categories (LOW,HIGH) and there are 30 occasions when coding has taken place. With two observers, an agreement takes place when they both use the same code for the occasion. If they use different codes then that is in disagreement. The pattern of agreements and disagreements is shown in the 2 dimensional matrix in table 2. The diagonal from top left to bottom right, indicate agreement between the two observers; scores off this diagonal indicate their disagreement. The evaluators were in agreement 25 times, and of these, 15 of the usability problems were judged to be high quality and 10 low quality. The remaining 5 were judged to be both high and low. Using Cohen’s Kappa (K), the results of the confusion matrix provide a significance rating of 83%, which according to this formula gives an ‘excellent’ degree of confidence for the result [Robson95]. Therefore, of the 30 usability-related problems, 15 were judged in agreement to be of high quality and, of these, 7 (~50%) were exteriorised following a question from the evaluator to the user. On analysis, the users’ 30 usability-related verbalisations were reduced to 11 related, similar or duplicate usability problems. Refer to table 3 below. Down the left hand column are the 11 overall usability problems by users, along the top row. Indicated in each segment is the number of times each usability problem was verbalised by each user, and whether this was judged to be of low (L) or high (H) value to designers of desktop VR virtual environments. Entries with just one letter, either 'L' or 'H', indicate that each evaluator was in agreement. Two entries, 'L' and 'H', indicate that the evaluators were in disagreement, and 'E' indicates that the usability problem was exteriorised following a question from the Evaluator to the user.Usability Problem User1User2User3User4No of TimesUsabilityProblemVerbalisedNo. of UsersVerbalisingUsabilityProblemCollision with objects within studio (HE)(HE)(L/HE)(L/H)(L/H)(L/H)(L/H)7(2 HE,4 L/H,1 L/HE)4Initialinteractionwith on-screencameras(H)(H)(H)(H)4H4Moving through objects Disorientation (HE)(HE)(H)3H(2HE)3Confusion with camera control on-screen icons (L)(HE)(HE)3(2HE,1L)3Confusion with other on-screen icons (LE)(L)(LE)(L)4L(2L,2LE)3No represent-ation of free/flying Camera (H)(H)2 H1System shouldshow where useris, and what’shappening(HE) 1 HE1Confusion withmain and miniviews(L)(L)2L2Confusion about camera movements (L)(L)2L1Feedbackselecting camerain plan view(H)1H1Problems withspecialisedterminology(L)1L1Total9123630Table 3. Reduction of ‘think-aloud’ usability verbalizations to similar usability problems (L = Low quality; H = High quality; E = exteriorized following Evaluator’s question) Finally, an attempt to capture the users’ awareness of the layout of the studio and objects was made. Essentially, the plan views drawn by all users, for both the conventional and the flying camera were very similar. The greatest difference between them was in the increased level of detail, including, the layout of the studio and the number of objects in the studio that were contained in the flying camera’s plan view. Refer to table 4 below. The percentage of the total number of objects and their correct location that were drawnin the plan view using the free or flying camera to navigate around the studio, was 92.5%. This is in contrast to 72.5% of the total number of objects and correct locations drawn in the plan view using the conventional floor camera 1. However, the plan view using the flying camera was drawn after the users had already navigated around the studio using camera 1, drawing a plan view as they went, and therefore, a cumulative effect may well have had an influence on these results.Camera 1: Plan View Free Camera: Plan ViewUser:12341234 Studio Objects:PresenterGuesttable (main)table (far end)Monitor (TV)Audio speakergaps in studio wallsCamera 2Camera 3Camera 4Table 4. Studio objects drawn on the plan views using camera 1 and the free camera5 DiscussionThe quantity of ‘think-aloud’ verbalizations made during the study was 130. 30 of the verbalizations were usability-related, and of these, 15 were judged in agreement to be high quality to designers of future versions of the desktop VE used in the study. Out of the total number of usability-related verbalizations, 10 (33%) were solicited using the questioning technique of co-operative evaluation. Of these, 7 were judged to be high quality usability related verbalizations, that is, almost 50% of the total high quality verbalizations were exteriorized following questions to users. Many of these however, may possibly have not been exteriorized without this additional probing technique. For instance, questioning the user during the study permitted the evaluator to immediately clarify certain issues or concerns raised during the study, or hesitations made in response to a user’s performance with tasks. This additional probing technique proved to be quite an effective way to pursue further lines of inquiries, whilst the thoughts relating to these were still in the users’ mind. In some cases, additional usability problems were identified that may have not been exteriorized otherwise. Although quizzing the user may provide the evaluator with a means to extract more information, the process of questioning the user whilst performing a task may however, introduce further problems. In effect, asking questions of the user is momentarily stopping the evaluation and users may view the questioning as intrusive to a task. For instance, users responding to questions may lose their trail of thought, and possibly the contents of short-term memory may be altered or worst still, lost forever. Hence, it may take users some time to resume their current task following a question. However, during the evaluation of the desktop VR system, there did not seem to be any noticeable addition in time for users to resume their task following a question. Three out of four users said during the debriefing that they had “no problems”with the questioning technique, the remaining user said however, “asking questionsbothered me while doing the task”. One possible solution to this might be to postpone the questioning until the task is complete. Although, this is effectively now conducting a retrospective evaluation and may introduce associated problems, such as, users forgetting and even worse, inventing things.Three out of four users experienced no difficulties talking aloud and one of these added that it helps when learning new skills. Only one user had difficulties with talking aloud and had to be constantly reminded to talk aloud during the evaluation. In the debriefing, the user suggested that this was because of their shy personality. The same user also had difficulties answering questions during the study (see above). This does seem to suggest avoiding the use of users in ‘think-aloud’ evaluations, and especially, using co-operative evaluation, with a shy or introvert disposition. However, whilst the user did have to be constantly reminded to talk aloud, was asked the most questions from the evaluator, and stated that asking questions was bothersome during the study, the same user highlighted the second most usability-related verbalizations and gave the greatest number of high quality verbalizations.The method used in an attempt to capture a user’s awareness of the spatial layout of the studio and objects, and their positions within the studio failed for two main reasons. Firstly, the plan view using the flying camera was drawn after the users had already navigated around the studio using camera 1. A cumulative effect may well have had an influence on these results, and so any comparisons that are made between the two are irrelevant. Secondly, users drew the map whilst they moved around the studio and therefore, this may not allow the user to fix in their minds or commit to memory the studio layout or objects' positions. The lessons learned from this study have been used to inform evaluation methods in an attempt to capture a user’s awareness of the spatial layout of a HMD stereoscopic VR system. Whereby, a user explores a virtual environment for a predetermined period of time (unknown to the user). After this time has elapsed, the display screen is blanked (HMD removed) and immediately following this, the user is asked to draw the layout of the VE and the objects and their positions within the VE from memory. A user's awareness of the spatial layout of the VE is then proportion to the accuracy of their plan view and number of objects and objects' positions. This is similar to a technique (SAGAT) proposed by Endsley (1989) to measure pilots’ situation awareness. In addition to this, a study was carried out to test the effectiveness and appropriateness of the co-operative evaluation technique to evaluate the usability of the stereoscopic Head Mounted Display (HMD) VR system. The results from this recent study are currently being analysed and will be available soon.6 AcknowledgementsThanks to Shamus Smith for his help with the evaluations. This work was supported in part by the UK EPSRC INQUISITIVE project, Grant GR/L53199.References[Endsley89]Endsley M. R., “Situation Awareness Global Assessment Technique (SAGAT)”, Proceedings of the National Aerospace and Electronics Conference (NAECON), May,1988.[Hackman92]Hackman G. and D. Biers, Team Usability Testing: Are Two Heads Better than One? In Proceedings of 36th Annual Human Factors Society Meeting, 1992.[Monk93]Monk A., Wright P., Haber J. and Davenport L., Improving You Human-Computer Interface, Prentice Hall, London, 1993.[Robson95]Robson C., Real World Research: A Resource for Social Scientists and Practitioners-Researchers, Blackwell, Oxford, 1995.[VPP97]Virtual Production Planner (VPP), COLT Virtual Reality Ltd., 1997.[Wright89]Wright P. C. and A. F. Monk, A cost-effective evaluation method for use by designers, International Journal of Man-Machine Studies, 35, 1989.[Wright92]Wright P. and A. Monk, Co-operative Evaluation: The York Manual, CHI92 Tutorial, CHI 1992.AppendixUser Task:Verbalisations: Usability ConcernsEvaluator 1Evaluator 2Low High Low High 1Task 1 1. User clicks on graphical representation of camera 1;nothing happens.2. Evaluator question to user: “have you any idea why thereare two sets of numbers?User reply: “NO”3. Evaluator question to user: “have you any idea whythere are two views?”User reply: “NO”Task 2 i 4. User cannot find free/flying camera icon.ii 5. User goes through the roof/ceiling of the studio. Theentire display screen is grey.Evaluator question to user: “have you any idea where youare or what is happening?”User reply “NO”.6. User: “I think its not very friendly”Evaluator: “why don’t you think it is very friendly”User: “the system should show what happens, where I am,and what you need to do to get out of this”iii7. In plan view, user tries selecting a camera with themouse, by clicking and dragging. The camera doesn’tappear to be selected.User: “the colour should be changing to let you know thatyou’ve clicked it”Task 3 i8. User bumps into studio set.Evaluator: “do you know what’s happening”User: ”its moving”Evaluator: “is the camera moving as you expect it to”User: “NO”ii9. User bumps into studio.Evaluator: “what do you think is happening”User: “it’s going forward whenever I try to go backward”Evaluator: “why do you think the camera moves like that”User: “I don’t know, it shouldn’t behave like that”Evaluator 1Evaluator 2 User Task:Verbalizations: Usability ConcernsLow High Low High 2Task 1 1. User clicks on graphical representation of camera 1;nothing happens.2. User: “I don’t know where the free camera is, I can’t seeit”3. Evaluator: “have you any idea why there are two views”;referring to the main and the mini views.User: “I found it confusing at first” Evaluator and userdiscuss the reason for the mini view. User is eventually toldwhat its purpose is.Task 2 4. User appears confused about camera movement.User: “I would have thought that the camera rotates on itsbase, rather, or maybe its moving on a large circle" User:“all the figures have stopped moving”User is referring to the camera co-ordinates along thebottom of the screen.5. User bumps into set.User: “its moving left and I’m pulling it backwards”Evaluator explains that the camera is colliding with objectsor the studio set.6. User thought that the icons directly above each camera,controlled the movement of each camera, and therefore, didnot use all the camera movement icons until the evaluatorsuggested using all controls for more precise movements.7. Flying camera goes through the wall of the studio. Theentire display turns grey. User looks puzzled.Evaluator: “have you any idea what the grey is”User: “run out of the scope of the programme”8. User: “ the camera went in the direction the head waspointing. I would have thought it would move in thedirection of the camera base”9. The user selects the plan view.User : “I don’t know what the representation of the flyingcamera that they use”Task 3 i10. User bumps into wall and camera moves left instead ofahead. User: “I don’t want to go that way I want o gostraight ahead”11. Evaluator: ”have you any idea why the camera moves inthat way in response to your camera movement selection.User: “maybe it’s on a lead or cable and I’ve got to the endof the cable”.ii12. User: “I would expect one icon to move the wheels andone icon to control the movement of the head. But thecamera moves in the direction the head is pointing in”Evaluator 1Evaluator 2 User Task:Verbalizations: Usability ConcernsLow High Low High 3Task 1 1. User: “Now, the obvious method is to click on thecamera itself"User clicks on graphical representation of camera 1;nothing happens.User: “But we get no reaction, no feedback”.Task 2 i 2. User is moving around the outside of studio set, bumpsinto studio one wall, then bumps into another wall.User: “I’m struggling”.User appears to lose co-ordination and control.3. User fails to use zoom in/out.Evaluator: "is there any particular reason why you didn’tuse those buttons”Evaluator points to zoom.User: “In actual fact, I thought that I could achieve the sameeffect with the other buttons, referring to track and pan"Evaluator 1Evaluator 2 User Task:Verbalizations: Usability ConcernsLow High Low High 4Task 1 1. User clicks on graphical representation of camera 1;nothing happens.User: “ trying to click on actual camera, but that doesn’twork”2. User experiments with zoom and track icons. Thereappears to be some confusion between the zoom and trackicons.Evaluator: “do you know the difference between, or thefunction of these two icons”Evaluator points to zoom and track.User: “No, they appear to do the same thing”Task 2ii er: “what is a plan view”Evaluator explains that a plan view is a bird's eye view.Task 3 i 4. User: “how come when I select camera 2 there ”User points to main view camera 2 icon.User: “it doesn’t highlight 2 there?”User points to mini view camera 2 icon.Evaluator: “have you any idea”User: “No”Evaluator: “try clicking on the buttons”User experiments, becoming familiar with the functions.User: “I suppose this is for previewing and this is what theviewers see”.ii 4. User zooms through the ceiling and the whole screen goes grey for a few seconds.User: “Oh, what happened, disappeared”.er moves camera very close to studio set, and appearsdisoriented.User: “I’m entirely lost”.Task is abandoned。