Simultaneous remote haptic collaboration for assemblingtasks.

Multimedia Systems(2008)13:263–274
DOI10.1007/s00530-007-0108-7
REGULAR PAPER
Simultaneous remote haptic collaboration for assembling tasks Rosa Iglesias·Sara Casado·Teresa Gutiérrez·
Alejandro García-Alonso·Wai Yu·Alan Marshall
Published online:30October2007
©Springer-Verlag2007
Abstract Stand-alone virtual environments(VEs)using haptic devices have proved useful for assembly/disassembly simulation of mechanical components.Nowadays,collabora-tive haptic virtual environments(CHVEs)are also emerging.
A new peer-to-peer collaborative haptic assembly simulator (CHAS)has been developed whereby two users can simulta-neously carry out assembly tasks using haptic devices.Two major challenges have been addressed:virtual scene syn-chronization(consistency)and the provision of a reliable and effective haptic feedback.A consistency-maintenance scheme has been designed to solve the challenge of achiev-ing consistency.Results show that consistency is guaran-teed.Furthermore,a force-smoothing algorithm has been developed which is shown to improve the quality of force feedback under adverse network conditions.A range of R.Iglesias(B)·S.Casado·T.Gutiérrez
Labein Tecnalia,Edificio700C/Geldo,
48160Derio,Bizkaia,Spain
e-mail:riglesias@ikerlan.es
S.Casado
e-mail:sara@labein.es
T.Gutiérrez
e-mail:tere@labein.es
A.García-Alonso
University of the Basque Country,
P.Manuel de Lardizábal,1,
20018Donostia-San Sebastián,Spain
e-mail:alex.galonso@ehu.es
W.Yu·A.Marshall
School of Electrical and Electronic Engineering,
Queen’s University Belfast,University Road,
BT95AH Belfast,UK
e-mail:w.yu@
A.Marshall
e-mail:a.marshall@ laboratory experiments and several real trials between Labein(Spain)and Queen’s University Belfast(Northern Ireland)have verified that CHAS can provide an adequate haptic interaction when both users perform remote assem-blies(assembly of one user’s object with an object grasped by the other user).Moreover,when collisions between grasped objects occur(dependent collisions),the haptic feedback usu-ally provides satisfactory haptic perception.Based on a quali-tative study,it is shown that the haptic feedback obtained during remote assemblies with dependent collisions can con-tinue to improve the sense of co-presence between users with regard to only visual feedback.
Keywords Assembly simulation·Collaborative virtual environment·Consistency·Haptic feedback·Peer-to-peer 1Introduction
The benefits of using haptic devices have already been addressed by several researchers.For instance,in industrial applications these devices significantly enhance task per-formance and help designers and engineers to design and evaluate computer generated mock-ups prior to building any physical prototype or,for instance,decide the most appropri-ate assembly sequence[1,2].On the other hand,new CHVEs are also emerging in different areas,for instance,general tele-operations[3]or applied to medical training(tele-surgery) [4],tele-mentoring[5,6],industrial applications and test-ing[7,8]and art applications[9].Such environments allow users to perform tasks such as collaboration and training in real time.The effects of haptic feedback on a collaborative task performance have been studied,and the results showed that adding force feedback to a collaborative environment enhances effectiveness and reduces the time required to com-plete the task[10,11].
264R.Iglesias et al.
This paper introduces a new collaborative haptic assem-bly simulator(CHAS),whereby two users share a common goal and collaborate toward its achievement.For instance, they can collaborate on either preliminary reviews or defect reviews.Each user can touch an object of the virtual scene, grasp it,move it detecting collisions with other objects in a scene and assemble it into another object.Most times the collisions experienced by the user’s object are with static objects;however,sometimes it may also collide with an object grasped by another user.The latter case also occurs when assembling:in this case both users work simultaneou-sly;however,they do not perform a cooperative work,such as lifting an object together.
The closest work to this article is the research carried out by Marsh[7],although their approach differs from the research described.In their approach,collaboration is per-formed by taking turns among participants or by grasping the same virtual object(a cooperative task).
In CHAS,there are two types user interaction,namely, independent interaction and dependent interaction.Indepen-dent interaction is the interaction between the object grasped by a user with static objects.Dependent interaction hap-pens when an object grasped by a user interacts with an object grasped by a different user.Two types of dependent interaction are distinguished:dependent collision and remote assembly.In the case the objects grasped by both users are very close and a collision occurs(dependent collision),the system provides users with the corresponding force feed-back.A similar behavior is required when one user assem-bles a grasped object into an object grasped by a remote user (remote assembly).
The CHAS is based on a single-user system but is dis-tributed in its operation.Section1.1explains this single-user system,the actions that the user can perform in CHAS and identifies some design challenges addressed by this research (Sect.1.2).More detailed descriptions of the single-user sys-tem and CHAS appear in[2,8].
1.1The single-user system(HAS)
The haptic assembly simulator single-user system,HAS is composed of an Assembly Simulator and a Haptic Assem-bly Simulator.Both simulators are based on the geometric modeler DATum developed by Labein,which creates3D vir-tual scenes or automatically imports3D CAD components to simulate assembly and maintenance operations.The objects created or imported into the system can be virtually touched or grasped by the user.The following actions are considered: touch,move a grasped object and assemble/disassemble a CAD component.
The assembly simulator allows inter-object collision detection and automatic recognition of potential assembly constraints between a grasped object controlled by a
user Fig.1Types of assembly
and the rest of the components of the mechanical assem-bly within the virtual scene.Before applying a new position (translation and/or rotation)to an object,it must be checked if the object’s new position will result in a collision with any other objects in the virtual scene.Part of the collision detec-tion algorithms are based on the RAPID library implemented by the University of North Carolina[12].
During an assembly the system automatically recognizes potential assembly constraints between the grasped compo-nent(being moved by the user)and the rest of the compo-nents within the virtual scene.Once the system detects a con-straint,the movement of the component will be constrained to satisfy the active constraint.Figure1shows two generic assembly methods implemented:pin–hole and plane–plane. During the assembly task,collisions with other models of the virtual scene are also detected[13].
The haptic assembly simulator allows the user to interact with the components of the assembly in a more realistic way,using haptic devices.HAS provides the force feed-back according to the user action or interaction:touch,grasp, move,collide,assemble or disassemble.Figure2shows an overview of the HAS system.The system comprises three threads:the rendering thread,the haptic thread and the simulation thread.The rendering thread performs graphi-cal rendering tasks and analyzes user actions introduced by the keyboard.The haptic thread running at1kHz deals with the communication with the haptic device which receives the movement requests by the user and calculates the force to be sent to the haptic device.The simulation thread is responsi-ble for validating the movement requested by the user(i.e., check if a collision occurs or if a new constraint is active,and constraining the movement if necessary).
Simultaneous remote haptic collaboration for assembling tasks265 Fig.2Overview of the HAS
system
1.2Collaborative haptic assembly simulator(CHAS)
The CHAS has the same functions as the single-user sys-tem.However,in this new system two users,which can be geographically distributed or co-located,may haptically and simultaneously interact within the same virtual scene.In this framework,for a fast haptic interaction response,each user has their local copy of the virtual scene and haptic rendering is performed locally at each user’s computer.
A key challenge of this system is to maintain consis-tency(virtual scene synchronization)and haptic rendering in the case of dependent interaction.To maintain consistency, CHAS must avoid each user working with a different repre-sentation of the virtual scene.A user action affects the repre-sentation of the virtual scene at other users and additionally, a user action may affect the remote user action.Therefore, one of the main challenges in CHVEs is maintaining a con-sistent virtual scene in the presence of inevitable network delay,packet jitter or other network conditions.
Because the haptic thread runs faster than the assembly simulator,a multi-threaded solution is employed in the sin-gle user system to avoid degradation of the haptic response (the haptic loop).The haptic thread provides smooth forces either when the simulation thread have not received yet a new valid position,or when the new position generates brusque force changes.However,in CHAS,even with acceptable network communications,a third loop affects system syn-chronization:communications between users.This adds an additional problem to the goal of achieving a reliable and effective force feedback.The computation of force feedback is more demanding when users interact with each other,that is,when dependent collisions and remote assemblies occur. Additionally,network effects can only exacerbate this.
This paper gives an explanation of the consistency-maintenance scheme and haptic-smoothing strategies used. The results of laboratory experiments and real trials carried out between Labein and Queen’s University Belfast are also presented.Furthermore,the haptic perception experienced by users during collaborative assembly tasks was also observed and this experience was used in thefinal tested system.
2Challenges and related research
There is no network topology that is optimal for all types of networked virtual environments.Different topologies may be adopted to build collaborative haptic applications with spe-cific goals,such as client-server[14–17],peer-to-peer[14, 18–21]or a mixture of them,such as,in[7]which adopts a multiple-servers approach.
Client–server architectures have the advantage that con-sistency is more easily achieved[7,14];however,the local representation of the scene is only updated after a round-trip delay to the server.In peer-to-peer architectures,while ensur-ing consistency among peers is a challenge[7,22],the update delay is only one-way.Therefore,with regard to updates, peer-to-peer architectures are less dependent on network communications.Such architectures provide a better perfor-mance with haptic interaction,which becomes important in case of worsening network conditions(e.g.congestion).Fur-thermore,they have the potential to scale to a larger number of users than client-server architectures.
After performing different experiments using client–server architectures[23],a peer-to-peer architecture is chosen because it addresses the two main challenges:a better per-formance with haptic interaction and scalability.Moreover, in peer-to-peer architectures other additional features were considered,such as,reducing the number of messages and parallel computation(i.e.reducing sequential bottlenecks). On the other hand,the main challenges presented by peer-to-peer haptic architectures are consistency maintenance[7,22] and the quality of haptic interaction[15,18,19,24].Both of these issues are affected by network conditions.
A collaborative virtual environment(CVE)is composed of virtual objects,which may be static or dynamic.Some CVEs
266R.Iglesias et al.
allow static objects to be grasped and moved within the virtual scene by means of traditional devices(i.e.,keyboard,mouse) or haptic devices.Consistency has been addressed by many researchers in different ways depending on the nature of the collaborative environment,that is,object behavior and user interaction.In[25]a summary of concepts and approaches to deal with the problem of maintaining consistency in distrib-uted virtual applications under network delay are reported. However,such approaches are more focused on visualization rather than on haptic interaction.
On the other hand,in CVEs where cooperative manipula-tion of dynamic objects is required,the scene synchroniza-tion becomes extremely demanding[26],because concurrent actions on the objects may lead to inconsistent scenes.This is further exacerbated if a user feels another user when pushing the same object through a haptic device.
Diverse synchronization methods or strategies to mini-mize the impact of network on synchronization have been proposed to overcome this problem in CHVEs.For instance, to achieve the collaboration,Kim et al.[20]added damping at the haptic tool interaction,in the dynamics of the object being manipulated and in the haptic tool trajectory.In a later research on the same issue,Cheong et al.[21]proposed a synchronization controller for the motion of the object being jointly manipulated.
On the other hand,to achieve consistency and effective haptic collaboration,Hespanha et al.[18]restricted inter-actions among ers with high delay became pas-sive observers,which could only watch the virtual scene but not interact within it.Moreover,they introduced the concept of object ownership;where periodically the owner of each object broadcasted updates of the owned objects to all other peers.In that way,after a time each object position was the same for all users.Other frameworks that aimed to provide users with the necessary consistency requirements to carry out a task,as well as supporting effective and compelling haptic interaction are described in[8].
It is recognized that haptic devices require an update frequency of1kHz.to achieve robust and effective inter-action.Haptic rendering is the process of computing and generating forces in response to user interactions with virtual objects.In[27]the leading basic haptic rendering algorithms are discussed depending on the degrees of freedom of haptic devices.A classification for different methods is:based on a point-based[28,29],ray-based[30]and3D objects[31]. In this research,a point-based method,as well as the use of geometrical algorithms(provided by DATum),are used to compute haptic response.In both the HAS and CHAS sys-tems,the haptic rendering algorithm depends on user action: touching,grasping and moving an object,detecting collisions and assemblies among objects.
Haptic rendering has been the subject of many authors,for instance,to create different haptic sensations:texture per-ception[32,33],deformable elements[34,35]or collision impacts[36].The algorithm used in HAS to touch an object and move along its external surface,detecting its edges and corners is based on the penetration depth of the HIP(hap-tic interface point)into the virtual object(i.e.a point-based method).However,for colliding with another object,a sim-pler but effective solution was proposed.The penetration depth of the grasped object into the object of collision is not computed.Instead the collision force response is deter-mined by the actual HIP and the last HIP which did not cause collision.For assembling one object into another,the deve-lopment of sliding forces was required.These algorithms are described in detail in[2]and they are also used in the CHAS system.
On the other hand,regarding network conditions,vari-ous experiments have shown how delay,jitter and message loss affect haptic interaction[15,19].Some researchers have studied the effects of delay or jitter on task performance and their acceptable performance limits[37,38].The issue of “network-based force feedback”is addressed in[39].This work describes two strategies to stabilize haptic feedback in the presence of unpredictable delays.Other researchers, such as,Jeffay[40]and Belghit[41]deal with“distant haptic interaction”.
3Consistency
The CHAS system allows two users to interact haptically and simultaneously to undertake assembly and maintenance operations.Both users can simultaneously grasp a different object through a haptic device and move it across the scene. Clearly,with this peer-to-peer system,users do not have to wait for messages from the server,they control their own hap-tic loop and validate all movements.Although haptic inter-action is improved,this approach makes it more difficult to maintain consistency.
Due to the network delay,at an instant of time the grasped objects can be placed in the same space,that is,an inter-object penetration may occur.This leads to the need for maintain-ing consistency in such situations.Whenever dependent col-lisions occur it is necessary to decide to which position each grasped object must be moved to guarantee the same scene at both users.Moreover,providing fast haptic response is also a key factor for effective haptic interaction.The next section looks at the update messages required to maintain consistency in CHAS.Section3.2describes the consistency-maintenance scheme proposed.
3.1Update messages
At any one instant of time each user knows:
Simultaneous remote haptic collaboration for assembling tasks267
1.The position of its grasped object,from here onwards
referred to as:the local object(grasped by the local user), and
2.The current valid position of the other user’s grasped
object,from here onwards referred to as:the remote object(grasped by the remote user).
Messages are used to inform remote users of required scene updates.Messages contain information about local and remote objects at each user,as well as,data to discard dupli-cated or out-of-order messages.From the point of view of data transfer the following characteristics were considered to be important:small messages and low message frequen-cies.A new message is sent out by the simulation thread only when a new local position is validated and changed from the former position.In this way the message rate depends on both the frequency of the assembly simulator and on the result of the validation.The size of the messages sent in each cycle is small(less than200bytes).Messages were trans-ferred via UDP[42]because it has better round-trip delay than TCP.Moreover,UDP has no buffering delays and lower jitter because it does not resend lost packets[9].
3.2Consistency-maintenance scheme
A key feature of CHAS is that object movement is validated only once,at the user where it is moved.For a fast haptic response,each user moves or assembles the grasped object (validates object movements and updates his/her scene)with-out waiting for any message from the remote user.Once the position of the remote object has been received,it is also taken into account in a way that scene consistency is guaranteed.
The proposed consistency-maintenance scheme in[8] guarantees consistency and fast haptic responses in the case of dependent collisions.However,different results analyzed in the qualitative study(Sect.4.5.1),showed how a few instabilities appeared during pin–hole assembly tasks.A consistency-maintenance scheme was therefore designed and tested,in the case where a user has begun an assembly task and the axis or constraint has been detected.Otherwise,if the assembly constraint has not been detected,the consistency-maintenance scheme described in[8]is valid.
3.2.1Scheme without assembly constraint detection
This scheme was designed after the experience acquired from several preliminary tests with dependent collisions.In the case of an inconsistent state,the users’scene rolls back to the last consistent state received from the remote user.Moreover, this scheme gives priority to the validation of interactions between objects grasped by users for fast haptic response.In the case of dependent collision,this schemefirst detects col-lisions between both grasped objects and chooses the valid positions for remote and local grasped objects(see[8]for details on the scheme).
3.2.2Scheme with assembly constraint detection
This new scheme is used when the system has already detected an assembly constraint between both objects.The scheme proposed in this section here corresponds to the case of pin–hole assembly.For the plane–plane assembly,the scheme can similarly be described.The following paragraphs describe the scheme in more er A(local user)moves the grasped object A(O A),and user B(remote user)moves the grasped object B(O B).
When the validation cycle starts,the following data are defined:
P AlastValid:last valid position of O A at user A.
P Anew:new position for O A at user A.Note that,this position has not been validated yet.
P Breceived:new position for O B(received from B)at user
A.Note that,it was validated at user
B.
P Areceived:one position of O A received from B.This posi-tion corresponds to a previous position of O A.
This position went from A to B and came from
B in the same message as P Breceived,so it was a
valid position.
The simulation thread uses these data to achieve consis-tency,that is,to select a new pair of valid positions for O A and O B.In general,the new pair will be P Anew and P Breceived,but sometimes this pair is not valid because both objects collide at such positions,so a different pair must be considered.Fur-thermore,this thread computes assembly constraints.This objective requires the following steps at user A.
O B is placed at P Breceived.Check that if O A placed at P Anew whether the remaining objects are colliding or not:
1.In the case of collision with an object different from O B:
O A is placed at P AlastValid.No messages are sent because P AlastValid was previously sent.
2.In the case of collision with O B and if the insertion of
O A into O B has not begun:Check if O A and O B collide at P AlastValid and P Breceived.
Collision:Note that the pair(P Areceived,P Breceived)is valid because only the valid pairs are sent.O A is placed at P Areceived.
No collision:No message is sent because P AlastValid was previously sent.The new valid pair is(P AlastValid, P Breceived).
3.Otherwise:
The pair(P Anew,P Breceived)is valid and the constraint movement is calculated to satisfy the assembly condition with O B.A new message is sent with new information.
268R.Iglesias et al.
4Experiments and results
A set of experiments were designed to evaluate consistency and the quality of force feedback.Although numerous exper-iments were carried out,in this section the most signifi-cant results are briefly reported.More detailed results are described in a report[43].
Three types of experiments have been considered:in labo-ratory,real trials between Labein(Bizkaia,Spain)and Queen’s University(Belfast,Northern Ireland)and a prelim-inary qualitative study was conducted withfive pairs of users. The results show that the proposed scheme maintains con-sistency.Furthermore,haptic rendering for dependent inter-action is efficient and compelling for undertaking assembly tasks with non-optimal network conditions.
This section is organized as follows.In Sect.4.1the VEs used for the experiments are described.Next,the labora-tory experimental platform and network conditions between Labein and Queen’s University are explained(Sects.4.2and 4.3).Section4.4outlines the results obtained to verify that consistency is successfully guaranteed.Haptic rendering in case of remote assembly and dependent collisions are studied (Sect.4.5).
4.1Virtual environments
Two VEs were used for the experiments.Figure3shows the first scene(handles and screws VE):a scene with different components of a CAD model.The dimension of this scene was270W×180H×150D mm.The other virtual envi-ronment is depicted in Fig.4,and it is the CAD design of an electrical box for an aircraft engine(electrical box VE) provided by SENER.This consisted of much more geometri-cally complex CAD components,which increased the valida-tion time of object movements.The dimension was500W×350H×270D mm.The handles and screws had the same size for both scenes.Each screw was approximately9mm base and20mm length and each handle was125mm length and26mm
wide.
Fig.3Handles and screws VE
In Figs.3and4the different models that the users grasped are also er S(User Queen’s)grasped the screw and User H(User Labein)grasped the handle.Dur-ing the experiments,each user was presented with the same virtual assembly scene,and each of them grasped a different object.The type of assembly conducted was pin–hole,that is, the screw was assembled into one of the holes of the handle.
4.2Experimental platform
An experimental platform was used for the laboratory exper-iments(Fig.5).It was composed of three co-located comput-ers via100Mbps links connected by two high-speed switches. Two computers were each connected to a PHANToM Omni device[45].The other computer was used to simulate the different values of delay and jitter[44].End-to-end delays of 1to100ms were tested and jitter values from1to100
ms. Fig.4Electrical box for an airplane engine
VE
Fig.5The experimental platform
Simultaneous remote haptic collaboration for assembling tasks269
4.3Network conditions between Labein and Queen’s
University Belfast
The Internet introduces both dynamically changing delays and the possibility of message loss.This negatively affects consistency and haptic rendering.The average round-trip delay between Labein and Queen’s University was53ms. Jitter was from0up to5ms and the number of hops was20. Figure6shows a user’s scene at Labein during a real trial with Queen’s University.
4.4Consistency:results
Consistency is required for the interaction to be useful and compelling,that is,to undertake successfully collaborative assembly tasks.This section shows that the consistency-maintenance scheme proposed(Sect.3.2)is adequate.Dur-ing the experiments,users were able to perform collisions between their grasped objects and moreover,they collabo-rated on assembly tasks(remote assemblies)and consistency was successfully
maintained.
Fig.6A user at Labein during a real trial with Queen’s University Belfast
Various experiments were carried out with the handles and screws VE varying end-to-end delays:less than1,50,100, 200ms and jitter using the experimental platform.Results for 200ms delay are presented in Fig.7(see[8,43]for exper-iments with lower delays).To assess how consistency is achieved with the consistency-maintenance scheme we mea-sured the displacement between an object’s position at one user’s scene and the last received position of the same object (e.g.the distance between the position P screw and P screw received).This displacement was called positional dis-crepancy.
These positions were measured and stored at user S once the simulation thread hadfinished.Therefore,the local updated position of screw was compared with the remote received position.In each experiment three consecutive phases were performed:independent interaction,dependent collisions and remote assembly.These phases are shown in thefigures separated by vertical lines.
Table1shows the mean and standard deviation of the posi-tional discrepancies measured in mm during the entire trial (1st column)and the three consecutive phases.As can be seen in the table,positional discrepancies are quite dependent on network delay.Moreover,it is shown that in all the exper-iments with different delays,discrepancies for dependent interaction are quite similar or smaller,than those measured for independent interaction.This means that the consistency scheme behaves satisfactorily.Furthermore,when dependent collisions occur,the system rapidly returns to a consistent state.When discussing Figs.7and8this becomes apparent for the higher delay experiment(200ms delay).In this test, collisions have been produced at a frequency that is much higher than that encountered in working conditions,to study the worst case.
Figure7a,b illustrates how the synchronization is achieved with200ms end-to-end delay with the handles and screws VE.Figure7a represents the different positions of the same object(screw)in the x-axis at both users(to simplify just positions along the x-axis are shown since it was the predom-inant movement in the undertaken task).Figure7a shows as if the screw position at user H were the same as at user S but
DELAY 200 ms - x position
150
a)
Fig.7X-position of screw S(left)and positional discrepancies of screw(right)at both users with handles and screws VE and200ms end-to-end delay。