Shin Nakajima 1

Image Guided Microscopic Surgery SystemUsing Mutual-Information Based RegistrationNobuhiko Hata1,2, William M. Wells III 1,3, Michael Halle 1,4Shin Nakajima1, Paul Viola3, Ron Kikinis1, Ferenc A. Jolesz11 Surgical Planning LaboratoryDepartment of RadiologyHarvard Medical School and Brigham and Women’s Hospital75 Francis Street, Boston MA 02115E-mail: noby@2 Faculty of Engineering, The University of Tokyo3 Artificial Intelligence Laboratory, Massachusetts Institute of Technology4 Media Laboratory, Massachusetts Institute of TechnologyAbstract. We have developed an image guided microscopic surgery systemfor the navigation of surgical procedures that can overlay renderings ofanatomical structures that are otherwise invisible. A new histogram-basedmutual information maximization technique was applied for alignment of thescope view and three-dimensional computer graphics model. This techniquedoesn’t require any pre-processing nor marker setting but is directly applied tothe microscope view and the graphics rendering. Therefore, any special setup in image scanning or preoperative preparation is not necessary.Graphics technique were implemented to compute three-dimensional sceneinformation by using a graphics accelerator, increasing the algorithm’sperformance significantly.Experiments are presented that demonstrate the approach registering aplastic skull to its three-dimensional reconstruction model generated from aCT scan. The tracking performance in the experiments were nearly real-time.1. IntroductionWith the advancement of medical imaging and computer technology, the effective use of diagnostic images has been a focus of discussion. One effective use of medical imaging is image-guided surgery, where information extracted from pre-operative imaging guides the surgical procedures. Previous applications have been presented by transferring medical image data or its three-dimensional (3D) reconstruction to the surgical field and registering it to the patient. Kelly, et al., [4] pioneered computer assisted frame-based and frameless stereotactic surgery, in which region of interest in the operative target can be localized and displayed using the preoperative images. This technique has contributed to a reduction of the invasiveness for the patient.The surgical microscope is an attractive vehicle for such applications. Roberts, et al., [6] reported the development of a surgical microscope guided by pre-operative CT by using an ultrasonic tracking sensor. The outlines of a tumor were projectedinto the optics of the microscope. Edwards, et al., [2] presented a stereo operating microscope that projects stereo projections of 3D model into the microscope eyepieces.All the techniques listed above require alignment of patient and preoperative image to perform intraoperative navigation based on preoperative image. Some applications benefit from surgical instruments or pointing devices but these require additional registration. Edward, et al., used an optical tracking sensor to register the microscope and patient in the operating room. The position and orientation of the microscope was tracked by the LED markers attached on the microscope. Patient registration was performed by identifying fiducials or anatomical landmarks on the skin surface. This process is reported to take 1-2 minutes and to extend the duration of the procedure.Ideally, the method of registration should be non-invasive for the patient and convenient for medical staff.In this paper, we describe preliminary experiments with an image guided microscopic surgery system that navigates surgical procedures by overlaying an 3D rendering in the microscopic view. A new histogram-based mutual information maximization technique was applied for alignment of scope view and three-dimensional computer graphics model. This technique doesn’t require any pre-processing nor marker setting but is directly applied to the microscope view and graphics rendering. Therefore, no special set up in image scanning or preoperative preparation is not necessary.In addition, the implementation of the graphics techniques used with method is also presented. The general problem is to compute the 3D scene information needed by the registration with the use of a graphics accelerator.2. Material and Methods2.1 Registration by Maximization of Mutual Information“Registration by maximization of mutual information” is applied to a scope view -model registration.As the method is applied to images directly without any preprocessing, the method is quite general. Therefore, it can be applied for a wide variety of sensors. In previous studies, the method was applied to register medical images of different modalities, 3D model to images and 2D view-based images to images.Some previous work on image guided surgery has used feature points or markers to register the patient to preoperative medical images and operative tools to preoperative images. The additional preparations and procedures needed for their use are cumbersome not only for patient but also for medical the staff. As the presented method is applied directly to the 3D models and medical images, it is especially suitable for application in video image guided surgery, such as endoscopic surgery, and microscopic surgery.The detailed description of registration by maximization of mutual information appears in [8][9] and [10].In this application, x represents the location of a surface patch, T is the transformation placing the object with respect to the camera, u is the surface normal of a patch at the object, and v is the observed image. The general problem of registration is formulated as,arg max ((),(()))T I u x v T x T =(1),where I represents mutual information calculated between u and v . Mutual information is defined in terms of entropy in the following way:I u x v T x H u x H v T x H u x v T x ((),(()))(())((()))((),(()))≡+−(2),where H ()⋅is entropy of a random variable, and can be interpreted as a measure of uncertainty, variability or complexity.The first term on the right is the entropy in the model. The second term is the entropy of the part of the image into which the model projects. It encourages transformations that project u into complex parts of v . The third term, the(negative)joint entropy of u and v , takes on large values if u and v are functionally related. It encourages transformation where v explains u well. Together the last two terms identify transformations that find complexity and explain it well.2.2 Mutual Information with HistogramIn this section, we describe a histogram-based computation of mutual information.The entropy of random variable x is described as an expectation of the negative logarithm of the probability density. In [8][9] and [10], the Parzen Window method is used to approximate the underlying probability density f z () by a superposition of Gaussian densities centered on the elements of a sample B drawn from z .In this study, we used a histogram for density estimation and smoothed finite differences of the histograms for estimating derivatives of density. Collignon et al.,[1] used histograms for computing entropies and mutual information, in a non-derivative optimization using Powell’s method. In experiments, we have found significant speed improvements using a simple gradient search model. We can approximate the entropy of random variable z :ln (,)H N p z B i z A i ≡−∈∑1,(3)where A denotes a sample of observations of z and B denotes second sample of observations of z . The derivative of ()Hz is computed for the usage in finding its local maximal. We compute the derivative of Hfor a gradient search,dH z dT d dT N p z B N d dT p z B pz B N d dz p z B dz dT d dB p z B dB dT p z B iz A i i z A ii i i i z A i i i ()ln( (,)) (,) (,) (,) (,) (,)≈−=−=−+∈∈∈∑∑∑111.(4).The numerator of the derivative has two components. The first component is the change in entropy that results from changes in z i . The second one is a measure of the change in the histogram density estimate that results from changes in the sampleB . For histograms computed from a finite sample, a differential change in T will move a small number of sample points from one bin to another. As a result, we will assume that the second term is zero.2.3 Maximization of Mutual InformationIn order to maximize the mutual information described in Equation (1), we approximate its derivative of it with respect to transformation matrix T by implementing Equation (4),d dT I T I v dv dT i v A i i () =∈∑∂∂(5),where A is the intensities of the image pixels that correspond to visible surface patches of the model.Steps are repeatedly taken that are proportional to the approximation of the derivative of the mutual information with respect to the transformation,T T dIdT ←+λ(6)where λ is a step size parameter. The steps are repeated a fixed number of times or until convergence is detected.2.4 Density Estimation with Graphics TechniquesA computer graphics technique is applied to increase the performance of the registration computation.In order to calculate the histograms of v and u in I T ()of Equation 1 and its derivative in Equation 5, a computer graphics technique is applied. With the helpof high performance graphics implemented in acceleration a workstation, computational efficiency is increased.These equipments were all developed and performed using Ultra 2 with Creator 3D graphics accelerator (SUN Microsystems). In the software development, the Visualization Tool Kit (VTK) [7] is implemented for graphics manipulation and XIL (SUN Microsystems) for video capturing. The VTK stands upper level of the XGL graphics library and controls data loading, rendering and event handling.The registration algorithm requires the calculation of surface normals for the model at the current position and orientation. These can be performed by manipulating the placement of light sources. Given the view camera for rendering is placed at (0,0,+a[>0]) in the world coordinate system and model at (0,0,0), visible surface patch on the model has all positive z directional normal.Two of the reflectance component, ambient and diffuse contribution from the light, are activated and the other component, specular contribution, is inactivated. Activated components have ambient reflection coefficient 0.5 and diffuse coefficient 0.5. A directional light on (+a, 0, 0) has diffuse reflection effect according to its x directional normal value. In the same way, a light on the y axis has diffuse reflection effect to according to y directional lighting value.To achieve the desired reflectance, a positive green light is placed at (+a,0,0), a negative green light at (-a,0,0), a positive blue light at (0,+a,0) and a negative blue at (0,-a,0). The positive light has a RGB color component 1.0 out of its range 0.0 to 1.0, and negative light has -1.0. In addition to setting directional lights, we set the ambient color of the object blue-green using this scheme, the green component of color represents its x directional normal value and blue represents y directional normal value.This scheme requires placement of negative light sources. In our preliminary experiments, we have verified that XGL and OpenGL can accommodate negative light sources.Another advantage is that surface normal calculation and user interface can share its graphics property: geometry data storage, position and orientation matrices, pipeline with graphics accelerator, window and event management. By sharing most of this functionality with the already existing rendering platform, development becomes simpler and faster.Copying the rendered image in the raster memory to conventional memory requires time and that influences the performance seriously. Therefore, rendering the window size for the computation of normal was minimized.2.5 DisplayAfter the registration is performed, the rendered image of the 3D model is transferred to the microscope though a VGA signal. The microscope has a monochrome display that can be merged to the right eyepiece. Although 3D model is aligned to match the microscope view, additional scaling and translation are required to be displayed on VGA display in the scope.After the registration with visible anatomical structure is performed, normally invisible internal structure is displayed for surgical navigation. Visible structure may be turned off for better perception of internal objects.3. Results3.1 System ConfigurationThe system consists of a microscope for surgical use (ZEISS) and a workstation (Ultra SPARC 2 with Creator3D, Sun Microsystems, Moutainview, CA) (Fig.1). The Microscope has video (NTSC) output for both right and left microscope view. The video output from the right scope is transferred to the workstation and digitized with video capture board (SUN Video) at the resolution of 600 x 480 [pixels].After the registration of 3D model with microscope view is performed, the rendering image of the aligned 3D model is transferred to the microscope as a VGA signal. The microscope has monochrome VGA display in the right scope that can be overlaid on the conventional microscope view.3.2 Tracking ExperimentWe performed an experimentation to evaluate the tracking performance of the algorithm. The iteration in Equation (6) is used to perform plastic skull tracking under the configuration described in Section 3.1.A plastic skull was used as an phantom. 3D model around the right auditory hole of the plastic skull is generated from its CT scan (Siemens Somatom Plus, 512x512[pixels]x274[slices], 0.98x0.98x0.92[mm/voxel]) followed by iso-surface processing[5] and surface smoothing. This experiment also aims the simulation of skull vase surgery that has difficulty in localizing nerves around auditory hole.The plastic skull was placed under the microscope and lighted up by a light attached to the microscope. The microscopic view had an area of 54.5x 40.9 [mm] at its focal plane. An initial alignment to an images was performed manually. Then the random offset is added to each transitional axis and randomly selected one rotational axis.The Table 1 shows the result of each experiment with selected range of offset. In each experiment, 50 uniformly randomized offsets lower than the maximal offsets are given as an initial guess. Each trial was terminated either when the number of iteration reached 30 or the convergence was detected.Additionally, Figure 2 shows an example of tracking result. This convergence required 21 iteration with about 10 seconds.Table 1. Result of Phantom ExperimentMaximal Offset Initial FinalTranslation[mm]Rotation[deg.]Translation[mm]Rotation[deg.]Translation[mm]Rotation[deg.]Experiment 115.020.0 5.211.60.87.9 Experiment 225.020.010.08.2 2.48.2 Experiment 315.040.07.918.8 2.38.7 The experiment demonstrated that the algorithm performs efficiently and reliably. If the initial guess is placed approximately 15 [mm] and 20 [degrees] away from the result, the model can be aligned correctly within 30 iterations.4. DiscussionThis system has been designed to overcome the difficulties of visibility and visualization that can occur during microsurgery.A number of factors can cause a surgeon to have difficulties orienting the current microscope view to the anatomical structure of the patient and to previously acquired 3D models. Among these are motions of the microscope, limitations on the field of view, and other obstructions to visibility, such as bleeding.These difficulties in visual identification and orientation may be directly addressed by the superimposition in the microscope of correctly registered renderings of 3D anatomical structures. This capability is also useful for determining the location of otherwise invisible structures, for example, if a surgeon can detect a nerve inside a bone, it will be very useful while drilling the bone. Similarly, it is also helpful to visualize the position of blood vessels in a tumor during a resection procedure. Although the CT of today can scan with nearly 1mm in gap and less in pixel size, the microscope provide more precise information. However, the scope used in the experiment has 54.5x40.9[mm] of widest view field, which means the size of the pixel in the digitized and overlaid image (640x480 [pixels]) is 0.08x0.08[mm]. This mismatch of preciseness between microscopic view and 3D model may causes fatal error in registration: the gradient searching cannot escape from a local maxima. The microscope may have additional wider setting for initial registration to restrict the translation and orientation of the model in the following registration procedure. For a scope view - model registration, “Registration by maximization of mutual information” is applied. In this method, the registration of 3D model and video (microscope) view can be achieved by maximizing the mutual information. One advantage of this method is that it applies directly to the video image and 3D model. In other words, no pre-processing and feature detection is not required. Therefore, surgical procedures should not need to be interrupted for registration processing and surgeons can concentrate on their operative maneuvers. Additionally, the system canbe set up by adding workstation with functions of graphics acceleration and video input to the operative microscope. Thus the system is easy to be implemented to a conventional microscope which usually equipped with a video output and sometimes with a video input.As the registration method only requires a video image and a 3D model, its prospective field of application in medicine potentially broad. Endoscopic and Laparoscopic surgery, where conventional feature based registration is hard to apply, may benefit from the simplicity and efficiency of our technology.5. AcknowledgmentWe are grateful to the authors of “The Visualization Tool Kit”, Drs. Will Schroeder, Ken Martin and Bill Lorensen for their technical assistance. We also wish to thank the colleagues in Surgical Planning Laboratory of Brigham and Women’s Hospital, and Artificial Intelligence Laboratory of MIT. The assistance and advice of Prof. Dohi of the University of Tokyo is gratefully appreciated.6. References1. Collignon A, et al.: Automated multi-modality image registration based oninformation theory. Proc. Information Processing in Medical Imaging Conf., Kluwer Academic Publishers: 263-274, 19952. Edwards PJ, Hawkes DJ, Hill DLG, Jewell D, Spink R, Strong A, Gleeson M:Augumented realty in the stereo operating microscope for optolaryngology and neurosurgical guidance, Proc. CVRMed 95, 19953. Friets EM, Strohbehn JW, Hatch JF, Roberts DW: A frameless stereotaxicoperating microscope for surgery. IEEE Trans Biomed Eng. 36 (1989) 608-6174. Kelly PJ: Vlumetric stereotaxis and computer-assisted stereotactic resection ofsubcortical lesions, in Lunsford LD (ed): Modern Stereotactic Neurosurgery.Boston, Marinus Nijhoff, (1988) 169-1845. Lorensen WE and Cline HE: Marching cubes: A high resolution 3D surfaceconstruction algorithm. Computer Graphics 21 (1987) 163-1696. Roberts DW, Strohbehn JW, Hatch JF, et al: A frameless stereotaxicintegration of computerized tomographic imaging and the operating microscope. J Neurosurg 65 (1986) 545-5497. Schroeder W, Martin K, Lorensen B: The visualization toolkit : an object-oriented approach to 3D graphics, Prentice-Hall, New Jersy, (1996)8. Viola PA and Wells WM: Alignment by maximization of mutual information.Proc. 5th Intl. Conf. Computer Vision (1995)9. Viola PA: Alignment by Maximization of Mutual Information, Ph. D. thesis,Massachusetts Institute of Technology, (1995)10. Wells WM, Viola PA and Kikinis R: Mutti-Model Volume Registration byMaximization of Mutual Information. Proc 2nd Med. Robotics Comp. Assisted Surg., (1995)。