2013A Fast Restoration Method for Atmospheric

A Fast Restoration Method for Atmospheric Turbulence Degraded Images Using Non-RigidImage RegistrationKalyan Kumar Halder ,Murat Tahtali,and Sreenatha G.AnavattiSchool of Engineering and Information TechnologyThe University of New South WalesCanberra,ACT2600,AustraliaEmail:k.halder@.auAbstract—In this paper,a fast image restoration method is proposed to restore the true image from an atmospheric turbu-lence degraded video.A non-rigid image registration algorithm is employed to register all the frames of the video to a reference frame and determine the shift maps.The First Register Then Average And Subtract-variant(FRTAASv)method is applied to correct the geometric distortion of the reference frame.A performance comparison is presented between the proposed restoration method and the earlier Minimum Sum of Squared Differences(MSSD)image registration based FRTAASv method, in terms of processing time and accuracy.Simulation results show that the proposed method requires shorter processing time to achieve the same geometric accuracy.Keywords—atmospheric turbulence;computational time;image registration;image restoration;shift mapsI.I NTRODUCTIONIn long-distance imaging systems,the prevailing effects of atmospheric turbulence comprise random geometric distortions as well as non-uniform image blurring[1]–[3].The image degradation effects arise from random inhomogeneities in the temperature distribution of the atmosphere,causing variations of refractive index along the optical transmission path which are the most prominent near to the ground[3]–[5].These cause significant effects in thefields of surveillance and astronomy where the undisturbed image is extremely important[6],[7]. Imaging through the atmosphere involves the restoration of an image as close as possible to the true one from a turbulence degraded image or video.Over the last few decades,several image processing techniques have been proposed for the com-pensation of blurring effects.A few methods have also been proposed for the geometric corrections.Yitzhaky et al.[8]propose a deconvolution technique to restore the blurred image using the Modulation Transfer Function(MTF)and knowledge statistics.It needs the weather information and deals only with the blurring problem.In [9],a blind image deconvolution approach is proposed using independent component analysis and genetic algorithms.The image processing is done in the frequency domain,therefore computationally expensive.Huebner and Greco[3]present a performance comparison of four blind deconvolution al-gorithms that they applied to the restoration of turbulence degraded images.The methods are:linear Inverse Wiener Filter (IWF),non-linear Lucky-Richardson Deconvolution(LRD),Iterative Blind Deconvolution(IBD),and deconvolution using Principle Component Analysis(PCA).Among these methods, PCA based blind deconvolution provides better results,espe-cially shorter processing time and more robustness to noise[4].In[6],the authors propose a blur identification and imagerestoration method by minimizing Second-Order Central Mo-ment(SOCM)of images.A lucky imaging system is proposed in[7]to restore turbulence degraded astronomical images.In[10],an iterative maximum-likelihood-estimation algorithm isproposed for reconstruction of superresolved images.Further modification is needed for the above methods in order to improve compensation for image dancing.Several image restoration methods greatly depend on im-age registration which computes geometric deformation using the turbulence degraded frames.Real-time implementation of restoration methods requires a faster and accurate image registration technique.There are several image registration algorithms such as differential elastic image registration,B-spline based non-rigid image registration,gradient based op-ticalflow,MSSD cross-correlation.The FRTAAS method in[1],[2]is proposed for the restoration of non-uniformly warpedimages based on differential elastic registration.FRTAAS,in conjunction with differential elastic registration,provides very accurate and stable results,though it takes several hours for processing.In[5],an image restoration method is described using non-rigid image registration and Bayesian reconstruction framework.It needs several minutes for each restoration.Mao and Gilles[11]propose a novel approach for image restoration based on a variational model solved by Bregman Iterations and the operator splitting method.A gradient technique for opticalflow estimation is described in[12].In[13],a statis-tical method is proposed forfiltering atmospheric turbulence degraded sequences.This method provides better geometric correction than the FRTAAS method using differential elastic registration.These methods are also slower and not suitable for near-real time implementation.A fast FRTAASv image restora-tion method based on MSSD image registration is implemented in[14],but the limitation of using MSSD algorithm is the loss of sub-pixel accuracy in pixel registration.In this work,the FRTAASv method based on non-rigid image registration is proposed to restore a true image from atmospheric turbulence degraded video.The performance of the proposed method is compared with the MSSD registration based image restoration method in terms of computational time 394978-1-4673-6217-7/13/$31.00c 2013I EEEand accuracy.The rest of the paper is organized as follows:Section II presents an insight into the non-rigid image registration algorithm proposed in this work.This section also describes the existing MSSD image registration technique.Section III describes the FRTAASv image restoration method.Simulation experiments are included in Section IV .Finally,Section V concludes the paper.II.I MAGE R EGISTRATIONImage registration is often used to match two or more im-ages of the same scene taken at different times,from differentviewpoints,and/or from different sensors.It determines the geometrical transformation that aligns points in one view of an image with corresponding points in another view of that image.For the purpose of restoration of geometric deformations,it is useful to have the shift maps in the form of backward mapping rather than forward mapping [14].A.Non-Rigid Image RegistrationIn general,a non-rigid image registration algorithm can be resolved into three components [15],[16]:•the transformation model which specifies the geomet-ric transformation between the source and reference images.•the similarity metric which measures the degree of alignment between the source and reference images.•the optimization method that varies the parameters of the transformation model to maximize the similarity measure.Assuming that F represents a reference image and G denotes a source image,and they are related by [17]:F (p )=G T (p )+S +η,(1)where p =(x,y,t ),S is an intensity correction field,ηis zero mean Gaussian noise,and T (p )is the geometric transformation that aligns F and G .Then T (p )can be defined by [18]T (p )=p +w (p ),(2)where the shift map w (p )= x s (p ),y s (p ),1.The matrices x s (p )and y s (p ),also called deformation parameters,are the horizontal and vertical components of the shift maps,respectively.According to the gray value constancy assumption,the gray value of a moving pixel should be consistent over time and the shift maps should be piecewise smooth [19].This results in the following objective function [17]:E T (p ) = F(p)−G T (p ) 2,(3)whereF andG are in column-vector form,and · is theEuclidean norm.Considering that r (p )=F (p )−G T (p ) is the residual vector (difference image)that explicitly depends on the transformation T (p ),substituting this value into the objective function (3),we obtain:E T (p )= r (p ) 2.(4)The goal is to estimate the deformation parameters by maximizing the matching criterion.In the B-spline based registration approach,the deformation vector can be estimated by minimizing the following cost function:E T (p ) =pr (p ) 2.(5)The simplest statistical measure of image similarity is based on the Sum of Squared Differences (SSD)between images F and G .The SSD can be measured as [15]:SSD =1nF (p )−G T (p ) 2,(6)where n is the number of pixels in the region of overlap.This measure is based on the assumption that both imaging modalities have the same characteristics.If the images are correctly aligned they only differ by Gaussian noise.The transformation T (p )is modeled using the Free Form Deformation (FFD)transformation with three hierarchical lev-els of B-spline control points [17].The gradient descent opti-mization method is applied to iteratively update the transfor-mation parameters.The method requires the estimation of the similarity measure’s gradient with respect to the parameters.The number of iterations is chosen to achieve an acceptable registration accuracy.B.MSSD Image RegistrationThe MSSD cross-correlation technique typically produces shift maps by calculating SSD at each pixel within a neighbor-hood.This is achieved by taking a square window of certain size around the pixel of interest in the reference frame and finding the homologous pixel within the window in the source frame,while moving along the corresponding scanline.Fig.1shows a typical search block of 3×3pixels in a search field of 9×9in a window of 11×11.The bold traced block is located at the zero shifts with respect to the search window,whereas the dashed block is located at shifts -4and -4pixels in the x −and y −directions.The target is to find the corresponding (correlated)pixel that minimizes the associated error and hence determine the x −and y −shifts.At last,the shift maps are run through median filters to soften the outliers[14].Fig.1.Search block (bold and dashed outlines)and search field (shaded)for MSSD image registration.2013I nt er nati o nal Co n fere n ce o n Ad v an ce s in Co m pu ting ,Co mm u ni c ati o ns and I n for mati c s (IC A CCI)395III.I MAGE R ESTORATION A LGORITHMLet a turbulence degraded video consist of N warped frames.In the FRTAASv method,the first frame is considered as the reference frame.This method considers that the average wander of each pixel of a static scenery over a long enough period of time is zero.Therefore,the average wander of a pixel with respect to any fixed point should yield its true position [14].The backward mapping pixel registration is run for each warped frame to obtain the shift maps x s (x,y,t )and y s (x,y,t ).By using the shift maps,the centroids,C x and C y ,which are used to calculate the pixel locations of the original frame,are calculated by averaging:C x (x,y )=1N Nt =1x s (x,y,t )C y (x,y )=1N Nt =1y s (x,y,t ).(7)The inverse of C x and C y are then calculated asC −1x(x,y )=−C x x −C x (x,y ),y −C y (x,y ) C −1y(x,y )=−C y x −C x (x,y ),y −C y (x,y ) .(8)Using these inverse of centroids,one is therefore able to obtainF ∗(x,y,t )=F x +C −1x (x,y ),y +C −1y (x,y ),t ,(9)where F ∗(x,y,t )is the restored version of the warped refer-ence frame at time t .In real-time processing,the centroids areaccumulated progressively with each new frame and individual shift maps are discarded after they are used to calculate the centroids.IV.S IMULATION E XPERIMENTSThe proposed method was implemented in MATLAB and tested on an Intel Core i7-2600CPU 3.40GHz machine with 8GB RAM.At first,two video sequences of 80warped frames were generated using the Lena (512×512pixels)and Theophilus (512×512pixels)test images to compare the proposed method with the MSSD registration based FRTAASv method.Gaussian noise was added to the both sequences resulting in Signal-to-Noise Ratio (SNR)26.13dB for Lena sequence and 26.04dB for Theophilus ter on,the method was applied on a real-life video.A.Per f ormance Analysis and C om p arisonFig.2(a)and (b)show the original Lena image and the reference frame,respectively.The restored versions of the reference frame using the MSSD based FRTAASv method and the proposed method are shown in Fig.2(c)and (d),respectively.Fig.2(e)and (f)show the difference between original and restored frames for the two methods.For the pixel registration of a pair of Lena frames and dewarping,it takes about 196.9s for the MSSD based method and 19.5s for the proposed method.Similarly,Fig.3(a)and (b)show the true Theophilus image and the reference frame,respectively.The restored Theophilus frames using the MSSD based FRTAASv method and the proposed method are shown in Fig.3(c)and (d).Difference images for the two methods are also presented in Fig.3(e)and (f).It takes approximately 206.1s and19.9s(a)(b)(c)(d)(e)(f)Fig.2.(a)Original Lena image,(b)reference frame,(c)restored frame using MSSD registration based FRTAASv method,(d)restored frame using proposed method,(e)intensity difference between (a)and (c),and (f)intensity difference between (a)and (d).for the MSSD based method and the proposed method for the single registration and dewarping of each Theophilus frame.The proposed image restoration method is nearly 10times faster than the earlier MSSD based method for 512×512images.Also,visual comparison reveals that the difference image for the proposed method is darker than those using MSSD based method for the both video sequences,indicating lower residuals.The intensity Mean Square Error (MSE)of the warped frames and the reference frames restored again after each new frame are calculated with respect to the true images.Fig.4shows a frame by frame MSE comparison between the proposed non-rigid registration and MSSD registration based restoration for the Lena video sequence.The plot gets better3962013I n t er na t i o nal Co n fere n ce o n Ad v an ce s in Co m pu t in g ,Co mm u ni c a t i o ns and I n for ma t i c s (IC A CCI)(a)(b)(c)(d)(e)(f)Fig.3.(a)Original Theophilus image,(b)reference frame,(c)restored frame using MSSD registration based FRTAASv method,(d)restored frame using proposed method,(e)intensity difference between (a)and (c),and (f)intensity difference between (a)and (d).as the centroids settle down.The MSE values of the restored reference frames are 230.12and 173.11for the MSSD based method and the proposed method,respectively.Fig.5shows similar comparison for the Theophilus video sequence and the MSE values for the MSSD and non-rigid registration based restoration are 51.19and 51.67,respectively.It is clearly evident that the proposed method yields lower MSE for the Lena sequence,but has no clear advantage with the Theophilus sequence.Beside this,the proposed method is capable of providing better geometric corrections of the restored frame by increasing the number of iterations,i.e.the run time.The performances of the methods are also compared by their restoration capability under noisy conditions.Fig.6and 7show the intensity MSE plots for variation of SNR for theLena and Theophilus sequences,respectively.The performance of the restoration expectedly degrades with increasing noise,i.e.with decreasing SNR.It is apparent that the proposed method works somewhat better than the MSSD registration based FRTAASv method under noisy environments.Fig.4.Intensity MSE plot of warped,restored frames using MSSD and non-rigid registration based FRTAASv methods for Lena sequence.Fig.5.Intensity MSE plot of warped,restored frames using MSSD and non-rigid registration based FRTAASv methods for Theophilus sequence.Fig.6.Intensity MSE of restored frame versus SNR for Lena sequence.2013I n t er na t i o nal Co n fere n ce o n Ad v an ce s in Co m pu t in g ,Co mm u ni c a t i o ns and I n for ma t i c s (IC A CCI)397Fig.7.Intensity MSE of restored frame versus SNR for Theophilus sequence.(a)(b)(c)(d)Fig.8.(a)Time averaged Hillhouse frame,(b)reference frame,(c)restoredframe using MSSD registration based FRTAASv method,and(d)restoredframe using proposed method.B.A pp lication on a Real-Li f e VideoAfter having good restoration performance of the proposedmethod on synthetic warped and noisy video sequences weapplied it on a real surveillance Hillhouse video sequence.Inthe case of the real video,since no truth image is availablethe restored frames cannot be compared with the original.TheHillhouse sequence consists of75warped and noisy frameseach of512×512pixels.Fig.8(a)and(b)show the timeaveraged frame and the reference frame,respectively.Therestored versions of the reference frame for the MSSD basedmethod and the proposed method are presented in Fig.8(c)Fig.9.Intensity MSE between the registered version of each frame and thereference frame using MSSD and non-rigid image registration techniques forHillhouse sequence.and(d).The intensity MSE is calculated for the registeredversion of each frame with respect to the reference frame andis presented in Fig.9.It is clearly evident that the proposedmethod yields somewhat lower MSE than the MSSD basedmethod consistently.V.C ONCLUSIONWe successfully implemented the FRTAASv image restora-tion method using a non-rigid image registration algorithm forgeometric corrections of non-uniformly warped sequences.Wealso compared the results to another recent algorithm proposedto compensate for geometric distortions.It was verified thatusing the proposed method a faster restoration of the referenceframe is possible while still being comparable in terms ofMSE.Further work will involve implementation of the methodon GPU for real-time operation.R EFERENCES[1]M.Tahtali,D.Fraser,and mbert,“Restoration of non-uniformlywarped images using a typical frame as prototype,”in Proc.TEN C ON,2005,pp.1–6.[2]M.Tahtali,mbert,and D.Fraser,“Restoration of nonuniformlywarped images using accurate frame by frame shiftmap accumulation,”in Proc.SPIE6316,Image Reconstr u ction f rom Incom p lete Data IV,2006.[3] C.S.Huebner and M.Greco,“Blind deconvolution algorithms forthe restoration of atmospherically degraded imagery:a comparativeanalysis,”in Proc.SPIE7108,O p tics in Atmos p heric Pro p agation andAda p ti v e Systems XI,2008.[4] D.Li,R.M.Mersereau,and S.Simske,“Atmospheric turbulence-degraded image restoration using principal components analysis,”IEEEGeoscience and Remote Sensing Letters,vol.4,no.3,pp.340–344,July2007.[5]X.Zhu and anfar,“Image reconstruction from videos distortedby atmospheric turbulence,”in Proc.SPIE7543,Vis u al In f ormationProcessing and C omm u nication,2010.[6]L.Yan,M.Jin,H.Fang,H.Liu,and T.Zhang,“Atmospheric-turbulence-degraded astronomical image restoration by minimizingsecond-order central moment,”IEEE Geoscience and Remote SensingLetters,vol.9,no.4,pp.672–676,July2012.[7]S.Zhang,Y.Wu,J.Zhao,and J.Wang,“Astronomical imagerestoration through atmosphere turbulence by lucky imaging,”in Proc.International C on f erence on Digital Image Processing(I C DIP),2011. 3982013I n t er na t i o nal Co n fere n ce o n Ad v an ce s in Co m pu t in g,Co mm u ni c a t i o ns and I n for ma t i c s(IC A CCI)[8]Y.Yitzhaky,I.Dror,and N.S.Kopeika,“Restoration of atmosphericallyblurred images according to weather-predicted atmospheric modulation transfer functions,”O p tical Engineering,vol.36,no.11,pp.3064–3072,Nov.1997.[9]H.Yin and I.Hussain,“Blind source separation and genetic algorithmfor image restoration,”in Proc.International C on f erence on Ad v ances in S p ace Technologies,2006,pp.167–172.[10] D.R.Gerwe and M.A.Plonus,“Superresolved image reconstruction ofimages taken through the turbulent atmosphere,”Jo u rnal o f the O p tical Society o f America A,vol.15,no.10,pp.2620–2628,Oct.1998. [11]Y.Mao and J.Gilles,“Non rigid geometric distortions correction-application to atmospheric turbulence stabilization,”In v erse Problems and Imaging,vol.6,no.3,pp.531–546,2012.[12] D.Clyde,I.Scott-Fleming,D.Fraser,and mbert,“Application ofopticalflow techniques in the restoration of non-uniformly warped im-ages,”in Proc.International C on f erence on Digital Image C om pu ting: Techniq u es and A pp lications(DI C TA),2002,pp.195–200.[13]R.Abdoola,B.Wyk,and E.Monacelli,“A simple statistical algorithmfor the correction of atmospheric turbulence degraded sequences,”in Proc.Ann u al Sym p osi u m o f the Pattern Recognition Association o f So u th A f rica,2010.[14]M.Tahtali,mbert,and D.Fraser,“Graphics processing unitrestoration of non-uniformly warped images using a typical frame as prototype,”in Proc.SPIE7800,Image Reconstr u ction f rom Incom p lete Data VI,2010.[15]W.R.Crum,T.Hartkens,and D.L.G.Hill,“Non-rigid imageregistration:theory and practice,”The British Jo u rnal o f Radiology, vol.77,pp.140–153,2004.[16] D.Rueckert and P.Aljabar,“Nonrigid registration of medical images:theory,methods,and applications,”IEEE Signal Processing Magazine, vol.27,no.4,pp.113–119,July2010.[17] A.Myronenko and X.Song,“Intensity-based image registrationby minimizing residual complexity,”IEEE Transactions on Medical Imaging,vol.29,no.11,pp.1882–1891,Nov.2010.[18]S.Y.Chun and J.A.Fessler,“A simple regularizer for B-spline nonrigidimage registration that encourages local invertibility,”IEEE Jo u rnal o f Selected To p ics in Signal Processing,vol.3,no.1,pp.159–169,Feb.2009.[19]T.Brox,A.Bruhn,N.Papenberg,and J.Weickert,“High accuracyopticalflow estimation based on a theory for warping,”in Proc.E u ro p ean C on f erence on C om pu ter Vision(E CC V),2004.2013I n t er na t i o nal Co n fere n ce o n Ad v an ce s in Co m pu t in g,Co mm u ni c a t i o ns and I n for ma t i c s(IC A CCI)399。