立体信息检测苹果表面缺陷(平行的结构光)

Q .Y A N G230 T he proposed system has been tested with samplea pples and the test results are presented .2.S egmentation of dark patchesD ark patches in apple images represent boths tems / c alyxes and patch-like blemishes , such asb ruises , wounds and many rots . They appear asc onnected regions of relatively small size and havev ariable shape and contrast . An example image iss hown in F ig .1 . When viewing the intensity image asa topographic relief , the patches appear as concavea reas in the grey-level surface .Therefore , they can bet reated as catchment basins in grey-level landscapes .B ased on this , a region-based segmentationa lgorithm , 5 called a flooding algorithm , has beend eveloped to segment the patches .Here , we explaint he underlying concepts of the algorithm in terms ofd igital elevation models ; the implementation detailsa re described in reference 5 .T he flooding algorithm is based on the concept ofw ater catchment basins . A catchment basin is a regiona ssociated with a minimum . To define the spatiale xtent of the region around the minimum , we considert he properties of water flowing over a topographicr elief . Water will always flow downhill until a localm inimum is reached . As more water is added ,ita ccumulates at the lowest points and gradually coverst he region surrounding the minimum , until it reaches ac ertain level at which the basin can no longer hold anym ore water and the surplus water spills over thel owest points of the basins’ rim .Thus , the basinb ecomes a lake .The corresponding elevation level ofF ig . 1 . A n example apple image with a blemish and a stema rea t he lake’s surface is called the spillover level .The s pillover level is used to delimit the spatial extent of a c atchment basin .Therefore , a catchment basin con- s ists of those points , which are connected to a m inimum and have a grey-level lower than the spillo-v er level . To detect all the dark patches ,we gradually flood the relief with water , all the catchment basins w ill be filled up and become lakes of various sizes and d epths . The lakes , which are equivalent to catchmentb asins , are the output of the flooding algorithm and r epresent the patches being detected . T he concept of a lake is extremely useful in that it h olds most of the characteristics desired in the follow-i ng classification . For instance , the surface area of a l ake is a clear indicator of the size of the detected p atch ;the average depth , i . e . the ratio of the water v olume to the lake surface area , is a measure mostly o f contrast , but contains gradient information as well . A nother obvious advantage of this presentation is that t he surface of each individual lake is always con-n ected . The segmentation result of the example image i n F ig .1 is shown in F ig .2 . The detected patches are h ighlighted with bright borders . T he algorithm has no tuning parameters , and the s egmentation output depends purely on image data . T here is little need for thresholding . It has no strong a ssumption about the contrast and shape of patches . T he sole working assumption used is that the patches a re not adjacent to the boundary of the fruit in i mages . Apart from the segmentation output ,some d escriptive parameters such as the minimum height ,c urrent volume and depth of the basins can also be c omputed as the flooding proceeds , and they are a vailable as outputs of the procedure .Once the F ig . 2 . R esults of patch segmentation for the image in Fig . 1 . D etected patches are highlighted by bright bordersA P P L E S T E M A N D C A L Y X I D E N T I F I C A T I O N W I T H M A C H I N E V I S I O N231p atches are segmented out,measurements of the c orresponding lakes can be made to extract other g eometric parameters.These parameters quantify the c haracteristics of each patch under consideration and a re ready to be used as inputs to the following c lassification step.3.T hree-dimensional shape information froms tructured lightT he surface of an apple can be divided roughly, a ccording to shape,into two basic parts,convex parts a nd concave parts.Most of the surface is convex and s ometimes nearly spherical.The concave parts are u sually in the areas around the stem and calyx.The s tem itself stands out from the middle of the concave a rea.On the other hand,blemishes,such as bruises a nd abrasions often occur on the convex surface of a f ruit.Therefore,the three-dimensional information a bout the dif f erence in surface shape can assist iden-t ification of the stem and calyx.A technique called l ight striping6i s used,in which the projected light p lanes and a camera make up an active stereo system s o that three-dimensional geometric information ab-o ut an object surface may be derived from optical t riangulation.F or fast processing,a set of evenly spaced parallel l ight stripes are projected onto the apple surfaces s imultaneously.An image with the stripes is grabbed f or each view of a fruit.This stripe image is obtained w ith the same camera as for the normal apple image w ithout the structured light.In stripe images,the s tripes on dif f erent parts of the surface have dif f erentF ig .3.A n example image with light stripes on the same apple a s shown in Fig .1. Stripes on dif f erent parts of the surfaceh a␷e dif f erent shapes s hapes.The stripes on convex surfaces are continuous a nd parabolic,and their curvature directions are m aintained.Stripes on nearly flat surfaces are almost p arallel.For concave surfaces around stem and calyx a reas,the appearance of stripes is complex.Stripes are n ot always parallel,can touch adjacent stripes due to s harp changes in depth,and can appear broken due to o cclusion.In the case of the continuous stripes,their c urvatures change sign when the surface changes from c onvex to concave.The stripes across stems are b roken owing to the depth discontinuity.F ig .3shows a n example image with light stripes on the same apple a s in F ig .1.T he correspondence between the shape of the s tripes and the shape of surface parts,provides the n ecessary three-dimensional information and this can b e obtained from the analysis of the dif f erent shape p atterns of curved stripes.Since stems and calyxes a ppear as patches in images,the analysis is directed to t he local area around each patch,which is first s egmented out with the flooding algorithm.Each s egmented patch in the apple image has a correspond-i ng area at the same location in the stripe image. T herefore,a rectangular window is placed around this a rea in the stripe image.The derivation of the t hree-dimensional information is thus reduced to the e xtraction of stripe patterns within each window.Note t hat in this way,the time- consuming reconstruction of t he actual three-dimensional geometric surface and t he associated calibration,as in the usual application o f light striping,are avoided.A principal parameter for describing the shape of t he stripes is curvature.On a convex surface,the s tripes are continuous and therefore have smoothly c hanging curvatures which may keep the same sign.If t he curvature of the stripes within a window varies d ramatically and changes sign,then the surface inside t he window should normally be a stem or calyx area (see discussion section for other cases).Before the c urvature is calculated,the stripe image is binarized b y a local thresholding process and then the stripes a re thinned to one pixel wide skeleton curves by a s hape preserving thinning method.7T he stripes are t hen represented by their skeleton curves,from which t heir curvatures are computed.The thinning result of F ig .3is shown in F ig .4.T he normal curvature calculation for planar digital c urves is notoriously inaccurate.Here we use a c urvature estimator for the skeleton curves.Let xϭx[s(t)]a nd yϭy[s(t)] be a skeleton curve in its p arametric form,where s is the arc length of the curve a nd is represented as a linear function of parameter t,sϭs(t).Let ␺(s) represent the tangent angle of the c urve at s.The point curvature k o f the curve isQ.Y A N G 232F ig .4.T he thinning result of Fig .3d efined as the instantaneous rate of change of ␺w ith r espect to arc length sk(s)ϭd␺(s)/d s(1) w here␺[s(t)]ϭt anϪ1y᝽(t)x᝽(t)(2)T o reduce the noise caused by taking the second-order d erivatives in the usual calculation of curvature,we u se the following practical method to approximate the c urvature.Let the increment ds be a small constant, t hen the dif f erenced␺(t)Ϸ␺(tϩ1)Ϫ␺(t)(3) c ontains curvature information.We,therefore,define t he dif f erence as an estimator of the point curvaturek␺ϭ␺(tϩ1)Ϫ␺(t)(4) T o reliably calculate ␺i n Eqn (4),we smooth the c urve,xϭx(t)and yϭy(t),with a Gaussian kernel g(t,␴)o f standard deviation ␴.The smoothed curve, X(t,␴)a nd Y(t␴),is expressed asX(t,␴)ϭx(t)ءg(t,␴)ϭ͵3␴Ϫ3␴x(u)1␴42πeϪ(tϪu)2/2␴2d u(5) Y(t,␴)ϭy(t)ءg(t,␴)ϭ͵3␴Ϫ3␴y(u)1␴42πeϪ(tϪu)2/2␴2d u(6)w here u i s a dummy integral variable and ءi s the s tandard symbol for convolution.Their first deriva-t ives are expressed asX᝽(t,␴)ϭx(t)ءѨg(t,␴)Ѩtϭ͵3␴Ϫ3␴x(u)Ϫt␴342πeϪ(tϪu)2/2␴2d u(7) Y᝽(t,␴)ϭy(t)ءѨg(t,␴)Ѩtϭ͵3␴Ϫ3␴y(u)Ϫt␴342πeϪ(tϪu)2/2␴2d u(8) S o ␺(t) in Eqn (4) is computed as␺(t)ϭt anϪ1Y᝽(t)X᝽(t)(9)N ote that whilst the angle function ␺(t) in Eqn (9) is m ultiple-valued,under practical constraints on con-t inuity of the derivative,it is uniquely defined.4.F eature selectionI n this study,the features are intuitively determined t o provide reasonable coverage of the feature space. O nce a small patch is segmented out from an apple i mage,some features of the patch are already avail-a ble,which include the lowest grey-level I l,the highest g rey-level I h,the area A,the boundary length C p a nd t he water volume V o f the lake representing the patch. O ther features,including curvature,are extracted f rom both the apple image and the stripe image.F or the curves within a focusing window,some s tatistics of curvature are computed.These are ave-r age positive and negative curvatures,K p a nd K n,the a verage absolute curvature ͉K͉,and the frequencies of p oints with positive,negative and zero curvature,F p, F n a nd F z,respectively.These statistics describe the c oarse distribution of the curvature.A n important descriptor for the shape patterns of s tripes is the similarity between the shapes of curves. T he stripes crossing a convex surface have greater s imilarity (i.e.are more parallel),than those crossing s tem and calyx areas.The similarity can be measured b y the correlation of curvature between curves.Ther-e fore the average correlation coef ficient R c i s selected. T he length of the analysed curves within a window is a g ood indicator of their continuity,which is strongly r elated to blemishes on convex surfaces.Hence,the a verage curve length L m i s included as a feature.On t he other hand,the relative number of short curveA P P L E S T E M A N D C A L Y X I D E N T I F I C A T I O N W I T H M A C H I N E V I S I O N233s egments within a window is a clue to the irregularity o f the stripes.The short curve segments refer to the s mall curves that are not long enough for computing t heir smoothed curvature.They usually occur on thec oncave stem and calyx areas where stripes ared iscontinuous owing to sudden change in depth.So, t he relative number of short curve segments S s i s c omputed.I n apple images,when a standing stem appears as a p rotrusion or the small spikes of a calyx appear on theb oundary of a patch,the boundary is likely to be morec omplex than that of a blemish.Hence,the compact-n ess P o f the patch is selected,which is defined as the r atio of the squared perimeter C p2to the area A,i.e. PϭC p2/A.The perimeter C p i s simply the count of p ixels on the patch boundary.Two shape measure-m ents of the water volume of the lake are also s elected.These are the average depth h a o f the v olume,defined as V/A,and the ‘‘conical’’ angle ␣o f t he volume,which is defined as the average radius d o f t he boundary to the lake’s centre divided by the m aximum depth of the water,i.e.d/(I hϪI l).To c ompute this latter measure,the geometric centroid of t he lake surface is found by the first moment method. T o complete,the primary intensity properties,mean I m a nd variance I␷o f the grey-level in a patch,are i ncluded.A ltogether,eighteen features are extracted for each p atch and are summarized as follows.F rom the stripe image :K p a verage positive curva-t ure,K n a verage negative curvature,͉K͉a verage a bsolute curvature,F p f requency of points with posi-t ive curvature,F n f requency of points with negative c urvature,F z f requency of points with zero curvature, R c a verage correlation coef ficient of curvature bet-w een curves,L m a verage curve length,S s p ercentage o f short curve segments,F rom the apple image :I m a verage grey level inten-s ity,I␷v ariance of grey level intensity,I l l owest grey l evel of the patch,I h h ighest grey level of the patch,A a rea of the patch,P c ompactness of the area,V w ater v olume of the lake,␣c onical angle of the volume,h a a verage depth of the volume.A s can be seen,these features are not independent.B ut in this study,we do not address the issue of s alient feature selection.Since the dependencies are n ot easily modelled and the features are not dif ficult t o compute,we retain all the features for the following c lassification.5.C lassification of patches with neural networksO nce the feature vector is extracted for each patch, s tems and calyxes can be identified by feeding the f eatures into a classifier and classifying the patch as e ither blemish or stem/c alyx,which are two possible o utput classes.The input feature data to the classifier p ossess some nonlinear relations to their correspond-i ng surface types.F ig .5a s hows a set of average p ositive and negative curvature data for two types of p atches.F ig .5b s hows mean intensity and variance. T he data are from the first group of samples (Training1).It can be seen from these figures that the twoc lusters representing the two classes are not linearly s eparable for these variables.A non-linear classifier is r equired.I n this work, a multi-layer feedforward neural n etwork was constructed for the classification.Three d if f erent network configurations,(18 :10:2),(18:27:2) a nd (18:10:6:2) were tested.No attempt was made to find an optimal network configuration and it might be p ossible to reduce the complexity of the net.The n etwork was fully connected.The number of nodes in t he input layer was the same as the number of input–0·1–0·2–0·3–0·4Averagenegativecurvature0·000·050·100·150·200·25Average positive curvature(a)(b)0·70·60·50·40·30·20·10·00·00·20·40·60·81·0Average grey-levelVarianceofgrey-levelintensityF ig .5.D istribution of two classes in feature sub -s pace .(a) S cattergram of a␷e rage positi␷e cur␷a ture ␷e rsus a␷e rage n egati␷e cur␷a ture for blemish and stem/c alyx ;(b) s cattergram of mean grey -l e␷e l intensity ␷e rsus ␷a riance of g rey -l e␷e l intensity for the two types of patches .ᮀ,blemishes ;ϫ,stalks/c alyxesQ.Y A N G 234f eatures (18).The output layer of the neural network w as composed of two nodes to correspond to the two o utput classes.The hidden layer contained ten nodes i n the first configuration,27 in the second.The third c onfiguration had two hidden layers with ten and six n odes respectively.The nodes in the output layer and h idden layer(s) had a non-linear transfer function of s igmoidal shape.The network was trained with a m odified back-propagation learning algorithm.8T he a lgorithm accelerated the learning by changing the t wo coef ficients used in the conventional back-p ropagation algorithm,learning rate and momentum f actor.D uring the training,the weights of the network w ere updated after each pass through all the training s amples.The desired outputs for each nodes in the o utput layer were arranged in an on –o f f manner,i.e. (1,0) represented blemish and (0,1) represents s tem/c alyx.The convergence of the learning was j udged by two conditions :whether the mean squared e rror for all training samples was smaller than a c riterion and whether the output errors for each t raining sample were all smaller than another c riterion.6.E xperimental resultsT he proposed techniques were tested with Golden D elicious and Granny Smith apples obtained from two p ackhouses in southern France.The apples were g raded in terms of surface defects by skilful inspec-t ors.Table 1 lists the number of sample fruits from e ach variety and grade.T he experimental arrangement for image acquisi-t ion comprised a CCD monochromatic camera (Pana-s onic,model WV-CD20),a projector for structured l ighting,and a light chamber for dif f use lighting.The a ngle between the optical axes of the camera and the p rojector was 20Њ.The light stripes were formed by p rojecting light through an optical grid of thin tran-s parent lines.The camera was mounted above the topT able 1S ample applesG radeA pple ␷a riety E xtra I I I I II T otalG oldenD elicious1214462698G ranny Smith810382480o f the lighting chamber.Apples were placed in the m iddle of the field of view.The orientation of the s ample fruits was chosen in a random manner.For t he blemish-free fruits from the Extra grade,the s tems/c alyxes were made visible to the camera.The b ackground of the viewing field was black.The video s ignal of the camera was sent to a transputer-based f rame grabber,where it was digitized into video m emory with 512ϫ512 spatial resolution and 8 bit g rey-level resolution.All images were stored and s hown in this paper in the size of 256ϫ256 pixels.T wo images were acquired for each view of an a pple.One was under dif f use illumination,another w as under structured light.A few apples had more t han one blemish on dif f erent sides of the fruit,hence m ultiple pairs of images were recorded for each of t hese apples.The flooding algorithm was applied to s egment out dark patches from apple images.For each s egmented patch,the selected 18 features as described i n the previous section were extracted from both i mages.The type of the patch,i.e.blemish or s tem/c alyx,was recorded together with the features, t o form a labelled sample.As shown in Table 2,a t otal of 274 samples were collected and arranged in f our sets.Two of them were training samples,the o ther two were used to test the performance of the t rained neural network classifiers.There were no t raining samples included in the test sample sets.The b lemish category included bruise,russet,scab,wound, i nsect bite,sun-burn and rot.T he extracted features had very dif f erent ranges of v alues,so to input them to the neural networks,the f eature values were normalized to the range of [0,1и0].The neural networks were implemented on a t ransputer system.In all the trainings the initial l earning rate and momentum factor were fixed at 0и1 a nd 0и5 but the training algorithm employed was not v ery sensitive to these initial values.The weights of t he networks were all initialized with random numbers i n the range of [0,1и0].The convergence criteria of the n eural network during learning were set to be less t han 0и001 for mean squared error and 0и01 for a bsolute output error of each training sample.These v alues are relatively small and were chosen to allow a ccurate learning with little loss of the generalizationc apability of the networks.All trainings with somed if f erent sets of initial weights converged rapidly, w ithin about 100 to 800 iterations.I n the performance evaluation for the trained neu-r al networks,a range of 0и3 was set in the interpreta-t ion of classification outputs,instead of the crisp o n-of f outputs (1,0) and (0,1).That is,a node in the o utput layer was regarded to be ‘‘on’’ if its output v alue was larger than 0и7,‘‘of f’’ if less than 0и3,A P P L E S T E M A N D C A L Y X I D E N T I F I C A T I O N W I T H M A C H I N E V I S I O N235T able 2D if f erent types of samplesA pple ␷a riety S amplesB lemish S tem/c alyx S ubtotal T otalG olden Delicious T raining 1473784T est 1442569274G ranny Smith T raining 2511566T est 2411455‘‘unknown’’ if between 0и3 and 0и7.The pattern p resented at the input layer was classified to belong to t he category that the ‘‘on’’ node represented.Thus,if t he outputs of the two nodes in the output layer were i n the range (1и0–0и7,0и3–0и0),the input patch was a ccordingly classified into the blemish category,and if t hey were in (0и0–0и3,0и7–1и0),the stem/c alyx c ategory.If both nodes were either ‘‘on’’ or ‘‘of f’’ and i f a node had ‘‘unknown’’ output,the input patch was c lassified into the unknown category.The selection of r ange 0и3 was conservative and imposed a strict r equirement for the networks.The range could be r elaxed to a slightly greater value.T he classification results are listed in Tables 3 and 4. T able 3 shows the classification accuracy in each test. T he accuracy was computed by dividing the number of c orrectly classified samples by the number of samples i n the test group.Table 4 shows the number of c lassification errors.Error I r efers to the misclassifica-t ion of stem/c alyx as blemish,Error I I i s vice versa,a nd Error I II r efers to the case in which either ab lemish or stem/c alyx input patch is classified into the u nknown category.It can be seen from Table 3 that t he classification accuracy of the three network con-figurations is about the same.As shown in Table 4, t he number of errors in the unknown category is s lightly reduced in the case of Test 1 with the network c onfiguration (18:10:6:2) and the case of Test 2 with t he network configuration (18:27:2).Since the first c onfiguration with 10 nodes in the hidden layer has m uch less complexity and its performance is accep-t able,it is to be preferred.The same number of errors i n Errors I a nd I I f or dif f erent configurations indicatesT able 3C lassification accuracy of the trained neural networksC lassification accuracy (%)N etworkc onfiguration T est 1T est 218:10:295и7 94и518:27:295и7 96и418:10:6:297и1 94и5T able 4C lassification errors of the trained neural networksN umber of errors S ampleg roup E rror type(18:10:2)(18:27:2)(18:10:6:2)T est 1I 0 0 0I I 1 1 1I II 2 2 1T est 2I 1 1 1I I 1 1 1I II 1 0 1t hat one or two specific pattern samples have feature v alues in the range of the ones for the other class.In t hese cases,persistent errors would occur.On the w hole,good classification accuracies have been a chieved.7.D iscussionT he technique developed in this paper is based on a n assumption that blemishes only occur on convex or flat surfaces of apples and therefore the light stripe p atterns on them reflects the type of the surface.It c an cope with the cases in which the blemished s urfaces are slightly indented.But,if the surface of a b lemish has a significantly concave shape as in cases of s evere mechanical damage,or a blemish occurs on the c oncave areas around stem or calyx,the shapes of the s tripes on the blemish will be similar to those on s tem/c alyx areas and the system will fail.Another l imitation is that if a dark patch occurs adjacent to the b oundary of a fruit in a view,the segmentation p rocess will not output it and no subsequent classifica-t ion will be made.However,in practical implementa-t ion,multiple views of a fruit are expected,so there is a good chance that the patch will not be adjacent to t he boundary of the fruit in one of the views.A n advantage of the technique is that,since onlyQ.Y A N G 236q ualitative features of stripe patterns are required, t here is no need for conventional three-dimensional s urface fitting and the calibration for the triangulation o f the structured lighting system is eliminated.This w as deliberately avoided because the complex and u npredictable stripe patterns,especially those caused b y out-standing stems,may present dif ficulties to those fitting techniques.T he technique has potential for high speed im-p lementation because of its simplicity.The necessary t hree-dimensional information is reduced to simple c onvexity or concavity,which is represented by the s hape pattern of the stripes.All segmented patches a nd all stripes on each patch can be analysed in p arallel.In the experimental arrangement,the two i mages are grabbed in sequence.If the wavelength of t he structured light is chosen in the appropriate s pectral range,the two images can be grabbed simul-t aneously by two sensors.9I t is also worth pointing out t hat the technique has the potential to be used for o ther fruits.8.ConclusionA n image analysis technique for the identification of a pple stems and calyxes has been developed.Struc-t ured light is used to provide the necessary three-d imensional shape information of apple geometric s urfaces.The analysis is focused on dark patches of f ruit surfaces,which are first segmented out by a flooding algorithm.For each patch,the qualitative f eatures of the stripe pattern are extracted from the s tripe image and the two-dimensional intensity pro-p erties are extracted from the apple image under n ormal dif f used light.When the three-dimensional a nd two-dimensional information is fused with a m ulti-layer feedforward neural network,the segmen-t ed patches are classified as stem/c alyx or blemish.T he proposed technique was tested with samplea pples and an average identification accuracy of 95%w as achieved.This result demonstrates that the system c an ef f ectively identify stems and calyxes which ares urrounded by concave surfaces,and distinguish them f rom discoloured blemishes which occur on convex or flat surfaces.A cknowledgementT his work was partly funded by the EC (CAMAR : 8001-CT91-0206).The author would like to thank P rof.A.K.Thompson at Cranfield University and Dr R.D.Tillett for valuable discussion.R eferences1W olfe R R ;Sandler W E A n algorithm for stem detection u sing digital image analysis.Transactions of the ASAE 1985,28 :641 –6442M iller B K ;Delwiche M J P each defect detection with m achine vision.ASAE Paper 89 –60193Y ang Q F inding stem and calyx of apples using structured l puters and Electronics in Agriculture 1993,8:31 –424Y ang Q C lassification of apple surface features using m achine vision and neural puters andE lectronics in Agriculture 1993,9:1–125Y ang Q A n approach to apple surface feature detectionb y machine puters and Electronics in Agri-c ulture 1994,11 :249 –2646B allard D H ;Brown C M C omputer vision,EnglewoodC lif f s,Prentice-Hall,1982,pp.52 –547G onzalez R C ;Woods R E D igital image processing,A ddison-Wesley Publishing Company,1992,pp.492 –4948C han L W ;Fallside F A n adaptive training algorithm forb ack propagation puter Speech and Lan-g uage 1987,2:205 –2189C rowe T ;Delwiche M R eal-time defect detection in fruit.I nternational Conference on Agricultural Engineering(AgEng’94),Milano,Italy,29 August –1st September 1994,Report No.94-G-027。