计算机视觉+人体姿态识别+双目视觉

Computer vision application院（系）电子与信息工程学院专业集成电路工程学生吕广兴14S158054Computer vision applicationThe directoryReport: Computer vision application (2)1.The object of the project (2)2.The method and the principle applied to the project (2)2.1 Platform (2)2.2 The principle of transform the RGB image to the gray image (2)2.3 The principle of image enhancement (2)2.4 The principle of thresholding (3)2.5 The principle of classifier (3)3.The content and the result of the project (4)3.1 The main steps in the project (4)3.2 About human body posture recognition (4)About three kinds of methods are most common: (4)3.3.Stereo vision (11)4.Reference (18)Report: Computer vision application1.The object of the projectThe object of the project is Gesture recognition and location in the interior of people.2.The method and the principle applied to the project2.1 PlatformThe platform is based on Visual Studio 2012 and OpenCV 2.4.10.2.2 The principle of transform the RGB image to the gray imageThere are three major methods to transform the RGB image to the gray image.The first one is called the maximum value that is set the value of R, G, and B to the maximum of these three.Gray=R=G=B=max(R, G, B)The second one is called mean value which is set the value of R, G, and B to the mean value of these three.Gray=R=G=B=(R+G+B)/3The third one is called weighted average that is giving different weights to the R, G and B according to the importance or other indicators, and then adding the three parts together. In fact, human’s eye is very high se nsitive to green, then red, last blue.Gray=0.30R+0.59G+0.11B2.3 The principle of image enhancementImage enhancement is the process of making images more useful. There are two broad categories of image enhancement techniques. The first one is spatial domain technique, and it is a direct manipulation of image pixels that includes point processing and neighborhood operations. The second one is frequency domain technique, and it is a manipulation of Fourier transform or wavelet transform of an image.The principle of the median filter is to replace the value of a pixel by the median of the gray levels in the neighborhood of that pixel(the original value of the pixel is included in the computation of the median). It forces the points with distinct gray levels to be more like their neighbors.In addition, we also apply the morphological image processing after smoothing. Morphological image processing (or morphology) describes a range of image processing techniques that deal with the shape (or morphology) of features in an image. The basic ideal of Morphology is to use a special structuring element to measure or extract the corresponding shape or characteristics in the input images for further image analysis and object recognition. The mathematical foundation of morphology is the set theory. There are two basic morphological operations: erosion and dilation.2.4 The principle of thresholdingThresholding is particularly useful for segmentation in which we want to isolate an object of interest from a background. At the same time, thresholding segmentation is usually the first step in any segmentation approach. The blow formula is the basic principle of image segmentation. When the gray level is no bigger than the threshold, we will set the pixel value zero(black). In contrast, when the gray level is bigger than the threshold, we will set the pixel value 255(white).thresholdr thresholdr s ><==,255,0{ When it comes to the threshold, we get the value through the image histograms 2.5 The principle of classifierThe classifier is a algorithm or device that separates objects into different classes. Usually, a classifier consists of three parts. First one is the sensor, for instance, imaging device, fingerprint reader, etc...Second one is feature extractor, for example, edge detector or property descriptor. Third one is classifier which uses the extracted features for decision making or Euclidian distance, or other methods.Features should can be regarded as the descriptors we introduced before. And the feature should be representative and useful for classification.When it comes to the feature space, the set of all possible patterns form the feature vector. Each feature vector is a point in the so-called feature space. Similar objects yield similar measurement results. Nearby points in feature space correspond to similar objects. Distance in feature space is related to similarity. Points that belong to the same class form a cloud in feature space.Divide the data set into a training set and a test set. The performance of a classifier should be assessed by the classification error on an independent test set. This set should not contain objects that are included in the training set. Determine a decision boundary by minimizing the classification error of the training set. Determine the classifier performance by computing the classification error of the test set.3.The content and the result of the project3.1 The main steps in the projectBefore we segment the vessel and classify the vessel, we can find these images afforded are color. So first we should load these pictures into gray images. Because we use the method of SVM (Support Vector Machines), we should divide these images into a training set and a test set. Next is image enhancement, and the next is thresholding, and then object extraction. Next is feature extraction and vessel classification. We first use the training set to train these images with representative features and then test the test set to recognize the human’s gesture.The depth information of human is obtained by the binocular stereo vision.And then obtain the position of the people and prepare for the three-dimensional reconstruction.3.2 About human body posture recognitionAbout three kinds of methods are most common:1.Method based on template matching.2.Method based on Classification.3.Prediction based approach.The method based on template matching maybe the most accurate of the all three. However this method will consume a lot of time. So it is not real time.The method based on Classification meets the accuracy requirements in the process of dealing with small data and Implementation ,so, in a single scene, for the time being, this method is used for the time being.About third methods:If the data processed by the computer in a complex scene, the data will be expanded in a geometric scale.Dealing with this problem is the most difficult problem in artificial intelligence.However, in recent years, the neural network based on deep learning's voice recognition and image processing has shown the advantages.3.2.1 Foreground extractionMoving target detection is the basis of the whole target detection and tracking system.Based on the video, but also for further processing (such as encoding, target tracking, target classification, target behavior understanding foundation). The purpose of moving target detection is to extract the moving object (such as human, vehicle, etc.) from the video image.Frame difference method, background subtraction method and optical flowmethod. Based on the three kinds of commonly used methods. There are many kinds of improvement methods, one is the inter frame difference method and the background difference method combining method and good results have been achieved, but there are still retain less incomplete object contour detection and target point ing background subtraction method is better than the direct access method, access to background method and statistical average method, it is a method that through carrying on the statistics to the continuous image sequence averaged to obtain image background.In addition to get better background, R.T.Colin proposed to established a single Gauss background model.Grimson et al. Proposed an adaptive hybrid Gauss background model to obtain a more accurate background description for target detection.At the same time, in order to increase the robustness and reduce the impact of environmental changes, to update the background is important.For example based on the recursive updating of the statistical averaging method, a simple adaptive filter is used to update the background model.In this paper, according to the algorithm proposed in KaewTraKulPong et al.[1]Zivkovic et al.[2][3], we use the update variance to the 3 parameters in the root model, and finally, the algorithm is realized by using OpenCV basic function. The main processes are as follows:1.Firstly, the mean, variance and weight of each Gauss are set to 0.2.The T model used in the video is used to train the GMM model. For each pixel, the establishment of its model number of the largest GMM_MAX_COMPONT Gauss GMM model. When the first pixel is set up, the initial mean, variance, and weight are set to 1.3.The first frame of the training process, when back to the pixel value, compared with the previous Gaussian mean, if the pixel value and mean value model in three times the variance of the difference, the task is the Gaussian. At this point, the following equation is used to update.4.when training frames in T, different GMM pixel number adaptive selection. First of all, with the weight divided by the variance of the various Gauss from big to small sort, and then select the most in front of B Gauss, so thatWherec is generally set to 0.3.fSo that we can eliminate the noise points in the training process.5. during the testing phase, the new pixel value is compared with every mean values of B a Gaussian, if the difference between 2 times the variance of the words, that is the background, or that the foreground. And as long as there is a Gauss component to meet the condition is considered a prospect. Foreground assignment is 255, and the background value is 0. So as to form a two value chart.6. Due to foreground binary map contains a lot of noise, so the use of morphological opening operation noise is reduced to 0, followed by the closed reconstruction operation due to the edge portion of the opening operation loss of information. Eliminate the small noise points.The above is the algorithm to achieve the general process, but when we are in the specific programming, there are still many details of the need to pay attention, such as the choice of some parameter values. Here, after testing some of the commonly used parameter values are declared as follows:Among the 3 parameters of the value of the update variance, the learning rate is 0.005. That is to say T equals 200.Define the maximum number of mixed Gauss number for each pixel 7.Take video of the first 200 frames for training.Take Cf 0.3. That is to meet the weight of the number is greater than 0.7 of the number of adaptive Gauss B.During the training process, a new set of Gauss is needed, and the weight value is equal to the value of the learning rate, which is 0.005.During the training process, the need to build a new Gauss, take the mean value of the input pixel value for the input of the Gauss. The variance was 15.During the training process, the need to build a new Gauss, take the variance of the Gauss 15.The following picture is a dynamic background in the training process.Figure 3.1 the result of foreground extraction3.2.2 Feature extractionAfter an image has been segmented into regions. Representation and description should be considered.Representation and description used to make the data useful to a computer. Representing region in 2 ways1.In terms of its external characteristics (its boundary)focus on shape characteristics2.In terms of its internal characteristics (its region) focus on regional properties,e.g., color, texture.Sometimes, we may need to use both ways.Choosing a representation scheme, however is only the part of the task of making data useful to computer.The next task is to describe the region based on the chosen representation.For example:Representation boundary :Description of the length of the boundary, orientation of the straight line joining its extreme points, and the number of concavities in the boundary.To find the feature of the target , we need to extract the contour of the target, and to extract object from the background based on the area of every contour. And having the largest area is the destination physical contour. Here we use the blow function:(1) Find contoursfindContours(image,contours,//轮廓的数组CV_RETR_EXTERNAL,//获取外轮CV_CHAIN_APPROX_NONE);//获取每个轮廓的每个像素(2) Draw contoursdrawContours(result,contours,-1,//绘制所有轮廓cv::Scalar(0),//颜色信息为黑色2);//轮廓线的绘制宽度为2The following image is the result of contours extraction and object extraction.Figure 3.2 The result of extraction of contour and object At last ,we choose the characteristics of Length of boundary and the hight of Feret box to train and predict.We also test other characteristics but not as good as these two.3.2.3. Recognition and classification3.2.3.1 ClassifierWe use the SVM (Support vector machine) classifier to recognize the ships. Support vector machines are supervised learning models with associated learning algorithms that analyze data and recognize patterns, used for classification and regression analysis. Given a set of training examples, each marked as belonging to one of two categories, an SVM training algorithm builds a model that assigns new examples into one category or the other, making it a non-probabilistic binary linear classifier. An SVM model is a representation of the examples as points in space,mapped so that the examples of the separate categories are divided by a clear gap that is as wide as possible. There are four main steps to structure SVM:1. A given training set T={(x1,y1), (x2,y2),…, (x n, y n)}2.Solving two quadratic programming problem:We get the solution:3.Calculate the parameters of W, and selects a positive component to compute b4.To construct a decision boundary:, thus the havethe decision function:All the above work can be done with the OpenCv functions, but it need us to make the train file and test file for it. Training file used for learning, and with the features, we can make classification of these four vessels(testing file). In short SVM steps can be summaries as: training (learning), testing and predicting.When choosing the images, we main keep this in mind: In order to train SVM effectively, we could not choose vessel image for training freely, instead we need to select the vessel shape with more obvious characteristics, and can be representative for vessel image type. If the vessel shape too special or similar would interfere the SVM learning. Because too diverse the sample is, it will increases the difference between feature vectors, reduces the classification of objects. As a result, it increase the burden of SVM learning.Some main SVM codes used:// The parameters of support vector machine settingsCvSVMParams params;params.svm_type = CvSVM::C_SVC; //SVM type: C using support vector machineparams.kernel_type = CvSVM::LINEAR;// The type of kernel function: linearparams.term_crit = cvTermCriteria(CV_TERMCRIT_ITER, 100, 1e-6); //Termination criterion function: when the number of iterations reaches the maximum// SVM trainingCvSVM SVM; // Establish an instance of the SVM modelSVM.train(trainingDataMat, labelsMat, Mat(), Mat(), params); //The training model, parameters are: input data, response, ··,··, featuresFigure 3.4 the result of pattern recognition3.2.3.2 Recognition resultsTest resultsTest samples Correctprecisionidentification number550 550 100%3.2.4. ConclusionFrom the block above, we know that the method we use can distinguish the several ships, but still exist errors. Because the number of given picture not very large .When test the category of one picture ,it is inevitable to make error. And some categories of pictures also have some same points, which the feature is similar, hence it’s hard to distinguish them. What’ more the classifier SVM also exist some errors, it could not classify exactly. It also can be the features we chose are not enough, so some more work should be done.3.3.Stereo vision3.3.1 StereopsisFusing the pictures recorded by our two eyes and exploiting the difference (or disparity) between them allows us to gain a strong sense of depth. This chapter is concerned with the design and implementation of algorithms that mimic our abilityto perform this task, known as stereopsis. Reliable computer programs for stereoscopic perception are of course invaluable in visual robot navigation (Figure 7.1),cartography, aerial reconnaissance, and close-range photogrammetry. They are alsoof great interest in tasks such as image segmentation for object recognition or the construction of three-dimensional scene models for computer graphics applications.figure 3.4: Left: The Stanford cart sports a single camera moving in discrete incrementsalong a straight line and providing multiple snapshots of outdoor scenes. Center: TheINRIA mobile robot uses three cameras to map its environment. Right: The NYU mobile robot uses two stereo cameras, each capable of delivering an image pair. As shown by these examples, although two eyes are sufficient for stereo fusion, mobile robots are sometimes equipped with three (or more) cameras. The bulk of this chapter is concerned with binocular perception but stereo algorithms using multiple cameras are discussed [4]. Photos courtesy of Hans Moravec, Olivier Faugeras, and Yann LeCun.Stereo vision involves two processes: The fusion of features observed by two(or more) eyes and the reconstruction of their three-dimensional preimage. The latter is relatively simple: The preimage of matching points can (in principle) be found at the intersection of the rays passing through these points and the associated pupil centers (or pinholes; see Figure 3.5, left). Thus, when a single image feature is observed at any given time, stereo vision is easy. However, each picture typically consists of millions of pixels, with tens of thousands of image features such as edge elements, and some method must be devised to establish the correct correspondences and avoid erroneous depth measurements (Figure 3.5, right).Figure 7.2: The binocular fusion problem: In the simple case of the diagram shown on the left, there is no ambiguity, and stereo reconstruction is a simple matter. In the more usual case shown on the right, any of the four points in the left picture may, a priori, match any of the four points in the right one. Only four of these correspondences are correct; the other ones yield the incorrect reconstructions shown as small gray discs.However, camera calibration can eliminate the distortion.To obtain more accurate depth information[5].Figure 3.4 the result of calibration of cameras. And the distortion is eliminated. The camera parameters are as follows:extrinsics:1.0R: !!opencv-matrixrows: 3cols: 3dt: ddata: [ 9.9990360000625755e-001, 9.7790647772508701e-003,-9.8570069802389540e-003, -9.8969610939301841e-003,9.9987921323260354e-001, -1.1983701700849161e-002,9.7386269890257296e-003, 1.2080100886666228e-002,9.9987960790634012e-001 ]T: !!opencv-matrixrows: 3cols: 1dt: ddata: [ 3.4075702905319170e+000, 1.1739005828568252e-003, -7.9252820494919135e-002 ]R1: !!opencv-matrixrows: 3cols: 3dt: ddata: [ 9.9940336523117301e-001, 9.8399020411941429e-003, -3.3107248336674097e-002, -1.0040790862200610e-002,9.9993214239488748e-001, -5.9070402429666716e-003,3.3046877060745931e-002, 6.2359388539483330e-003,9.9943434851076729e-001 ]R2: !!opencv-matrixrows: 3cols: 3dt: ddata: [ 9.9972958617043228e-001, 3.4440467660098648e-004, -2.3251578890794586e-002, -2.0319599978103648e-004,9.9998152534851392e-001, 6.0751685610454320e-003,2.3253241642441649e-002, -6.0688011236303754e-003,9.9971118649640012e-001 ]P1: !!opencv-matrixrows: 3cols: 4dt: ddata: [ 8.9095402067418593e+002, 0., 3.2619792175292969e+002, 0., 0.,8.9095402067418593e+002, 2.1098579597473145e+002, 0., 0., 0., 1.,0. ]P2: !!opencv-matrixrows: 3cols: 4dt: ddata: [ 8.9095402067418593e+002, 0., 3.2619792175292969e+002,3.0368096464054670e+003, 0., 8.9095402067418593e+002,2.1098579597473145e+002, 0., 0., 0., 1., 0. ]Q: !!opencv-matrixrows: 4cols: 4dt: ddata: [ 1., 0., 0., -3.2619792175292969e+002, 0., 1., 0.,-2.1098579597473145e+002, 0., 0., 0., 8.9095402067418593e+002, 0.,0., -2.9338487571282823e-001, 0. ]intrinsics:1.0M1: !!opencv-matrixrows: 3cols: 3dt: ddata: [ 1.1136300108848973e+003, 0., 3.0020338800373816e+002, 0.,1.1136300108848973e+003,2.1821348683113223e+002, 0., 0., 1. ]D1: !!opencv-matrixrows: 1cols: 8dt: ddata: [ 1.0442196198936304e-001, -2.3958410365610397e-001, 0., 0., 0.,0., 0., 2.7243194967195151e+001 ]M2: !!opencv-matrixrows: 3cols: 3dt: ddata: [ 1.1136300108848973e+003, 0., 3.0559143946713289e+002, 0.,1.1136300108848973e+003,2.1736307957090108e+002, 0., 0., 1. ] D2: !!opencv-matrixrows: 1cols: 8dt: ddata: [ -1.8888461100187631e-001, 5.8249894498215049e+000, 0., 0., 0.,0., 0., 3.5710666837966521e+001 ]Then we can get more accurate depth information:After correction of the binocular camera,the error within 10 meters is within 4 cm.4.Reference1.KaewTraKulPong, P. and R. Bowden (2001). An improved adaptivebackground mixture model for real-time tracking with shadow detection.2.Zivkovic, Z. and F. van der Heijden (2004). “Recursive unsupervised learningof finite mixture models.” Pattern Analysis and Machine Intelligence, IEEE Transactions on 26(5): 651-656.3.Zivkovic, Z. and F. van der Heijden (2006). “Efficient adaptive de nsityestimation per image pixel for the task of background subtraction.” Pattern recognition letters 27(7): 773-780.PUTER VISION A MODERN APPROACH second edition avid A. Forsyth University of Illinois at Urbana-Champaign Jean Ponce Ecole Normale Supérieure:3-225. Learning OpenCV: Computer Vision with the OpenCV Library by Gary Bradski and Adrian Kaehler Published by O'Reilly Media, October 3, 2008。