华南理工大学《模式识别》(本科)复习资料

华南理工大学《模式识别》复习资料CH1.【Pattern Recognition Systems】Data Acquisition & SensingMeasurements of physical variablesPre-processingRemoval of noise in dataIsolation of patterns of interest from the background (segmentation)Feature extractionFinding a new representation in terms of featuresModel learning / estimationLearning a mapping between features and pattern groups and categoriesClassificationUsing features and learned models to assign a pattern to a categoryPost-processingEvaluation of confidence in decisionsExploitation of context to improve performance Combination of experts【Design Cycle】【Learning strategies】Supervised learningA teacher provides a category label or cost for each pattern in the training setUnsupervised learningThe systems forms clusters or natural grouping of the input patternsReinforcement learningNo desired category is given but the teacher provides feedback to the system such as the decision is right or wrong【Evaluation methods】Independent RunA statistical method, also called Bootstrap. Repeat the experiment "n" times independently, and take the mean as the result.Cross-validationDataset D is randomly divided into n disjoint sets D i of equal size n/m, where n is the number of samples in D i. Classifier is trained m times and each time with different set held out as a testing setCH2.【Bayes formula】【Bayes Decision Rule 】【Maximum Likelihood (ML) Rule 】When p(w 1)=p(w 2),the decision is based entirely on the likelihood p(x|w j ) --> p(x|w)∝p(x|w)【Error analysis 】Probability of error for multi-class problems:Error = Bayes Error + Added Error:【Lost function 】Conditional risk (expected loss of taking action ai):Overall risk(expected loss):zero-one loss function is used to minimize the error rate【Minimum Risk Decision Rule 】【Normal Distribution 】Multivariate Normal Density in d dimensions:【ML Parameter Estimation 】【Discriminant function 】【Decision boundary 】CH3.【Normalized distance from origin to surface 】【Distance of arbitrary point to surface 】【Perceptron Criterion 】【Pseudoinverse Method 】Problem:[[Exercise for Pseudoinverse Method]](2)【Least-Mean-Squared (Gradient Descent)】【Linear classifier for multiple Classes 】【linearly separable problem 】A problem whose data of different classes can be separated exactly by linear decision surface.CH4.【Perception update rule 】(reward and punishment schemes)[[Exercise for perception ]]【Error of Back-Propagation Algorithm 】Update rule for weight:【Weight of Back-PropagationAlgorithm 】The learning rule for the hidden-to-output units :The learning rule for the input-to-hidden units:Summary:【Training of Back-Propagation 】Weights can be updated differently by presenting the training samples in different sequences.Two popular training methods: Stochastic TrainingPatterns are chosen randomly form the training set (Network weights are updated randomly) Batch trainingAll pattern are presented to the network before learningtakes place【Regularization 】【Problem of training a NN 】Scaling input Target valuesNumber of hidden layers3-layer is recommended. Special problem: more than 3 Number of hidden units roughly n/10Initializing weightsWeight decayStochastic and batch training Stopped trainingWhen the error on a separate validation set reaches a minimum[[Exercise for ANN ]]forward pass ：g=0.8385reverse pass: (learning rate=0.5)CH5.【Structure of RBF 】3 layers:Input layer: f(x)=xHidden layer: Gaussian function Output layer: linear weight sum【Characteristic of RBF 】Advantage:RBF network trains faster than MLPThe hidden layer is easier to interpret than MLP Disadvantage:During the testing, the calculation speed of a neuron in RBF is slower than MLP[[Exercise for RBF ]]Solution:===>CH6.【Margin 】* Margin is defined as the width that the boundary could be increased by before hitting a data point* The linear discriminant function (classifier) with the maximum margin is the best.* Data closest to the hyper plane are support vectors.【Maximum Margin Classification 】* Maximizing the margin is good according to intuition and theory.* Implies that only support vectors are important; other training examples are ignorable.Advantage: (compare to LMS and perception) Better generalization ability & less over-fitting【Kernels 】* We may use Kernel functions to implicitly map to a newfeature space* Kernel must be equivalent to an inner product in some feature space【Solving of SVM 】* Solving SVM is a quadratic programming problem Target:maximum margin ->==>Such that【Nonlinear SVM 】The original feature space can always be mapped to some higher-dimensional feature space where the training set is separable【Optimization Problem 】Dual Problemfor(a iis Lagrange multiplier):Solution (Each non-zero a i indicates that corresponding x iis a support vector.):Classifying function (relies on an inner product between the test point x and the support vectors xi. involvedcomputing the inner products x i ‘ * x j between all trainingpoints):【Slack variables 】Target ：Dual Problem of the soft margin is the same for hard .Solution:Classifying function of the soft margin is the same. 【Kernel Trick 】* Map data points to higher dimensional space in order to make them linearly separable.* Since only dot product is used, we do not need to represent the mapping explicitly.Discriminant function: (No need to know this mapping explicitly, because we only use the dot product of feature vectors in both the training and test.)Kernel function : dot product of two feature vectors in some expanded feature space :【Nonlinear SVM optimization 】CH7.【Decision Tree Classification Task 】【Hunt's algorithm 】【Splitting attributes 】Attribute types: Nominal, Ordinal, ContinuousNumber of ways to split: Binary split, Multi-way splitFor Continuous attributes:For ordinal:【Homogeneous & Impurity 】【Impurity measurement 】GINI ：Entropy:Error:【Splitting continous value into binary 】For each splitting value, calculate it ’s GINI and choose theminimum one:【Address Overfitting 】Underfitting : when model is too simple, both training and test errors are large.Overfitting: when model is too complex, training error is small but test error is large.[[Exercise for decision tree ]] Calculate impurity:Build tree:CH8.【Advantages of ensemble learning】【Strength and weakness of AdaBoost】【AdaBoost Algorithm】【Reweighting】CH9.【Supervised&unsupervised learning】Supervised learningDiscover patterns in the data that relate data attributes with a target (class) attribute. These patterns are then utilized to predict the values of the target attribute in future data instances.Unsupervised learningThe data have no target attribute. We want to explore the data to find some intrinsic structures in them.【Properties of distance measurement 】【Error of k-means 】【Algorithm of k-means 】【elbow finding 】find the best k:【Strength and weakness of K-means 】--The algorithm is sensitive to initial seeds【To deal with outliers 】【Fuzzy C-means Clustering 】[[Exercise for decision tree]]。