Kernel Methods and Support Vector Machines for handwriting recgnization

In pattern recognition, S V M have been used in handwritten digit recognitioR object recognition, speaker identification, face detection, and text categorization. In handwriting recognition, in particular for word recognition based on a lexicon or words library, a word need to be segmented into a number of character hypothesis and each character hypothesis needs to be classified with character probability by a probabilistic multiclass classifier. These separate character hypathesis can then be combined into the most probable word in the lexicon by a Hidden Markov Model (HMM). Normally, a hybrid neural network (NN) and HMM system is used.
'
Abstract-This paper presents a review of kernel methods in machine learning. Suppon Vector Machine (SVM) as one of the methods in machine learning to make use of kernel is first discussed with the intention of applying it to handwriting recognition. S V M work by mapping training data for a classification task into a higher dimensional feature space using kernel function and then find a maximal margin hyper plane. which separates the mapped data. Finding the solution hyper plane involves using quadratic programming which is computationally intensive. The algorithms for practical implementation like Sequential Minimization Optimization (SMO) and its improvements are discussed. A few simpler methods similar to S V M but requires simpler computation are also mentioned for comparison. Usage of S V M for handwriting recognition is then proposed.
0-7803-7565-3/02/$17.00 02002 IEEE.
309
describes the various implementations of S V M and other kernel methods. The authors intended application of SVM into handwriting recognition is presented in section 4. Section 5 concludes.
When S V M is used in the non-linear case, the data sets (input space) need to be mapped into a higher dimensional feature space, which is linear before the linear large-margin leaming algorithm is applied. Often the mapping need not be explicitly done because, kemel functions can be used in ,order 10 perform the mapping implicitly. Io the high dimensional feature
Abdul Rahim Ahmad
'
Marzuki Khalid
Rubiyah Yusof2
Dept. o Computer Science and IT, f College ofEngineering, Universiti Tenaga Nasional, 43009 Kajang, Selangor, Malaysia. 'Centre for Art8cial Intelligence and Robotics, Universiti Teknologi Malaysia, Jalan Senrarak, 54100 Kuala Lumpur, Malaysia.
It is intended that the NN be replaced by SVM since it have shown higher recognition r recognition. Orig~nal SVM cannot be used to replace the NN classifier due to two facts: (1) it is a two-class classifier; (2) its output is not probabilistic. In order to make use of SVM, multi-class S V M need to be used and also the outputs of the SVM need to be converted to Probability value. To obtain probabilistic output, one way is to pass SVM output through a sigmoid function to map it into probabilities. This paper propose a method to incorporate S V M into a previously developed system at CAIRO of word recognition system combining Multi Layer Perceptron (MLP) neural network and HMM. is organized as It follows: in the next section, basic idea of S V M is first presented for linearly separable case followed by non-linear and imperfect separation case. Section 3
2002 Student Conference on Research and Development Proceedings, Shah Alam, Malaysia
Kernel Methods and Support Vector Machines for Handwriting Recognition
1.
space, simpler and linear hyper plane classifiers that have maximal margin between the classes can be obtained. The problem with S V M is that its solution is usually found by using a quadratic programming routine, which is computationally intensive, subject to stability problem and is non-trivial to implement (41. Due to that, SVM is limitedly used in practice. Some researchers have instead implemented linear learning algorithm for non-linear problems by making use of kernel but without the quadratic programming implementation as in SVM. Examples of such algorithms are kernel-Adaline [3] and kernel-Adatron 141.
SVM was introduced in 1992 during the COLT92 conference by Vapnik and his co-workers [I], [2]. In its original f o q SVM is a training algorithm used for classification that tunes the capacity of the classification function by maximizing the margin between linearly separable training patterns and the decision boundary. The solution is expressed as a linear combination of supporting patterns, (which are the subset of training patterns) that are closed to the decision boundary, called the support vectors.