google的机器学习

14
Approximations
Subsampling
• Simple and often effective
More advanced techniques
• Incremental training: (e.g., [Syed et al., 1999])‫‏‬
Data splitted to 10 parts train 1st part ⇒ SVs, train SVs + 2nd part, ...
12
Parallel: Multi-core/Shared Memory
• Most computation of decomposition methods:
kernel evaluations
• Easily parallelized via openMP • One line change of libsvm • Each core/CPU calculates part of a kernel column
Find a patient's blood pressure, weight, etc. After several years, know if he/she recovers Build a machine learning model New patient: find blood pressure, weight, etc Prediction
Google Beijing Research Team
• Web scale mining infrastructure
Parallel SVM (support vector machine) Parallel CCF (combinational collaborative filtering) Parallel LDA (latent Dirichlet allocation) Parallel SVD (singular value decomposition) Parallel Spectral Clustering Parallel Association Mining
Multicore 1 80 2 48 4 32 8 27 Shared-memory 1 100 2 4 8 57 36 28
Same 50,000 data (kernel evaluations: 80% time)
13
Parallel: Distributed Environments
What if data data cannot fit into memory? Use distributed environments
• Traditional optimization methods:
Newton, Quasi Newton cannot be directly applied
9
Decomposition Methods
• Working on some variables each time
(e.g., [Joachims 1998; Platt 1998])
• PSVM: [Chang et al. 2007]
Details will be discussed later
• pi-SVM: • Parallel GPDT [Zanni et al., 2006] • All use Message Passing Interface (MPI)‫‏‬ • They achieve good speed-up
17
When problems happen in Google
• A lot of classic algorithm on text-books will not be straight
to apply in Google. – – – –
Single machine Fit into memory Disk space is relatively infinite Running time in seconds/minutes
– Variable w: may be infinite – The dual problem (finite # variables)
– Subject to
– Where – At optimum
7
Kernel Tricks
• Need a closed form
• Example
Popular SVM programs – scalability problem
• LibSVM, SVMLight, CVM • Still time consuming: 500k dataset -> 6 hours to several days
But training large data still difficult
• Kernel square to the number of data • Training millions of data time consuming • Will discuss some possible approaches
• Similar to coordinate-wise minimization • Modifying a few variables per iteration: a working set B • The new objective function
– Calculated when used – Trade time for space – Popular software such as svmlight and libsvm are of this
– Training data
– Maximizing the margin [Boser et al. 1992; Cortes and
Vapnik 1995]
– Subject to
– High dimensional (maybe infinite) feature space
6
Support Vector Classification (Cont'd)‫‏‬
• Support vector machines (SVM): a popular classification
technique
• At Google China: we develop its fast training strategies • Basic idea of SVM:
5
Support Vector Classification
• Training and testing
4
Machine Learning at Google
• In many places we need machine learning techniques
e.g., trend classification, image classfication
type
10
How Decomposition Methods Perform?
• An example of training 50,000 instances using libsvm
$svm-train -c 16 -g 4 -m 400 22features Total nSV = 3370 Time 79.524s
Machine Learning
Chih-Jen Lin/Kaihua Zhu June 12, 2008
What is Machine Learning
– Extract knowledge from data – Traditional tasks
Classification, clustering, and others
Classification
clustering
– An old area, but many new and interesting
applications/extensions: ranking, etc.
3
Data Classification
• Given training data in different classes (labels known) • Predict test data (labels unknown) • Classic example
• On a Xeon 2.0G machine • Calculating Q may have taken more time • #SVs = 3,370 << 50,000 • A good case where some remain at zero all the time
11
Issues of Decomposition Methods
Google Beijing Research Team
• Web scale mining infrastructure
Parallel SVM (support vector machine) Parallel CCF (combinational collaborative filtering) Parallel LDA (latent Dirichlet allocation) Parallel SVD (singular value decomposition) Parallel Spectral Clustering Parallel Association Mining
Training size Accuracy 50k 100k 550k 97.2% 97.4% 97.8%
• For large sets, selecting a right approach is essential • We will show details of a parallel SVM implementation