朴素贝叶斯分类算法课件(英文)

• Relevant Issues
• Conclusions
2
COMP24111 Machine Learning
Background
• There are three methods to establish a classifier
a) Model a classification rule directly
Play=No
2/9 4/9 3/9
3/5 0/5 2/5
2/9 4/9 3/9
Play=Yes
2/5 2/5 1/5
Play=No
Humidity
High
Normal
Play=Yes Play=No
Wind
Strong Weak
3/9 6/9
4/5 1/5
3/9 6/9
3/5 2/5
P(Play=Yes) = 9/14
•
Naï ve Bayes classification
– Assumption that all input attributes are conditionally independent!
P( X1 , X2 , , Xn |C ) P( X1 | X2 , , Xn ; C )P( X2 , , Xn |C ) P( X1 |C )P( X2 , , Xn |C ) P( X1 |C )P( X2 |C ) P( Xn |C )
P( X x |C ci )P(C ci ) P(C ci |X x) P( X x) P( X x |C ci )P(C ci ) fo r i 1,2 , , L
– Then apply the MAP rule
8
COMP24111 Machine Learning
P(Outlook=Sunny|Play=Yes) = 2/9 P(Temperature=Cool|Play=Yes) = 3/9 P(Outlook=Sunny|Play=No) = 3/5 P(Temperature=Cool|Play==No) = 1/5 P(Huminity=High|Play=No) = 4/5 P(Wind=Strong|Play=No) = 3/5
P(Play=No) = 5/14
12
COMP24111 Machine Learning
Example
• Test Phase
– Given a new instance, x’=(Outlook=Sunny, Temperature=Cool, Humidity=High, Wind=Strong) – Look up tables
Output: conditional probability tables; for X j , N j L elements
, , a – Test Phase: Given an unknown instance X ( a1 , n)
Look up tables to assign the label c* to X’ if
P( x |c L )
Generative Probabilistic Model
Generative Probabilistic Model

Generative Probabilistic Model
for Class 1
for Class 2
for Class L
x1
x2

xn x1
x2
c) Make a probabilistic model of data within each class
Examples: naive Bayes, model based classifiers
a) and b) are examples of discriminative classification • c) is an example of generative classification • b) and c) are both examples of probabilistic classification
– MAP classification rule: for x ( x1 , x2 , , xn )
[P( x1 |c* ) P( xn |c* )]P(c* ) [P( x1 |c) P( xn |c)]P(c), c c* , c c1 , , cL
•
3
COMP24111 Machine Learning
Probability Basics
• Prior, conditional and joint probability for random variables
– Prior probability: P( X )
– Conditional probability: P( X1 |X2 ), P(X2 | X1 ) – Joint probability: X ( X1 , X2 ), P( X) P(X1 ,X2 )
Probabilistic Classification
• Establishing a probabilistic model for classification
– Discriminative model
P(C |X) C c1 , , c L , X (X1 , , Xn )
P(C c* |X x) P(C c|X x) c c* , c c1 , , cL
•
Generative classification with the MAP rule
– Apply Bayesian rule to convert them into posterior probabilities
P(c1 | x) P(c2 | x)

P(c L | x)
Discriminative Probabilistic Classifier
x1
x2

xn
6
x ( x1 , x2 , , xn )
COMP24111 Machine Learning
Probabilistic Classification
P(Huminity=High|Play=Yes) = 3/9
P(Wind=Strong|Play=Yes) = 3/9 P(Play=Yes) = 9/14
P(Play=No) = 5/14
– MAP rule
P(Yes|x’): [P(Sunny|Yes)P(Cool|Yes)P(High|Yes)P(Strong|Yes)]P(Play=Yes) = 0.0053 P(No|x’): [P(Sunny|No) P(Cool|No)P(High|No)P(Strong|No)]P(Play=No) = 0.0206
– Relationship: P(X1 ,X2 ) P( X2 |X1 )P( X1 ) P( X1 |X2 )P( X2 )
– Independence: P( X2 |X1 ) P( X2 ), P( X1 |X2 ) P( X1 ), P(X1 ,X2 ) P( X1 )P( X2 )
• Establishing a probabilistic model for classification (cont.)
– Generative model
P( X |C ) C c1 , , c L , X (X1 , , Xn )
P( x |c1 )
P( x |c2 )
Example
• Example: Play Tennis
11
COMP24111 Machine Learning
Example
• Learning Phase
Outlook
Sunny Overcast Rain
Play=Yes
Play=No
Temperature
Hot Mild Cool
Play=Yes
•
Bayesian Rule
Likelihood Prior P(X|C)P(C) P(C |X) Posterior P( X) Evidence
4
COMP24111 Machine Learning
Probability Basics
• Quiz:
We have two six-sided dice. When they are tolled, it could end up with the following occurance: (A) dice 1 lands on side “3”, (B) dice 2 lands on side “1”, and (C) Two dice sum to eight. Answer the following questions:
ˆ (a |c* ) P ˆ (a |c* )]P ˆ (c * ) [ P ˆ (a |c) P ˆ (a |c)]P ˆ (c), c c* , c c , , c [P 1 n 1 n 1 L
10
COMP24111 Machine Learning
Examples: k-NN, decision trees, perceptron, SVM
b) Model the probability of class memberships given input data
Example: perceptron with the cross-entropy cost

xn
x1
x2

xn
x ( x1 , x2 , , xn )
7
COMP24111 Machine Learning
Probabilistic Classification
• MAP classification rule
– MAP: Maximum A Posterior – Assign x to c* if
9
COMP24111 Machine Learning
Naï ve Bayes
• Naï ve Bayes Algorithm (for discrete input attributes)பைடு நூலகம்
– Learning Phase: Given a training set S,
Fo r eac h targ et value o f ci (ci c1 , , c L ) ˆ (C c ) estimate P(C c ) with examples in S; P i i Fo r everyattribute value x jk o f eac hattribute X j ( j 1, , n; k 1, , N j ) ˆ ( X x |C c ) estimate P( X x |C c ) with examples in S; P j i j i jk jk
1) P( A) ? 2) P(B) ? 3) P(C) ? 4) P( A| B) ? 5) P(C | A) ? 6) P( A , B) ? 7) P( A , C ) ? 8) Is P( A , C ) equals P(A) P(C)?
5
COMP24111 Machine Learning
Naïve Bayes Classifier
Ke Chen
COMP24111 Machine Learning
Outline
• Background
• Probability Basics • Probabilistic Classification • Naï ve Bayes • Example: Play Tennis
Naï ve Bayes
• Bayes classification
P(C |X) P( X |C )P(C ) P( X1 , , Xn |C )P(C )
Difficulty: learning the joint probability P( X1 , , Xn |C )