Chinese Word Segmentation

What’s a word? (Cont.)

Gao Jianfeng defines Chinese words in his paper as one of the following four types:
(1) entries in a lexicon words, (2) morphologically derived words, e.g.研究研究 (3) factoids, e.g.日期、时间、货币, etc

Agenda
Why Segment? Significant Problems, Classic Methods Overview of IRSEG The Mathematics Model Best Path/N-Best Paths Algorithm Unknown Words Ambiguities, Solutions
Agenda
Why Segment? Significant Problems, Classic Methods Overview of IRSEG The Mathematics Model Best Path/N-Best Paths Algorithm Unknown Words Ambiguities, Solutions
W * arg maxw P(W ) P(S | W )
The Mathematics Model (Cont.)

According to Law of Large Number
P( wi ) ki
k
j 0
M
j
ki represents how many times wi appears in samples for training. M represents the total number of words.

Agenda
Why Segment? Significant Problems, Classic Methods Overview of IRSEG The Mathematics Model Best Path/N-Best Paths Algorithm Unknown Words Ambiguities, Solutions
Classic Methods (Cont.)

Shortest-path Method
Find a shortest path in a directed graph of
words. No frequency An Example
的确 他 说 的 2 1 1 确 实 实在 在 5 1 确实 1 在理 6 1 理 7
What’s a word?

Standard
《信息处理现代汉语分词规范》GB13715 汉语信息处理词汇 GB12200[2]

Definition
WORD: 最小的能独立运用的语言单位。[2]
Words for segment: 汉语信息处理使用的、具有
确定的语义或语法功能的基本单位。它包括本规 范的规则限定的词和词组。
1 10 0 5 2 5 2 1 10 8 2 5 2 9 3 7 10 9 3
1 ∞ 6 10 0 2 5 2 1 8 9 3 7 ∞ 9 6 7 7 6 14
9
7 5
8 9
3 7
13 6
7
10
0 5
8
9 9 3 7 6 7 2
0
2
5
5
The Best Path
According to the maximum likelihood model, we add a weight to every edge of the graph with the formula:
The most probable sequence of segmentation
The Mathematics Model (Cont.)

According to the Bayes Formula
W * arg maxw P(W | S )
P(W ) P( S | W ) W * arg maxw P( S )
Chinese Word Segmentation
Liqi Gao, Zhuoran Wang IR Lab, HIT 2003-11-8
Outline
Why Segment? Significant Problems, Classic Methods Overview of IRSEG The Mathematics Model Best Path/N-Best Paths Algorithm Unknown Words Ambiguities, Solutions
(4) named entities,
Agenda
Why Segment? Significant Problems, Classic Methods Overview of IRSEG The Mathematics Model Best Path/N-Best Paths Algorithm Unknown Words Ambiguities, Solutions

The Mathematics Model (Cont.)

Following the maximum likelihood the combined probability is
P(W ) P( wi ) (ki / k j )
i 1 i 1 j 0 m m M

For convenience,
Chinese personal names commonly consist of a
monosyllabic surname followed by a bisyllabic given name.

Ambiguities
Intersecting ambiguity e.g.: 结合成 Combining ambiguity e.g.: 他将来北京
Lw
[ln ( k
i 0 j 0
m
M
j
M ) ln( ki 1)]
Then a Dijkstra algorithm is used to find the best path.
的确 他 0 1 说 2 的 3 确 4 确实 实 5 实在 在 6 在理 理 7
The sequence found is of the maximum probability.

N-Best Paths

An improved best path searching [Zhang 2001]
0
1
1
3 1
4
Classic Methods (Cont.)
Statistical-based MM Expert system based method Neural network based method Hidden Markov Model method Brill transformation based method ……
P * (W ) ln P(W ) ln P( wi ) ln(ki / k j )
i 1
i 1 j 0
m
m
M
(ln k j ln ki )
i 0 j 0
m
M
The Mathematics Model (Cont.)

A statistical-based directed graph

Classic Methods
MM (The Maximum Matching Method) Example: 计算机科学和工程 计算机科学和工 计算机科学和 计算机

Classic Methods (Cont.)

RMM (The reversal directional maximum matching method) [3]

Significant Problems

Dictionary
No dictionary is going to be complete. No dictionary can capture the vocabulary across all
domains.[1]

Unknown names and words
It demands a special dictionary, which is
reversal.

OM (The optimum matching method)
FOM,BOM It optimized the searching time of dictionaries
but the segmenting complexity.

Overview of IRSEG

IRSEG Word Segment of IR Lab
Split Creating Atom a Segment Directed Graph The Best Path
Output
Numerical Unknown Word Ambiguity Word Recogni- Recogni- Section tion tion N-Best Paths

Why Segment?

There are many applications where accurately segmented text is a necessity, or at the very least where it is useful to know where the correct word breaks appear in a text. [1] Information Retrieval Text Processing