Using knowledge to improve N-Gram Language Modeling through the MGGI methodology

1 Introduction
A well known result in Grammatical Inference (GI) states that learning Regular Languages (RL) using only positive data is undecidable 10]. As a consequence, in order to learn RLs, additional information, not contained in the training (positive) examples themselves, is required. This information may take the form of negative data 11] 16] 13], equivalence queries 2] or probabilistic information 5] 19] 3]. Alternatively, this information can consist of the knowledge that the target languages of interest belong to a certain subclass of RLs that can be shown to be learnable from only positive data (characterisable methods 1]). Yet another alternative is to make more direct use of a-priori knowledge about the kind of syntactic constraints that are desired to be captured in the learned models. This generally leads to so called heuristic methods. A step towards formalising this heuristic use of knowledge was introduced by Garcia et al. with the name \Morphic Generator Grammatical Inference Methodology" (MGGI) 8]. In this work we apply the MGGI approach to the problem of Language Modelling (LM). This problem has become central in the eld of Automatic Speech Recognition (ASR). Current state-of-the-art ASR systems model the language of each ASR application through the use of N-Grams. However, it is argued that
Fig. 2. Left: a model for the language (a++ d)b+ (c++ e) which is 2-TS, 2-T and 1-Revers-
Real situations in LM in which syntactic restrictions of the kind just illustrated apply are rather common. For instance, in modelling the language for a telephone exchange, sentences like \please, I wanted to talk to Mr. X", or \may I speak with Mr. Y, please" can be expected, but a sentence like \please, I would like to talk to Mr. Z, please", would be very unlikely. Any attempt to (deterministically or probabilistically) model this language constraint would require
?
Work partially supported by the Spanish CICYT under grant TIC95{0984{C02{01

Other Classes of Languages Context-Free Regular TDR k-T
k-TS
k-Rdef
k-Def
(k-1)-Rev
b 5 b 2 b 4 a 1 d 2 b b 3 c e 4 1 d a 3 b e c 6
Fig.1. Some families of Languages.
ible. Right: a model for the language (ab c)+(db e) which is also 2-T and 1-Reversible but can not be modelled by a k-TS model (k-Gram), for any possible k.
Enrique Vidal and David Llorens e-mail: evidal@dsic.upv.es
Departamento de Sistemas Informaticos y Computacion Universidad Politecnica de Valencia SPAIN
Proc. of 3rd ICGI, Lect. Notes in Artificial Intelligence, Vol.1147, 1996. Copyright protected: only personal use is permitted
Using knowledge to improve N-Gram Language Modelling through the MGGI methodology
signi cant modelling improvements are required to overcome the most important limitations of current systems 12]. As we will see later on, languages described by N-Grams are equivalent to the so called stochastic k-Testable Languages in the Strict Sense (k-TS languages). This well known class of languages happens to be a very restricted subclass of (stochastic) Regular Languages that can only model local or short-term language constraints (Figure 1). Even if we restrict ourselves to the class of Regular Languages (RL), many families or subclasses of languages exist that are signi cantly more powerful than k-TS/N-Grams, in the sense that they can help modelling more global or long-term constraints. Moreover, in many cases, adequate techniques do exist to learn the structures of the corresponding models from training sentences of the unknown target language 20]. However, learning structures that are signi cantly more complex than those underlying k-TSs may require large amounts of training data since all the relevant, perhaps long-term relations among the data that entail the corresponding syntactic restrictions have to be \shown" to the learning algorithm. Clearly, this may result prohibitive in many practical situations. The following example illustrates this fact along with the limited nature of k-TS/N-Gram models to capture certain syntactic constraints of interest. Example 1 The regular language (a + d)b+ (c + e) (Figure 2 left), is 2-TS (2Gram), 2-T and 1-Reversible. Conversely, a language such as (ab+ c) + (db+ e) (Figure 2 right), which is also regular, 1-Reversible and 2-T, can by no means be adequately modelled by a k-TS model (k-Gram), for any possible k. t u