Neural Networks for Latent Semantic Analysis

Textual data ca n be represented by a usually large dimensional matrix. For tasks such as in­ for mation retrieval or i nfo rm atio n filter i ng, it is th e refo re necessary to red11 ce the dimension­
onal factor spanning the se m ant i c space is com­
[6].
This paper presents a neural network real­ ization of LSA. T he memory-space problem is
0-7803-6456-2/00/$10.00 ©2000 IEEE
1201
resolved by computing each of the orthogonal factors in a sequential fashion or quasi-parallel. The proposed neural network architecture and its learning rule for LSA are suited either for VLSI implementation [2J or for implementation on a Dynamic Data-Driven Multiprocessor ma­ chine [10] currentir being developed by our col­ leagues at KUT. Hence the computational-time problem could be resolved by parallel implemen­ tation. The remaining sections of this article are or­ ganized as follows. To make this article self­ contained, Section 2 gives a brief review of LSA. Section 3 describes the architecture and the learning algorithm of the LSA neural networks. Section 4 reports the experimental results. Dis­ cussions and conclusions are then given.
,
To find the underlying or "latent" structure in the pattern of word usage across documents, X is approximated by X as follows:
X;:::: X
=
TSD',
(3)
where the t x n matrix T, the n x n matrix S, and the d x n matrix D are composed of the first n columns of To, So, and Do, respectively. T his approximation allows the arrangement of the space to reflect the major a..'!sociative pat­ terns in the data, and to ignore the smaller, ran­ dom or less important influences. The resulting space is called a semantic space.
A word-by-document t x d matrix X is composed of elements Xij defined by
Xij =
(1) where t is the number of words, and d the num­ ber of document.s. To improve LSA's' results, several preprocessing techniques can be applied to X'ij, such as the. one in [8]. We, however, do not use them in this work.
proximated by linear combination of these fac­ tors. In addition, this set of factors spans a se­ mantic space wherein terms and documents that are closely associated are placed near one an­ other. In practice, however, implementation of LSA for large-size data sets can not be achieved with most numerical SVD algorithms, mainly due to enormous consumption of memory space and computational time
2.2
, I {
0,
if word i occurs otherwise,
in
document j
szm . (d d i )
,
=
(d, d;) IIdlilldill'
(5)
SVD
where d; is the ith row-vector in D, (d, d;) represents the dot product of vectors d and di, and II . II is the Euclidean norm operation.
ality. A statistical method called Latent Seman­ tic Analysis (LSA) can be used fOT' not only di­
both have the goal of retrieving information rele­ vant to what a user wants, while minimizing the amount of irrelevant information retrieved
Neural Networks for Latent Semantic Analysis
Ruck ThawoIlIu8S, Jun-ichirou Hirayama, Takayuki Tomoike, and Akio Sakamoto Department of Information Systems Engineering,
improve the retrieval performance, a statistical method called Latent Semantic A nalysis (LSA)
[8]
is often used. LSA uses singular-value decomposition (SVD)
mensional reduction, mensional,
but also for transforma­
[9J.
To
tion of the original space into a m'uch lower di­ but much more meaningful feature space, usually called a semantic space. However, LSA is known to consume a lot of both mem­
ory space and comp utational time.
This paper
to decompose a usually large word-by-document matrix into a set of much smaller dimensional or­
illustrates how to resolve these problems.
[1].
A textual database in information filtering or
retrieval systems can be represented by a word­
by-document matrix, whose elements represent the occurrence of a word in a document
185 Miyanokuchi, Tosayamada-cho, Kami-gun, Kochi 782-8502, Japan
E-mail: ruck@info.kochi-tech.ac.jp
Kochi University of Technology
1
Abstract
Introduction来自The greater availability of information in elec­ tronic form brought about a comparable growth in research on methods for information filter­ ing and information retrieval. These two tasks
.
Neu­
a
quasi p ara l le l fashion, namely only one orthog­