1_Computer Evaluation of Indexing and Text Processing

2. The S M A R T System
A. BASICORGANIZATION. The SMART system is a fully automatic document retrieval system operating on the IBM 7094. The system does not rely on manu~dly assigned keywords or index terms for the identification of documents and se~trch requests, nor does it use primarily the frequency of occurrence of certain words or phrases included in the document texts. Instead, tile system goes beyo~ld simple word-matching procedures by using a variety of intellectual aids in the form of synonym dictionaries, hierarchical arrangements of subject identifiers, phrase generating methods, and the like, in order to obtain the content identifications useful for the retrieval process. The following facilities incorporated into the SMAt{T system for purposes of document analysis are of principal interest: (a) a system for separating English words into stems and affixes, which can be used to reduce incoming texts into word stem form; (b) a synonym dictionary, or thesaurus, used to replace significant word stems by concept numbers, each concept representing a class of related word stems; (c) a hierarchical arrangement of the concepts included in the thesaurus which makes it possible, given any concept number, to find its "parent" in the hierarchy, its "sons," its "brothers," and any of a set of possible cross-references; (d) statistical association methods, used to compute similarity coefficients between words, word stems, or concepts, based on co-occurrence patterns between these entities in the sentences of a document, or iu the documents of a collection; associated items can then serve as content identifiers in addition to the original ones; (e) syntactic analysis methods, which are used to generate phrases consisting of several words or concepts; each phrase serves as an indicator of document content, provided certain prespecified syntactic relations obtain between the phrase components;
AI~STRACT. Automatic indexing methods are evaluated and design criteria for modern information systems are derived. K~Y WORDS AND PtmASES: information retrieval, indexing methods, automatic retrieval, information systems, document retrieval, text analysis, document handling, retrieval effectiveness, SMART, precision, recall ca CATEGOm~S: 3.70, 3.72, 3.74, 3.79
Journal of the Association for Computing Machinery, VoL 15, No. 1, January 1968, pp. 8-36.
Computer Evaluation o] Indexing-and Text Processing

The present study summarizes the results obtained with the SMART system over a two year period starting in 1964, and presents evalu~tion output based on the processing of three document collections in three different subject fields. Conclusions are drawn concerning the most likely analysis methods to be implemented in an operationM environment. The emphasis throughout is on text analysis procedures, since they form au finportant part of a document handling system. Sever~tl operational problems, including the actual network implementation of a retrieval system, are not covered; cost and timing estimates are also excluded, because these are t.ied directly to the specific environment within which a given system actually operates. The basic features of the SMART system are first described, and the design of the main experiments is outlined, including the statistical procedures used to test the significance of the evaluation output obtained. The principal ewduatiou results are then presented, and tentative conclusions are reached concerning the effectiveness of automatic text anMysis procedures as part of future information systems. The results derived from the present experiments are also briefly comp~tred with the output obtained with several other testing systems.
Computer Evaluation of Indexing and Text Processing
G. SALTON
Cornell University,* Ithaca, New York
AND M. E. LESK
Harvard Universily,~ Cambridge, Massachusetts
1.
Introduction
Throughout the technical world, a growing interest is evident in the design and implementation of mechanized information systems. Over the last few years, the general feeling that something should be done to help organize and store some of the available information resources has given way to the widespread impression t h a t modern computing equipment may in fact be capable of alleviating and solving to some extent the so-called information problem. Specifically, it is believed that the required capacity exists to store m a n y data or document collections of interest, that procedures are available for analyzing and organizing the information in storage, and that real time software and hardware can be used to ensure t h a t the stored information is retrieved in response to requests from a given user population in a convenient form, and at little cost in time and effort [1-3]. Before investing the necessary resources required for the implementation o f sophisticated information services, it becomes necessary to generate the detailed system specifications and to determine which of many possible alternative design features should in fact be implemented. This, in turn, must be made to depend on experimentation in a controlled environment to test and evaluate the effectiveness of various possible search and analysis procedures. The S M A R T document retrieval system, which has been operating on an IBM 7094 for over three years, has been used extensively to test a large variety of automatic retrieval procedures, including fully automatic information analysis methods, automatic procedures for dictionary construction, and iterative search techniques based on user interaction with the system [4-7]. * Department of Computer Science. Division of Engineering and Applied Physics. This study was supported i~ part by the National Science Foundation under grants GN-36 and GN-495.