Automatic Analysis and Indexing of Variable-Layout Documents

Automatic Analysis and Indexing of Variable-Layout DocumentsEnrico Appiani & Anna Maria CollaElsag spa, RES DepartmentVia G. Puccini, 216154 Genova, Italy{enrico.appiani, annamaria.colla}@elsag.itAbstractIn this paper a methodology for analysis and automatic indexing of imaged documents within an archiving and retrieval system is described. This system, which is being developed within the Esprit project STRETCH (STorage and RETrieval by Content of imaged documents), is based on a new generation Archiving and Retrieval Engine (ARE), which overcomes the bottleneck of document profiling by alleviating the existing limitations of pre-defined indexing schemes. The ARE exploits a structured document representation and can activate appropriate methods to characterise and automatically index heterogeneous documents with variable layout.The objective of this system is twofold. First, it aims at combining direct digitalisation, based on location of information fields and OCR, with advanced techniques derived from Image Analysis and Pattern Recognition. Second, it aims at offering ease of use and programming and ability to dynamically adapt to new types of documents.After experiments in some other document domains (i.e. invoices, medical images, circular letters), the system has been tested on bank documents. In this application, several classes of documents are involved. The indexing strategy first automatically classifies the document, thus avoiding pre-sorting, then locates and reads the information pertaining to the specific document class. Experimental results are encouraging.1. IntroductionWe describe in this paper a method for analysis and automatic indexing by content of imaged multimedia documents, implemented in an archiving and retrieval system under development within the Esprit project STRETCH (STorage and RETrieval by Content of imaged documents).One of the problems STRETCH addresses is that of designing document archives for documents other than forms. Forms are documents where the information can be located precisely, given an appropriate registration of the document image. The layout of a given set of forms is «conformal», i.e. it adheres to one or few specific models. It is usually rather easy to tune OCR software to read forms efficiently. However, in several practical applications, it is necessary to process documents with many different layouts, where information necessary to index the document is located in different positions. In order to read the relevant information for automatic indexing it is necessary to «understand» each document layout to classify the document and select the appropriate reading strategy. Even the most effective commercial OCR/ICR systems are unable to perform this task in an adequate way without expensive tuning for document profiling, required to locate and extract the pertinent information. Even more important, systems for different applications (i.e. different document domains) must currently be specialized individually: it is at least hard, if not impossible, to adapt one system to different applications, to obtain acceptable performance. It is similarly hard for end-users to define their own applications based on current systems, which are deeply tailored to a given application to obtain adequate accuracy. Therefore, indexing of a mixed batch of documents to create and maintain large archives is not adequately supported by state-of-the-art systems, and requires heavy intervention by skilled users, application experts and software engineers.STRETCH system aims at overcoming the limitations of the currently available systems, introducing reliable automatic indexing. This can be directly guided by users who are enabled to define the document domains of their interest in a straightforward way requiring no special skills. The business is potentially very conspicuous in several fields (banks, insurance companies, corporate databases, cultural institutions, public administration,…), where huge volumes of scanned documents must be archived efficiently.The system designed and implemented in STRETCH is based on a new generation Archiving and Retrieval Engine (ARE), which exploits a structured Object-Oriented document representation. The objects are enriched with appropriate methods to locate and automatically extract information to index heterogeneous documents with variable layout. From an architectural viewpoint (Appiani et al., 1999) the system relies on a distributed client/server solution, with standard Corba middleware. The core technology employed is document processing in terms of layout analysis, automatic field location, logo and tag identification, Intelligent Character Recognition (ICR). After layout analysis, the document is classified according to user’s specification and information contents are extracted from the relevant fields.The system has been tested in various domains: invoices (Appiani et al., 1999), medical images, public administration circular letters, and bank documents.The paper is organized as follows. In Section 2 the data model is introduced. Section 3 describes the system architecture, in particular the server components, with some details on the parallel computing strategy. Section 4 summarises the strategy for automatic document indexing. Finally, in Section 5 some experimental results in the domain of banking applications are discussed and in Section 6 some conclusions are drawn.2. Knowledge and Document RepresentationAny document can be described with respect to physical and logical aspects. The physical structure of a document, i.e. the layout structure, is the collection of the extracted objects, obtained by the repeated partition of the document image into increasingly smaller parts (physical objects), on the basis of the layout appearance. Similarly, the logical structure of a document is the collection of the extracted objects, obtained by the repeated division of the document content into increasingly smaller parts (logical objects), on the basis of human understanding of the «meaning» of the content.A domain of documents can be defined as a group of documents which can be clustered with respect to their subject or use, according to users’ view: for example, journals, tax forms, business letters, invoices, check forms, generic bank documents can be regarded as different domains. Since the documents of a domain share the same main subject and are used for similar or related functions, they are characterized by some logical and physical similarities. Some logical objects are common to all the documents of the domain. For example, in an invoice the logical object «Total», which contains the amount due to the issuer, is always present. Similarly, a logical object can maintain some physical features related to its position within the document also in different types of documents of the same domain. Documents belonging to a given domain can be further characterized by different layout or logical structures. Thus documents which feature physical or contextual similarities can be clustered into classes.2.1. Document Internal RepresentationIn STRETCH, the internal data structure to represent documents, the so-called Document Internal Representation(DIR), is defined in terms of «class objects», with their relevant information and methods to process such information. The DIR objects describe a document according to physical and logical viewpoints.For most applications our choice is to base the DIR on Modified X-Y trees, or M-X-Y trees (Cesarini et al., 1999). The M-X-Y tree is derived from the X-Y tree, a data-driven method for page layout analysis. The basic assumption behind this approach is the fact that structured elements of the page (columns, paragraphs, text lines) are generally laid out in rectangular blocks. These blocks can be separated by alternately making horizontal and vertical «cuts» along either white spaces, found by using the thresholded projection profile, or horizontal or vertical ruler lines. The result of such segmentation can be represented as a tree, where the root is for the whole page, while the leaves are for the found blocks. The tree structure is enriched with descriptions of inter-leaves relationships (adjacency links).A document can be classified by comparing its M-X-Y tree representation with pre-stored representations of documents of known class.2.2. Domain KnowledgeTo implement Domain Knowledge, relevant to specific documents and types of physical structures, we make use of a template structure named correlation graph (Boato et al., 1997), which allows the implementation of information extraction strategies based on image processing and reading techniques. The correlation graph describes a document understanding strategy which computes the search area forfields to be read, possibly based on the position of other already found fields, and then uses some predefined template elements to «read» the field contents inside the search area according to the field type (string, graphical element, date, total amount, etc.). Meaningful fields are of three main types: (i) fields to be read as ASCII strings by the recognition strategy; (ii) textual or geometric tags used by the recognition strategy to understand the document structure; and (iii) image fields that can be recognized by suitable methods (for instance logos). Each information field is assigned a basic type that defines how its value is interpreted during retrieval. For example a date field older than a certain threshold can be searched, as well as a string field similar to a certain word. A set of tags that have a significant spatial relation with information fields is internally employed by the reading strategy. The search areas for the information fields of a given document class are automatically learnt upon the presentation of appropriate document examples, and can be further refined by the presentation of new examples.The implementation of Domain Knowledge corresponds to the extraction of the indexes suitable to represent the documents in the archive. This document understanding and indexing strategy is described in more details in Section 4.3. STRETCH architectureThe STRETCH system is composed by a Client and a Server subsystems. Both are made of Java or native components, using Corba middleware between them for naming, location and remote method call. Beside evolving as a standard middleware, Corba in STRETCH allows for distributed configurations with dynamic Server resource allocation.The Client is entirely defined in Java for maximum portability and presents a few screens to perform the main archiving and retrieval functions, as well as to browse the entire document archive or to display and edit individual documents. The menus are essential and intuitive, not requiring particular skills in document management systems.The Server includes both Java and native components, the latter used for image processing and OCR functions. Despite Corba’s possibility to link components defined with different languages, Java has been employed to define all the front-end Server interfaces, while the pieces of native code are loaded and executed as internal resources for document processing. In fact, with its dynamic class loading and portability, Java has proven to be also a good «wrapper» for native resources, to be easily changed in case of porting the STRETCH Server to different platforms. The current STRETCH Server platform is Windows NT 4.3.1 Server ComponentsThe main components are presented in Figure 1; the upper part represents the entire Client set, while the lower part contains the individual Server components and their relations, in particular for the DIR objects exchanged between components during operation. In the same figure and in the following description, the DIR class names are indicated in Italics.The Naming Server is a Corba service pointing to the Server instances, whose logical names are used to The Docu-Base is the first component created to start a STRETCH Server and is connected to thedifferent databases, to enforce the correlation among the document structures and their respective components, basically text and images for scanned documents; and validating new session requests, creating dedicated ARE Manager instances.The ARE Manager is the Client front-end, exporting methods to archive and retrieve documents. It takesentire sets of images are submitted to the system for unsupervised archiving. In this case, the ARE Manager also enforces the scheduling strategy for parallel document processing (see next).On archiving, concurrent threads can be started by the Client; thanks to Corba, they result in corresponding server threads. Each thread: schedules a processing resource, waiting for its availability; calls it to process an Image; waits for the resulting Page, containing the classification, segmentation and reading results; and, if the Page is valid, calls the Docu-Base to archive it in a Document instance; finally, it returns the document identifier to the Client.On retrieval, the ARE Manager gets a high level query expression, including class-, date-, name-, text-and image-based atomic subexpressions, and calls the Docu-Base to combine them in a coordinated query, extracting a list of document identifiers, which are then returned to the Client with relevance-based or field-based ordering criteria. It is also possible to retrieve a specific document, i.e. for browsing and editing, by getting its identifier and returning a Document instance.The Document Processing Server (DPS) is a specialized object to process Images and return Pages, byInheriting from an abstract class offering the generic Corba interface for the above function, the DPS employs the Domain Knowledge by wrapping the Domain-specific resources and executing the related internal methods. Specialized DPS instances can be created in any number, either on the same node to exploit parallel processing or on different nodes for distributed processing. The scheduling strategy is presented in more details below.Figure 1: The STRETCH Components.3.2 Scheduling Strategy for Parallel Document ProcessingThe STRETCH Server scalability can deal with the processing power during the indexing phase (number of Clients connected – batch size from any Client), the retrieval power and the archive size. The latter two aspects are essentially related with the database machine(s) connected to the Docu-Base, since query processing and database coordination are not as computing intensive as query execution inside each database. Any relational database compliant with the ODBC interface can be used.The processing scalability has been enforced by allowing for DPS replication on the same server machine and/or on different ones. The main Corba commercial implementations include advanced object scheduling and load balancing services; however, in order to limit the STRETCH requirements to the « basic » Corba implementation, embedded in Java 2, we have implemented a simple yet effective DPS scheduling policy, achieving a load balancing based on the actual processing power of different DPS instances and graceful degradation in case of DPS crashes.When starting a batch archive, the ARE Manager re-checks how many DPS objects, specialized for the current Domain, are registered in Corba; this way, dynamic reconfiguration of a STRETCH Server during operations is allowed. During the batch, every thread gets an available DPS to process its respective Image, or is put in a waiting queue by means of Java inter-object synchronization. Every time a DPS terminates its processing, it performs a « notify » operation to the ARE Manager, that can schedule a new thread for execution.The wait-notify mechanism could be replaced by a much simpler synchronized execution of DPS processing method, achieved by a Java keyword in the method declaration. However, this way all the incoming threads should choose their serving DPS instances initially, for instance with a round-robinpolicy, and therefore wait on their respective « monitors ». Each DPS would receive nImg/nDPS processing requests on average, where nImg is the number of images to be processed in a batch and nDPS the number of available DPS instances, thus implementing an a-priori load balancing not taking each actual DPS power, especially with distributed configurations, into account. With wait-notify upon DPS availability, instead, each DPS receives as many requests as it can process; in case of problems while processing a batch, only the Image being processed by the faulting DPS does not return a Page for archiving, while the rest of the job is finished correctly at a lower speed.The execution of each DPS in a separate Java Virtual Machine (JVM), even when running in the same machine, allows for effective resource separation and exploitation of parallel machines. Figure 2 presents a sample archiving execution flow involving one Client (GUI), one ARE Manager (AREM) and three DPS instances. The DPS component represents the ARE Manager scheduler for specialized DPS instances, called DPS_Resource. The above described solution for parallel and distributed processing proved to be successful, even if it was obtained by just using Java and Corba, not particularly suited to parallel processing by their intrinsic properties and mechanisms.Figure 2: Multi-threaded and Parallel Document Processing.4. Indexing strategy4.1 Domain Knowledge implementationDomain Knowledge implementation in STRETCH is based on a flexible framework for document indexing, endowed with a learning tool, which, starting from samples suppied by users, automatically generates the software code to solve the application.The tool is an extension of a graphic environment for the development of document reading applications, that allows the user to define complex application-level recognition strategies in a declarative way. The automatic programming extension has three components:• a graphical interface that allows users to locate the relevant fields on a set of document examples,• a learning algorithm that performs a classification of the examples and detects the spatial cross-correlation between fields,• a synthesising algorithm that generates the Java code of the objects of the application.The learning algorithm performs a clustering of the input events (i.e., the examples marked up with the relevant fields) based on the minimization of the information required to describe the examples themselves. The number of classes is not predefined: it is learnt from the examples along with the structure of the classes.The automatic programming tool can learn from examples to automatically generate reading applications for both documents that have variable layout and a spatial structure, and documents whosesemantics can be recognized by looking for textual tags in the image, and also fixed-layout forms. The user has to prepare a training set for the learning algorithm, based on a set of positive examples of document images (input events). Different events can have a different number and different types of attributes. Event attributes include the position (surrounding rectangle) of the fields on the image, the field type, and other specific parameters (e.g., the dictionary of textual tags).After the input events have been classified, a recognition strategy is synthesised using pre-defined building blocks which implement the following functions:•locate the fields to be read by computing the search areas, possibly based on the position of already found fields;•extract the field content inside a search area on the image, taking into account the field type (read text, recognize images such as logos).The program synthesis algorithm also finds the optimal ordering of classes and of fields inside classes to minimize backtracking.4.2 Application to Bank documentsIn this section we describe in details the customized Indexing Strategy for Bank Documents. In this application, several classes of documents among those handled by branch cashiers are involved (accounting notes, cheques, document batch headers, different types of enclosures). For each document class, the set of the relevant information fields is variable, consisting of different and differently located couples of tag/information fields. The application aims at classifying the type of document and consequently applying the proper reading strategy to extract the information pertaining to the right document class. The implemented strategy includes three steps for each document:•first the DIR is generated, in terms of the M-X-Y tree of the input document;•then the classification procedure based on the M-X-Y tree produces the list of candidate classes for the document; in case of uncertain classification (certain types of enclosures cannot be distinguished from accounting notes by layout only), appropriate tags are searched;•last, the appropriate reading strategy for the given class is applied, which is based on ICR techniques including field location, tag finding and text reading.The goals of the application are:•Classify the documents according to their type, in particular accounting notes (class «CONTABILE»), checks (class «ASSEGNO») and batch headers (class «TESTAPACCO»); all the other documents are generically regarded as Enclosures.•For documents of class «CONTABILE» locate and read a specific information field, corresponding to the operation number.•For documents of class «ASSEGNO» locate and read a specific information field, corresponding to the check number.Each information field is characterized by a name, a type (associated to one leaf of the Strategy, i.e. the basic method to extract the relevant information from the field) and a number of parameters with various functions in the Learning procedure. Various basic methods can be employed by a leaf; for instance, the leaf for reading printed text can make use of different OCR’s (we have actually experimented both in-house developed and commercial solutions).To achieve the application goals, a top-level Strategy has been designed as depicted in Figure 3, where notation is as follows:is a template, implemented by means of a Java class, whose methods are capable ofextracting information from a given field type;input variable, characterized by its type;output variable; andis a method, which executes specific computations.The strategy includes the document classification template DOCTYPE, which can accept an external hint (input variable Input_doctype). According to the results of document classification, either the strategy to read CONTABILE or the strategy for ASSEGNO is activated. If the input document belongs to neither class, no reading strategy is executed, just the information about the document type is saved (output variable doctype). The Strategy for reading a specific class is a choice among sub-strategies to locate and read the interesting pieces of information in a given area of the document image. These areasare automatically defined by the learning strategy which, upon presentation of marked examples,computes the overlap of the locations in the single examples where a given information field has been marked. Each sub-strategy thus generated is an element of the choice . The full Strategy executes the various elements sequentially, following an automatically assigned priority which reflects the overall strategy optimization, until one sub-strategy is successful; if none is, the Strategy fails. The output of the reading strategy is a string containing all the extracted information, which is properly decoded by method parse to extract the required index (operation number for CONTABILE, check number for ASSEGNO ) and its location in the document image for visual presentation of the results.Figure 3 : Top-level Strategy for Bank Documents.5. Experimental resultsIn the preliminary experiments, the test set consisted of 541 heterogeneous documents issued by a large Italian bank group. These documents belong to any of the possible classes listed in Section 4.2and show several different layouts, where information fields are located with high variability.Digital binary (black and white) images were produced by high volume scanners, with 200 DPI x 200 DPI resolution. No specific filtering or enhancement was applied to the images. Two different commercial OCR’s were tested with the system for text reading.The document classification results, summarized in Table 1, are quite encouraging: the M-X-Y tree-based classification achieved 99% document classification accuracy 1 over the 3 main classes, 91%taking also all the enclosures into account. By adding classification confirmation based on tag recognition, 100% on the 3 main classes and 96% on all the documents was achieved. Table 2presents the reading results with the two different OCR’s. With OCR1, the information fields were correctly located in 96% cases. With both OCR1 and OCR2 the errors in reading the field values1Accuracy is defined as the fraction of non-rejected documents which are correctly processed.Type: Type: Variable: STRINGVariable: result Type: STRINGVariable: Type: AREAType:were partly due to lack of context processing (numeric-only fields were not forced, hence alphanumeric characters were confused, such as « S » for « 6 », « B » for « 8 », and so on). Serious problems were mainly encountered with very noisy or corrupted images, which are very common in this domain because of poor printing quality: in fact, the documents are always second copies of the original (which is given to the Bank customers). To solve this problem some appropriate image pre-processing appears mandatory.Total Accuracy Correct Error Reject Confusion Matrix Class#%#%#%Test.Ass.Cont.Encl.RejTestapacco19 1.0019 1.0000.0000.0019000Assegno360.97350.9710.0300.0003501Contabile2770.992730.9930.0110.001027321Enclosures2090.761340.64420.20330.162132713433Total5410.914610.85460.09340.06224830013734Table 1: Results on Bank documents: document classification.Reading OCR 1OCR 2Assegno Contabile Total Assegno Contabile Total Correct1911113030721021 error104353158592 errors4374112627≥ 3 errors---757533942Inserted char.112---77Field not located2101217576Total3627731336277313Table 2: Results on Bank documents: field location and reading.6. ConclusionsIn this paper we have presented a novel approach for classification and information extraction out of heterogeneous documents, based on a combination of layout analysis and automatic information field location. In the preliminary experiments on bank documents, very encouraging results have been obtained; in particular, document classification accuracy is satisfactory. An interesting prospective application of this methodology is to avoid document pre-sorting in the production line of service bureaus in charge of document acquisition and storage, thus greatly reducing costs and improving performance.AcknowledgementsWe would like to acknowledge Sandra Bruzzo, Pietro Pedrazzi and Christian Pisani (Elsag) for their valuable support in implementing and testing STRETCH.Bibliographical ReferencesAppiani, E., Boato, L., Bruzzo, S., Colla, A.M., Davite, M. & Sciarra, D. (1999). «STRETCH»: A System for Document Storage and Retrieval by Content, Proc. DEXA ’99, Workshop W07: DAUDD’99, Florence, Italy, Sep. ’99, 588--592.Boato, L., Cattani, E., Davite, M. & Villa, B. (1997). «Automatic Programming of Variable Layout Image Documents Reading Applications based on Minimum Description Length Induction», AI*IA Workshop on Automatic Learning and Natural Language, Turin, Italy, Dec. ’97Cesarini, F., Gori, M., Marinai, S. & Soda, G. (1999). «Structured document segmentation and representation by the modified X-Y tree», Proc. ICDAR ’99, Bangalore, India, Sep. ’99, 563--566。