Interactive evaluation of the Ostensive Model, using a new test-collection of images with m

Interactive evaluation of the Ostensive Modelusing a new test collection of images with multiple relevance assessmentsIain CampbellComputing ScienceUniversity of Glasgowiain@In Press, to be published in: Journal of Information Retrieval, Vol II, Issue 1, 2000.AbstractThe Ostensive Model proposes a manner of structuring the uncertainty associated with individual relevance judgements as sources of evidence in relevance feedback. It proposes temporal profiles of uncertainty, motivating the application of a particular class of discount function with respect to the age of the evidence. This paper presents an initial evaluation of the relative effectiveness of different uncertainty discount functions.A novel direct manipulation interface to a multimedia retrieval system embodying the Ostensive Model is outlined briefly. The paper describes the construction and characteristics of a new image test collection utilising multiple binary relevance assessments. The use of such multiple assessments and multiple interpretations of them are discussed. The evaluation environment is detailed in terms of the interface, test collection, and tasks set to users. Multiple interpretations of the results, and the statistical significance of comparisons are presented. The results obtained in the evaluation are consistent with the proposals of the Ostensive Model – reinforcing a particular evidence profile. The results give clear pointers to further, more specific, evaluations.1 IntroductionThis paper describes an initial evaluation of evidence discount functions that form a central part of the Ostensive Model (OM). It introduces the construction and characteristics of a new image test collection containing multiple binary relevance assessments. It also introduces a novel direct manipulation interface that provides an environment for multimedia information. Both the test collection and the interface were pre-requisites for the evaluation of the evidence discount functions as the OM was developed primarily for use in interactive multimedia environments.The OM and the interface will be presented first. They are presented only to an extent sufficient to allow an understanding of the nature of the evaluation. They will be followed by the main thrust of this paper – i.e. the test collection, the evaluation tasks, and the evaluation results obtained.The next two sections give short outlines of two components of the evaluation:•Section 2 outlines the OM and, in particular, its evidence discount functions that will be the subject of the evaluation.•Section 3 outlines the interface that embodies the OM to provide a query-less, multimedia, browsing environment.The main body of the paper then follows:•Section 4 describes the test collection, how it was constructed, and some of its characteristics.•Section 5 describes how the components come together with user-tasks to form the interactive evaluation.•Section 6 discusses and multiply interprets the evaluation results.Finally, Section 7 offers a summary and conclusions.Throughout this paper, I will talk of the interaction with ‘documents’. I do not restrict this to the traditional (in IR) view of documents being only text – rather, I use it to mean any single or multiple media object retrievable by a system. I restrict the obvious alternative term ‘object’ to discussions relating to document surrogates presented in the user-interface.2 The Ostensive ModelThe Ostensive Model (Campbell 1996) is a result of the perceived complementary nature of query-based and browse-based approaches to information seeking. It is a model that (like query-based approaches) allows directed searching, but (like browse-based systems) allows the seeking to be performed within an environment where there is no need to describe information needs. The removal of queries means that users can interact directly with documents, not the system’s internal representations of them.2.1 Querying, browsing, and ostensionTo offer directed searching within a browsing environment, a method is required to capture a representation of the instantaneous nature of the current information need, and use it to shape the onward path from any point. The OM provides an approach to capturing the intentionality of an information need that is assumed to be developing during the searching session. The capture of such spatio-temporal objects is inherent to the concept of Ostension (Quine 1953 & 1969).The Oxford English Dictionary (1993) defines an ‘ostensive definition’:ostensive definition (Philos.), the explanation of a word by presenting, pointing at, or otherwise indicating, one or more objects to which it applies.The classic example is that of communicating a definition of a colour. Adults might point to or hold up items to a child and say the word ‘red’. There is no description taking place, only ostension. In the OM, ostensive definitions are formed of an information need (c.f. “a word”) by combining evidence from documents (c.f. “one or more objects to which it applies”) that the user has indicated as relevant (c.f. “pointing at, or otherwise indicating”). The dynamic nature of such definitions can be exemplified in the changing nature of the child’s conceptualisation of red. As he grows older, and is presented with more examples, and examples that are more sophisticated and subtle, his ostensive definition of red may change to, for example, exclude hues of red that we regard as ‘pink’. Another example is the ostensive definitions we build of what kinds of clothes other people like to wear, simply by observation of repetition and gradual change in what is worn.Relevance feedback, as we are familiar with it in IR, is a form of ostension. The essential difference here, is that the OM, as it stands currently, adds temporal information to the relevance indications. Put simply, the Model motivates a view that favours more recent indications of relevance over older indications. It regards recently indicated documents as being more indicative of the current information need. This results from the explicit assumption that the information need (that causes the relevance indication) is developing – i.e. it is changing constantly as a result primarily of the user’s exposure to information in the encountered documents. It defines Ostensive Relevance:The Ostensive Relevance of an information object is the degree to which evidence from the object is representative/indicative of the current information need. (Campbell 1996)This inherently probabilistic conception of relevance, with its explicit reference to the current information need, accepts the varied and, more specifically, the changing strength of pieces of evidence. Further, with its reference to the representative/indicative nature of evidence with respect to the information need, it accepts that evidence is merely a set of clues from which to build a representation of the instantaneous information need. One could say that Ostensive Relevance is a particular (system-rooted or observer-rooted) example of ‘Pertinence’ where “the matter in hand” (Saracevic 1996) is the building of the representation.Ostensive Relevance takes a step back from saying that an object is, was, or ever could be objectively relevant. It says that relevance is a utility consideration with respect to how a piece of evidence is used or combined. That is, how useful a piece of evidence is to the task in hand – in this case, the building of an effective representation of the information need.The OM has a model of the iterating development of knowledge states within the mind of the searching user. It has a sequence of snapshots of the knowledge states responsible for individual indications of relevance. It states that each relevance-indication has a high Ostensive Relevance at the time of it occurring. As a knowledge state will be modified as a result of exposure to the relevant information, the information need embodied within it will also change. As the knowledge state (and therefore the information need) changes, the OstensiveRelevance of already-indicated documents will decrease. This motivates the idea of a profile of the change in Ostensive Relevance with respect to the age of the evidence.There has been recognition made of the dynamic nature of information needs, and cases made for its adoption as the norm (Belkin 82, Bates 89, Ingwersen 1996, Borlund 1998). Nevertheless, clear intuitions of how such dynamism might be incorporated into a retrieval algorithm in order to improve its effectiveness have remained illusive. One exception is the idea of Polyrepresentation (Ingwersen 1994, 1996) which, when extended from merely spatial to spatio-temporal, can be seen to be consistent with the OM. The OM offers such an intuition, and this paper is an initial evaluation of its appropriateness.2.2 Ostensive Relevance profilesIn Campbell (1996), the profiles were motivated in terms of uncertainty, before being translated ultimately (by simple inversion) into Ostensive Relevance. Here, only Ostensive Relevance is used – it appears to be more intuitive, and discussions around it easier to follow. Three of the Ostensive Relevance profiles discussed here will be the subject of the evaluation. The profiles are shown as graphs, with Ostensive Relevance increasing upwards on the vertical scale, and ‘age’ increasing leftwards on the horizontal scale (i.e. time increasing rightwards). The vertical axis corresponds to the current document.2.2.1 The “document in context” profileThe OM suggests a profile where the Ostensive Relevance of a relevant-indicated document decreases with age. In particular, a Fibonacci series where the Ostensive Relevance of any object is equal to the sum of that of all of its predecessors, gives a profile of the general form:OstensiveRelevanceAge of evidenceThe intuition is that with such a profile, the importance given to a document (when building a representation of the current information need) is equal to that given to its ‘context’. The particular profile described here can be regarded as a turning point in the relationship between document and context. A flatter curve would give more importance to the context; a steeper curve, more importance to the document.2.2.2 The “document only” profileIf one were to take the steepness of the curve to an extreme, one would get a step function where zero Ostensive Relevance is given to the route and maximum to the current document. This suggests that the information need is assumed to be changing radically in response to the current document, and that the form of the information need is wholly unrelated to any previously encountered documents. This would correspond to the models inherent in traditional IR browsing systems following the pattern of I3R (Thompson 1989), where only the current document is used to determine the next-steps available to a user:OstensiveRelevanceAge of evidence2.2.3 The “context biased” profileIf one were to flatten the curve completely, it would correspond to the model inherent in traditional relevance feedback as used in iterative query-based systems. It is assumed that all relevant-indicated documents have been indicated with respect to a single objective information need. There is no notion of the ageing of those indications of relevance. Over time, the influence of the context becomes stronger and stronger, overwhelming the effects of any new documents.RelevanceAge of evidence2.2.4 The “context only” profileThe true form of a context-only profile would have a step function with only the first document having any connection with the information need – all other indications being regarded as purely random with respect to the information need. This profile would result in a degenerate situation where users were trapped at their starting positions in the document space, with each step ‘away’ only showing the same document over and over again.This profile can be relaxed slightly into an increasing-with-age profile. This corresponds to the conception that each new document indicated as relevant by the user has progressively less effect on the information need. That is, the information need was well formed initially and each additional relevance-indication is being made with less and less respect to it.RelevanceAge of evidenceAs with the context-biased profile, the influence of the context becomes increasingly strong, but the rate of it overwhelming any new documents is exaggerated, with new documents having no significant effect after only a couple of steps.2.2.5 Three profiles for evaluationThe evaluation will compare three profiles which, for brevity, will be referred to using the following acronyms: CO: A relaxed form of the context only profile – intended to provide a form of lower bound to the discussion.CB:The flat context biased profile – intended to determine the utility of the model associated with traditional relevance feedback.DIC:The balanced document in context profile – favoured by the Ostensive Model.3 The retrieval system and user interfaceAs in the previous section, this description of the user interface is restricted to an extent sufficient to appreciate the evaluation environment.3.1 The components of the interfaceThe visual presentation is that of a surface with objects on it and links between those objects. There are four major concepts: A ‘starting point’, a ‘current object’, a ‘path’, and a set of ‘candidate next-steps’. The starting point is where the browsing originated. The current object is the one currently the centre of the user’s focus, and around which candidate next steps are displayed. In an effort to reduce clutter, the candidates are only shown around the current object. The path is the sequence of objects from the starting point to the current object. The user is free to select any object at any time as the current object.In classic browsing systems, such as I3R (Thompson 1989) and more recent browse/visualisation systems (e.g. Agosti 1994, Chalmers 1992, Rennison 1994), the particular objects to which one can move from a given object are based upon their similarity to the current object. The difference lies in that the criteria used to select these candidate next steps around a given object include not only the nature of the current object, but also that of the path taken to reach that object. In the current implementation, the Ostensive Model is used to combine evidence collected from all the objects in the path leading to, and including, the current object. Using the OM’s preferred Ostensive Relevance profile, objects closer to the current object are attributed a higher Ostensive Relevance, with the current object having the highest. In particular, any given object has an Ostensive Relevance equal to the sum of all those before it in the path.OstensiveRelevanceAge of evidenceWhen the user selects an object, the system presents those objects that it regards as the most likely to be interesting to the user as next steps. The system presents candidates that it believes to be consistent with the developing theme that has become ostensively apparent in the path.For the current evaluation, this was done using a binary probabilistic retrieval engine (Robertson 1976). The engine has been modified to incorporate the extension to the probabilistic model detailed in Campbell (1996). That extension implements the Ostensive Model in terms of a more sophisticated estimation of one of the probabilistic parameters. The OM-extended probabilistic model can be said to combine objective probabilities from term occurrences with subjective probabilities from the Ostensive Relevance profiles.3.2 The look and feelThe objects and paths grow out from the starting point in a generally rightward direction. This can create a large degree of clutter and potential overlap. A fish-eye view (Lamping 1994) has been employed to partially overcome that problem. Overlaps still occur, but they are less frequent and easier to distinguish.The current object and its candidate next steps (i.e. those that are, as yet, unexplored) are projected flat at the focus (i.e. centre) of the view, whereas all other objects and links are projected with the fish-eye spherical transformation. This effectively pushes all other objects away from the current and candidate objects and helps to further reduce overlaps, particularly in the area of the user’s most likely current attention. All objects are shown on screen at all times, but as their distance from the focus increases, their size and incremental distances tend towards zero.Clicking on an object scrolls the view around such that the current object is the focus of the view. The candidates around the previous current object are removed, and the candidates of the new current object are presented. The objects and links that remain visible on-screen at all times are those objects that, at some point, have been selected as a current object – i.e. have ‘visited’.This browsing view is presented in one window. The content of the current object is displayed within a second window – which is updated automatically. That is, to view the full contents of an object, it is clicked upon, it scrolls to the centre of the browsing view, and its content is displayed in the object window.It is not necessary to click on an object to gain an understanding of what it is or what it contains. Each object is presented as a miniature thumbnail of the document it represents. Simply moving the pointer over an object pops up a larger version of it. In the case of an image object, a large thumbnail is popped up, in the case of a text document it is the title or some other abstraction. The OM’s view of the developing theme along a path makes it possible to produce context sensitive abstracts. That is, abstracts that are intended to best help the user in assessing whether to continue seeking along this path based upon what the system believes the user is currently looking for.The user can therefore move the pointer around inspecting the pop-up abstracts of individual objects anywhere on screen without scrolling or clicking. Any objects that appear interesting can then be clicked on to both reveal their full content, and to present their candidate next steps.3.3 Issues when the system is in useThe use of the Ostensive Model to structure the space, within which the user browses, has a number of implications. Those most relevant to understanding the evaluation presented in this paper are as follows.Any individual document in the collection being searched can appear multiple times on the browsing view. The implication of using the Ostensive Model and this interface is that as the paths leading to each of the object instantiations of the document are different, the candidates are likely to be different. For example, a general review article on nuclear waste dumping will have different candidate next steps when the path to it is made up of predominantly technical, political, or geographical documents. Each different path will influence the process of candidate selection. The candidate objects presented will probably be general in nature (as was the current object). Nevertheless, they will have a bias toward the nature of objects in the path, with a greater bias toward the nature of the most recent objects. This contextual interpretation of an object is a novel feature of the Ostensive Model.As the whole path network is always on screen, the user is free to jump back and forth at will to various points on any path. That is, no information is lost by the system during the course of an information seeking session. This freedom to jump from one place to another (and develop the paths at those points) results in the user being free also to develop multiple paths simultaneously.The lack of explicit communication from the user to the system in terms of what is and is not a promising direction for the user’s information seeking is replaced by implicit communication. For example, if a user is following a path, with each step providing interesting documents, the system will attempt to provide more of such documents. If, at any point, the user does not feel the provided candidates are interesting with respect to his instantaneous information need, he simply looks elsewhere. ‘Elsewhere’, for example, could be a document seen at a previous step that was not followed up, in preference to some other document regarded as more interesting at the time.There is no interaction or exposure to the internal representations used by the underlying retrieval mechanism. In fact, there is no direct evidence (from the user’s perspective) to support the existence of a retrieval system at all. This makes the system potentially suitable for collections of multiple media where a combination of a number of complex and inaccessible representation and retrieval techniques are being used.4 The test collectionThe ‘Paris Collection’, as it has become known, was obtained from the French Ministry of Culture as part of the FERMI project – an ESPRIT multimedia research initiative (Fermi 1994). The image collection, as used here, consists of 666 annotated black & white photographs taken mostly in and around Paris, mostly at the turn of the century. Each image has a textual annotation consisting of both free and controlled vocabulary. An associated thesaurus provides a guide to the intention of the controlled vocabulary.This section first outlines the nature of the documents, then describes how the queries were generated, and then describes how the relevance assessments were generated.4.1 The documents of the collection4.1.1 ImagesEach document has a 640x512-pixel greyscale image (although there are about 12 colour photographs amongst them). The subject matter is varied and includes: Portraits (some of famous people); Buildings, bridges, gardens, monuments; Interiors of buildings; Street scenes; Coastlines; Public events; Close-ups of architectural details; Historical, religious, and decorative artefacts.4.1.2 AnnotationsThe annotations consist of seven text fields. The contents of the fields are either free-text or controlled vocabulary (distinguishable). The language used is exclusively French. All fields are in capital letters. There is an accompanying guide to the controlled vocabulary.The fields are as follows:Title TIT The title of the image.Connotation CNT Feelings conjured up by the image – e.g. love, hate, serenity.Content CON Important objects present in the image – e.g. a man, a horse, a house, a fountain.Morphology MOR Perspectives or positioning of the view – e.g. elevation, 3/4, plan, exterior, interior, reflection.Comment COM Extra information beyond the title – sometimes explanatory.Precision PRE Proper nouns, dates, events – e.g. place names, people’s names, building names.Author AUT The photographer, or agency from whom/which the image originated.Apart from the title, none of the fields are mandatory. The annotations were built by professional image cataloguers at the Ministry. There is no information as to which particular cataloguer is responsible for which annotations, how many cataloguers were involved, or what techniques (if any) were used to maximise consistency and minimise labelling errors. Therefore, one can only assume that the information in each annotation represents a single (and varying) individual’s view of the attributes of an image, at the time of indexing.Two example annotations:IMA:4567TIT:GARCONS DE CAFE OFFICIANT SUR DES BARQUES SOUS L’OEIL VIGILANT ET DISCRET DE L’ARMEE CNT:INSOLITE;CON:DEBIT DE BOISSON; BARQUE; PERSONNAGE; GARCON DE CAFE; RUE-D; PHOTOGRAPHIE EVENEMENTIELLE; INONDATION; PIED; MILITAIRE; REVERBERE; DEVANTURE; SCENE DE RUE;MOR:PLAN MOYEN;COM:INONDATIONS DE 1910 A PARIS ; LE PERSONNEL DU " CAFE DE LA CHAMBRE " TRAVAILLANT EN BARQUE DANS LES RUES INONDEESPRE:CAFE DE LA CHAMBRE; PLACE DU PALAIS BOURBON;AUT:SEEBERGER ATELIER;IMA:0133TIT:STATUE DE JUNON DU CAPITOLE D’APRES L’ANTIQUECNT:SOLENNEL;CON:SOCLE; FEMME; STYLE; TENANT; STATUE EN PIED-D; MARBRE; JARDIN PUBLIC; LEVE; DIVINITE GRECO ROMAINE-D; BRAS-D; ANTIQUITE-D; ARBRE-D;MOR:VUE D’EXTERIEUR; VUE DE TROIS-QUARTS; PLAN MOYEN;COM:PRE:JUNON; JARDIN DES TUILERIES;AUT:ATGET JEAN-EUGENE;4.1.3 ThesaurusThe accompanying guide provides thesaurus-like information relating to the controlled vocabulary. It has roughly 6,300 entries. Each entry contains up to six fields as follows:Domain DO The theme or domain of this term.Generic TG Broader, more generic terms.Associated TA Semantically similar or synonymous terms.Specific TS More specific terms.Replace EP Terms to be avoided and replaced by this term.Preferred EM Terms preferred over this term (e.g. because of ambiguity in this term).Three example thesaurus entries:NOM:DECOR D’ARCHITECTUREDO:ARCHITECTURETG:‘ARCHITECTURE COMMUNETS:BALUSTRADE; AMORTISSEMENT; ALCOVE; VERRIERE; BOISERIE; COURONNEMENT ARCHITECTURAL; PLAFOND;CARRELAGE; CHEMINEE; PARQUET; PAN DE BOIS; BALCON; ORDRE ARCHITECTURAL; MOULURE; DECORD’ARCHITECTURE-DTA:PEINTURE MURALE; ORNEMENTATION; PAPIER PEINTNOM:FJORDDO:GEOGRAPHIE PHYSIQUETG:VALLEETA:BAIE MARITIMENOM:CHAPITEAUDO:ARCHITECTURETG:COLONNETS:CHAPITEAU-D; TAILLOIR; CHAPITEAU DORIQUE; CHAPITEAU TOSCAN; CHAPITEAU CORINTHIEN; CORBEILLE DE CHAPITEAU; CHAPITEAU COMPOSITE; CHAPITEAU IONIQUETA:SCULPTURE; ORNEMENTATIONOne obvious application of the thesaurus would be to expand terms in annotations by adding the more specific terms in an effort to improve recall.4.1.4 The use of the documentsAs the details of retrieval were not the primary concern of this evaluation, the thesaurus was not used and the annotations were simply flattened to text. That is, the fields were concatenated together, no special meanings were attached to each field, and the controlled-vocabulary terms were treated as normal words. The flattened annotations were used as surrogates to index the images in a traditional binary probabilistic text retrieval system. Informal experimentation confirmed that ‘sensible’ retrieval was taking place, with relevance feedback finding images of similar title, content, connotation, morphology, etc.A more sophisticated indexing of the collection has been carried out by Chiaramella and Mechkour (1997) (and Ounis 1998) – also as part of the FERMI project. It provides a highly structured spatial and structural representation of the images based upon Conceptual Graphs. It details spatial and descriptive relationships between the various objects within the image. For example, a particular graph may encode “A woman is standing to the left of a man and in front of tree, the man is watering a horse, the horse is in a stable”. Included in the representation is the polygonal area of the image that each object occupies. An indexing environment was developed to allow the generation of the spatial and structural relationships in a graphical manner, and to allow their combination with the existing textual annotations.4.2 Generating the queriesThe members of the FERMI project group (twenty-five people, spread over four sites) were asked to suggest queries. Some of the queries were designed to show strengths and/or weaknesses in retrieval techniques that were being developed at each of the project sites. Within those suggested queries, three subjective classes were identified:Objects e.g. a man, a tree, a horse, the Eiffel TowerArrangements of objects. e.g. objects to the left of, in front of, below other objects.Implicit abstractions e.g. work, transportation, deathIt can be seen that the ‘arrangement of objects’ class was one particularly suitable to evaluating the Conceptual Graphs indexing mentioned above. From the submissions, a selection was made of those queries that offered a reasonable variety of subjects, and of nature. That is, they were chosen to be sufficiently different from one another and to retain examples of the three classes of query. All queries were generated with general knowledge of the contents of the collection. Thirty queries were produced:01. Military02. Women in contemporary dress03. Dining halls / restaurants04. Death05. A person to the right of a table.06. Horses, a crowd, and stairs ...where the horses are to the left of the stairs.07. Person to the right of a fountain.08. Streets where we see children.09. Door to the right of stairs10. Religious art (but not wall carvings)11. Buildings near water12. Examples of ’early modern’ architecture13. Staircases14. People at work15. Buildings where the roof can be seen clearly16. Transportation。