Lecture 6 Overview Issues in Processing Queries Text Classification – Naive Bayes (Chapt

Lecture 6 Overview●Issues in Processing Queries●Text Classification–Naive Bayes (Chapter 13, IIR)–kNN (Chapter 14, IIR)–SVMs (Chapter 15, IIR)●Midterm Review GuideQuery Evaluation (for ranked retrieval) Multi-step process1.Initialization2.Identify the terms in the query3.Look up information about each term•file offset into inverted file•the document frequency4.Loop over each inverted file entry for terms•compute a partial score for each document5.Sort documents by score6.Report results[1] Initialization●We assume that the lexicon and inverted file already exist●Since multiple queries may be presented, we don’t want to reload data structures multiple times●So load the lexicon data structure into memory[2] Identify terms in the query●Treat queries like documents–Tokenize and identify terms the same way–Don’t want mismatches (e.g., Case vs case)●Note query term frequency for each term–Example:“fast cars”•fast occurred once•cars occurred once–“bookish book lovers”•With stems:•book occurred twice•love occurred once[3] Find information for each term●File offset–Should be stored in term’s lexicon–Used to locate posting’s list in one disk seek ●Document frequency–Should be stored in term’s lexicon–Used to determine length of postings list–Also used if IDF-weighting is employed[4] Loop over each term’s IF list●We will score each document in which any query term occurs–Use a fixed size array (one entry perdocument)–Use a hashtable with docids as keys•What if we had a really large collection?●Looping down a posting’s list, we compute partial scores–The exact computation depends on thesimilarity metric we use–The score for each document changes aremore of the query terms are examinedComputing cosine term-by-term ●For purposes of ranking docsagainst a query –Can ignore query vector ‘length’–Only need consider terms which occur in both query and doc ●Part of the equation isn’t dependent on the query –Precompute document length●The sum can be computedterm by term())()(1),(),(,int ,,1,21,21,21,,t idf t tf w w w w q d Sim w w w w q d q d q d Sim d d t tdqi q i d i t i d i t iqi t i d i t i qi d i !=!!"!!=!•=#####$%====r r r r r rr r Example●Query = survivor island●Doc1 = survivor crusoe ●Doc2 = island bermuda ())()(1),(,int,,1,2t idf t tf w w w w q d Sim d d t t dqi q i d i t i d i !=!!"##$%=r r 9.79591bermuda 14.130crusoe 5.769635island 9.82578survivor idfdf N=524,000Doc1 w s w ^2Doc2 w s w ^2survivor 9.82393796.509750island 0 5.76479433.23285crusoe 14.09197198.58370bermuda 09.79184995.8803Sum 295.0934129.1132Sqrt 17.1782811.3628())()(1),(,int,,1,2t idf t tf w w w w q d Sim d d t t dq i q i d i t i di !=!!"##$%=r r Example, cont’d●Query = survivor island●Doc1 = survivor crusoe ●Doc2 = island bermuda idf Doc1 w s w ^2Doc2 w s w ^2Query QxD1QxD29.829.8296.510.009.8296.510.005.760.00 5.7633.23 5.760.0033.2314.0914.09198.580.000.000.009.790.009.7995.880.000.00Sum 295.09129.1196.5133.23Sqrt 17.1811.36 5.62 2.92[5] Sort documents by score ●Use any old sorting algorithm –Quicksort or Mergesort = O(n log(n))●But, if we want only the top, say 100 docs –Heapsort = O(n log(k))•for k << n, it matters[6] Report results●Trivially, just report docid and score–1Doc XX-1 5.61–2Doc YY-7 2.92–3….●In real world, need to summarize content and be able to present document–Google reports•document title, context of query search terms,HTML Meta description, categorization, hyperlink,size, if cached, dateEfficiency Ideas●Minimize memory - 3 main users–Lexicon (we’ve already talked about that)–Document lengths?–Document scores●For many documents and long queries, may not be able to score them all–Only look at some terms (IDF threshold)–Have a fixed number of accumulators•How should you pick them? FCFS?–Freq. sorted postingsText Classification●Text Classification●Methods–Feature Selection–NB, kNN, SVMs, etc…–Experiments on Reuters●Other filtering tasks–Document Clustering, Topic Detection, Spamfiltering●Collaborative Filtering●Filtering at TREC–Batch & Adaptive TasksDescription of the Problem●In ad hoc retrieval a static collection of documents is searched and a ranked list of documents is returned●What happens when not all of the documents are available ahead of time?–Don’t have global IDF statistics–Not possible to give rankings●What happens if you have standing queries?–Always want new documents, if germane–Generally have to make instant decisionsExample scenario●You are a safety engineer for a large automotive manufacturer and need to monitor accident reports–you want to find systemic problems early●You’d like to review appropriate accident reports each morning–You don’t care about manufacturing, labordisputes, car sales, fuel economy, etc…–Just information about accidents and theircausesUser satisfaction●This is not a ranked retrievalproblem–You want only new information●Too much irrelevant informationtakes you and your staff toomuch time to sift–You want just the good stuff●How can we build a system thatmeets this need?Enter Machine Learning●The leading strategy is to learn to separate data into classes using machine learning techniques–Neural Networks, Bayesian methods, DecisionTrees, kNN, Support Vector Machines, others–Often problems are cast as 2-class problems • e.g., relevant or not relevant●This approach requires supervised training data–For a high-volume application getting labeledexemplars is reasonable●IIR Chapters 16-17 focus on unsupervised scenarios. (Clustering)Unique Problems with Text●Many 1000s of features●Sparse vectors●Non-orthogonal features●Individual features not especially discriminating•“captured the gold medal”•“battled against cancer”–Here“captured/battled” are not about military conflict–If any term was a certain predictor, we couldjust use regular expressions to filter the dataReuters-21578●Approx 22k docs, 28 MB●Most common text classification test set /resources/testcollections/reuters21578/nasdaq gatt andriessen australia coconut, gold example 73239exchanges 93256orgs 15114267people 60147175places 57120135topics w/ 20+w/ 1+# categories Sample document<REUTERS TOPICS="YES" LEWISSPLIT="TRAIN" CGISPLIT="TRAINING-SET"OLDID="18430" NEWID="2012"><DATE> 5-MAR-1987 09:31:01.67</DATE><TOPICS><D>ship </D></TOPICS><PLACES><D>turkey </D></PLACES><TEXT><TITLE>BLIZZARD CLOSES BOSPHORUS </TITLE><DATELINE> ISTANBUL, March 5 - </DATELINE><BODY>Blizzard conditions halted shipping through the Bosphorus waterway and piled snow up to 70 cms deep in central Istanbul, paralysing the city for the second day running.Snow whipped by 48 kph winds continued to fall on Istanbul and northwest Anatolia after 36 hours and weather reports predicted no relief for another two days.Port officials said at least six large vessels in the Black Sea and 13 in the Sea of Marmara were waiting for conditions to improve.Istanbul's Ataturk international airport has been closed since yesterday.</BODY></TEXT></REUTERS>Test ProcedureFor each category:1.Select features2.Train a 2-class classifier on training data●Which algorithm?3.Evaluate performance on held-out data●How much data should be left out?Naïve Bayes●We want to select all and only the relevant documents●It would be nice if we could estimate the probability of a class assignment (c i )given a particular document, d ●Unfortunately, we don’t generally know how to compute thisP (c i d )Naïve Bayes derivation●Applying Bayes Rule:●For ranking classes, if we just maximize the numerator,we will select the best class●Even P(d|c i ) is hard to estimate–Requires looking at each combination of words –In how many docs of the class, did features tire and crash appear, but none of the other features appear? How abouto tire, crash, and drunk? How many combinations are there? P (c i d )=P (d c i )•P (c i )P (d )BestClass =argmax C i P (d c i )•P (c i )BestClass =argmax C i P (x 1,x 2,x 3...x n c i )•P (c i )Conditional Independence ●Assume that each document feature (x j )is independent from another, given belonging to the class●This is a naïve assumption ●To Do:–Estimate P(c i )–Estimate P(x j |c i ) for all x jBestClass =P (c i )•P (x j j =1#words"c i )NB Training (learning estimates)●P(c i )●P(x j |c i )●What if a word in the document (some x j) never occurred in training documents?–P(x j |c i ) is zero–Solution: smoothing (add one to counts) P (c i )=Ndocs (c i )TotalDocsP (x j c i )=count (x j , docs belonging to c i )count (x j , all docs )Binomial model●Instead of all possible words indocuments, restrict vocabulary for the class–Using some means of feature selection ●Use binary features–The word occurred (1) or didn’t (0)–Ignore word order (bag of words model)●Works with short documents–But long documents are almost all 1sFeature Selection●A critical problem in text classification isselecting which features to use to represent documents●Typically words or stemmed words are used ●Rather than use 20-100,000 words, often only the top-ranked features are used●For a classifier about Turkey, words like“Istanbul “ and “Ankara “ might be selected, but “the ” and “water ” probably wouldn’t.–How should features be identified?Same metrics as term similarity Contingency table()()()()c a b a ad b c a d c b a bc ad bc ad N c a b a a N y P x P y x P y x ine y x I +!+++++""++!=="==),(cos CosinesquaredChi log log ),(nInformatio Mutual ))()()(())((2))(()()(),(#N b+d a+c c+dd c Absent a+b b a Present Absent Present Ankara (term)Turkey (class)A paper by Y. Yang and J. Pedersen (A Comparative Study of Feature Selection in Text Categorization) concludes that Chi-squared, Information-Gain, andDocument-Frequency are better than Mutual-Information. ICML-97k Nearest Neighbor Classification●To classify document d into class c●Define k-neighborhood as k nearest neighbors of d●Count number of documents, n, that belong to c●Estimate P(c|d) as n/k●Choose argmax c P(c|d) (majority class) Classes in a Vector SpaceGovernmentScienceArtsExample: k=6 (6NN)P(science| )?GovernmentScienceArtsK Nearest-Neighbor●Using only the closest example (1-NN) to determine the categorization is subject to errors due to:–A single atypical example.–Noise (i.e. error) in the category label of asingle training example.●More robust alternative is to find the k most-similar examples and return the majority category of these k examples.●Value of k is typically odd to avoid ties; 3 and 5 are most common.KNN vs NB●Naïve Bayes works fairly well–Its standard (40+ year history)–Some concerns about independenceassumptions and over training on the data ●KNN requires no training, but...–Classification requires comparing to alltraining documents to find the best kcategories–May be slow●Both are used, but a new ML algorithm is increasingly popularSupport Vector Machines●Good with high-dimensional spaces–Hundreds of thousands of features–Just like text●Avoid over-training on data–If vectors are sparse●Other approaches use few dimensions–But low-ranked dimensions still have value!●Joachims compared SVMs to Naïve Bayes, Rocchio, kNN, & Decision Trees –Assigned reading, available on web page-Support Vector Machines●Large margin classification technique that canwork well for sparse high dimensionalclassification problems ●Not all training vectors are used in model See /+++++------Non maximal margin Joachims’s Experimental Setup ●Looked at 10 most-common Reuters (topic)categories●Measured ‘precision-recall breakeven point’85.982.379.479.972.0avg 85.377.987.762.247.3corn 85.276.685.579.460.6wheat 86.079.280.983.178.7ship 73.174.049.172.558.0interest 76.677.459.277.450.0trade 88.685.775.581.581.0crude 92.482.289.179.572.5grain 75.478.269.467.662.9money-fx 95.292.085.392.191.5acq 98.597.396.196.195.9earn SVM (poly/3)kNN DT Rocchio NBApples ‘n’ Oranges or Good Science?●Yang and Liu critiqued Joachims study (and others) for multiple reasons:–Many reported results used different subsets of ‘standard’collections–Differing numbers of features were used–Different means for selecting features were used–Parameter settings for learning methods might skew results –Statistic significance tests were not performed●They compared SVM, kNN, NN, LLSF, and NB using uniform methods●See “A re-examiniation of text categorization methods”, SIGIR-99Their setup●Used the Reuters 21578 collection –ApteMod split–Training set: 7769 documents–Test set: 3019 documents●Used F-metric for evaluation –Reported both micro and macro averaging –90 topics Differences tended to be significant ecisioncall ecisioncall F Pr Re Pr *Re *2)1(+==!0.38860.7956NB 0.37650.8287NN 0.50080.8498LSF 0.52420.8567KNN 0.52510.8599SVM F-macroF-microLearning taxonomies●Labrou and Finin (UMBC)looked at classifying pagesagainst Yahoo categories(CIKM-99)●Downloaded whole tree andbuilt train/test sets●Approach–Score page against Yahoocategories and pick best●Any page can be assignedto a Yahoo category●Dumais et al. did similarwork in SIGIR-2000 usingtop two levels and SVMsSpam Filtering●Spam detection can be set up as a 2-class problem●Many individual words have good correlations–make, money, fast, $$$, XXX, job, free●Other features–like source address / server–written with ALL CAPS●TREC ran spam detection evaluations in 2005-2007●Recent (2/2007) very good article in CACM by Goodman, Cormack, and HeckermanSpam FilteringSahami et al. created a collection of ~2000 documents –Using a 99.9 percent confidence threshold, lost 3 actual messages while using for one year.–One was a message, “See this spam!” passed on by a colleagueCollaborative FilteringCollaborative Filtering●Collaborative filtering departs from traditional text classification●The idea is that a user doesn’t have to review all content available, but can rely on recommendations of others●For example, recommender systems can suggest content that a user might like given a profile of that user●Users are compared to similar users.●Applications: books, movies, jokes, …Web sites●●●Can match based on contentor user profilesJester – a Joke recommender/humor/Text Classification Applications ●Legal Discovery–Find smoking gun in mounds of data●Military Intelligence / Homeland Defense ●Customized Information Delivery–Sports fans, hobbyist groups●Targeted advertisements●Child-proofing the InternetTREC evaluations / datasets●Several test collections have been used at various TRECs–TREC Disks 1 & 2–OHSUMED–Reuters RCV1 corpus (not Reuters 21578)•(Available from Reuters Ltd for research use)●Multiple tasks–Fixed profiles (Batch)–Routing–Adaptive filteringParameter settings matter48121620242k8k 0.200.220.240.260.280.300.320.340.360.380.40T10SU Neg:Pos Ratio# FeaturesTREC 2001 Batch Task2k 4k 8k 16k–TREC-2001 Batch Filtering task (non-adaptive profiles)•84 Topics; 850k docs (Reuters newswire); 26k training docs –JNI interface enabled exploration of SVM parameter settings •Importance of negative training exemplars, additional features notedHighest published score: 0.41TREC Median score: 0.25JHU/APL performance: 0.40References●Reuters 21578 collection (approx 20 MB)–/resources/testcollections/reuters21578/●Thorsten Joachims. Text categorization with support vector machines: learning with many relevant features. In ECML-98, pp.137-142, 1998.●David Lewis. Naive bayes at forty: The independence assumption in information retrieval. In ECML-98, pages 4-15, 1998.● D Hull and S Robertson, The TREC-8 Filtering Track Final Report,TREC-8, (/pubs/trec8/papers/filtering.pdf)●Yiming Yang and Jan O. Pedersen. A comparative study on feature selection in text categorization. In ICML-97, pp. 412-420.●James Allan, Ron Papka, and Victor Lavrenko. On-line new event detection and tracking. In Proc. ACM SIGIR, pp. 37-45, 1998.。