Apache OpenNLP Developer Documentation

Apache OpenNLP Developer DocumentationWritten and maintained by the Apache OpenNLP Development Community Version 1.5.2-incubatingCopyright © , The Apache Software FoundationLicense and Disclaimer. The ASF licenses thisdocumentation to you under the Apache License, Version2.0 (the "License"); you may not use this documentationexcept in compliance with the License. You may obtain acopy of the License at/licenses/LICENSE-2.0Unless required by applicable law or agreed to inwriting, this documentation and its contents aredistributed under the License on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, eitherexpress or implied. See the License for the specificlanguage governing permissions and limitations under theLicense.,Table of Contents1. Introduction2. Sentence DetectorSentence DetectionSentence Detection ToolSentence Detection APISentence Detector TrainingTraining ToolTraining APIEvaluationEvaluation Tool3. TokenizerTokenizationTokenizer ToolsTokenizer APITokenizer TrainingTraining ToolTraining APIDetokenizingDetokenizing APIDetokenizer Dictionary4. Name FinderNamed Entity RecognitionName Finder ToolName Finder APIName Finder TrainingTraining ToolTraining APICustom Feature GenerationEvaluationEvaluation ToolEvaluation APINamed Entity Annotation Guidelines5. Document CategorizerClassifyingDocument Categorizer ToolDocument Categorizer APITrainingTraining ToolTraining API6. Part-of-Speech TaggerTaggingPOS Tagger ToolPOS Tagger APITrainingTraining ToolTraining APITag DictionaryEvaluationEvaluation Tool7. ChunkerChunkingChunker ToolChunking APIChunker TrainingTraining ToolTraining APIChunker EvaluationChunker Evaluation Tool8. ParserParsingParser ToolParsing APIParser TrainingTraining ToolTraining API9. Coreference Resolution10. CorporaCONLLCONLL 2000CONLL 2002CONLL 2003Arvores DeitadasGetting the dataConverting the dataEvaluationLeipzig Corpora11. Machine LearningMaximum EntropyImplementation12. UIMA IntegrationRunning the pear sample in CVDFurther HelpList of Tables4.1. Genertor elementsChapter 1. IntroductionThe Apache OpenNLP library is a machine learning based toolkit for the processing of natural language text. It supports the most common NLP tasks, such as tokenization, sentence segmentation, part-of-speech tagging, named entity extraction, chunking, parsing, and coreference resolution. These tasks are usually required to build more advanced text processing services. OpenNLP also includes maximumentropy and perceptron based machine learning.The goal of the OpenNLP project will be to create a mature toolkit for the abovementioned tasks. An additional goal is to provide a large number of pre-built models for a variety of languages, as well as the annotated text resources that those models are derived from.Chapter 2. Sentence DetectorTable of ContentsSentence DetectionSentence Detection ToolSentence Detection APISentence Detector TrainingTraining ToolTraining APIEvaluationEvaluation ToolSentence DetectionThe OpenNLP Sentence Detector can detect that a punctuation character marks the end of a sentence or not. In this sense a sentence is defined as the longest white space trimmed character sequence between two punctuation marks. The first and last sentence make an exception to this rule. The first non whitespace character is assumed to be the begin of a sentence, and the last non whitespace characteris assumed to be a sentence end. The sample text below should be segmented into its sentences.P i e r r e V i n k e n,61y e a r s o l d,w i l l j o i n t h e b o a r d a s a n o n e x e c u t i v e d i r e c t o r N o v.29.M r.V i n k e n i sc h a i r m a n o f E l s e v i e r N.V.,t h e D u t c h p u b l i s h i n g g r o u p.R ud o l p h A g ne w,55y e a r so l d a n d f o r m e r c h a i r m a n o f C o n s o l i d a t e d G o l d F i e l d s P L C,w a s n a m e d a d i r e c t o r o f t h i sB r i t i s h i n d u s t r i a l c o n g l o m e r a t e.After detecting the sentence boundaries each sentence is written in its own line.P i e r r e V i n k e n,61y e a r s o l d,w i l l j o i n t h e b o a r d a s a n o n e x e c u t i v e d i r e c t o r N o v.29.M r.V i n k e n i s c h a i r m a n o f E l s e v i e r N.V.,t h e D u t c h p u b l i s h i n g g r o u p.R u d o l p h A g n e w,55y e a r s o l d a n d f o r m e r c h a i r m a n o f C o n s o l i d a t e d G o l d F i e l d s P L C,w a s n a m e d a d i r e c t o r o f t h i s B r i t i s h i n d u s t r i a l c o n g l o m e r a t e.Usually Sentence Detection is done before the text is tokenized and thats the way the pre-trained models on the web site are trained, but it is also possible to perform tokenization first and let the Sentence Detector process the already tokenized text. The OpenNLP Sentence Detector cannot identify sentence boundaries based on the contents of the sentence. A prominent example is the first sentencein an article where the title is mistakenly identified to be the first part of the first sentence. Most components in OpenNLP expect input which is segmented into sentences.Sentence Detection ToolThe easiest way to try out the Sentence Detector is the command line tool. The tool is only intended for demonstration and testing. Download the english sentence detector model and start the Sentence Detector Tool with this command:$b i n/o p e n n l p S e n t e n c e D e t e c t o r e n-s e n t.b i nJust copy the sample text from above to the console. The Sentence Detector will read it and echo one sentence per line to the console. Usually the input is read from a file and the output is redirected to another file. This can be achieved with the following command.$b i n/o p e n n l p S e n t e n c e D e t e c t o r e n-s e n t.b i n<i n p u t.t x t>o u t p u t.t x tFor the english sentence model from the website the input text should not be tokenized.Sentence Detection APIThe Sentence Detector can be easily integrated into an application via its API. To instantiate the Sentence Detector the sentence model must be loaded first.I n p u t S t r e a m m o d e l I n=n e w F i l e I n p u t S t r e a m("e n-s e n t.b i n");t r y{S e n t e n c e M o d e l m o d e l=n e w S e n t e n c e M o d e l(m o d e l I n);}c a t c h(I O E x c e p t i o n e){e.p r i n t S t a c k T r a c e();}f i n a l l y{i f(m o d e l I n!=n u l l){t r y{m o d e l I n.c l o s e();}c a t c h(I O E x c e p t i o n e){}}}After the model is loaded the SentenceDetectorME can be instantiated.S e n t e n c e D e t e c t o r M E s e n t e n c e D e t e c t o r=n e w S e n t e n c e D e t e c t o r M E(m o d e l);The Sentence Detector can output an array of Strings, where each String is one sentence.S t r i n g s e n t e n c e s[]=s e n t e n c e D e t e c t o r.s e n t D e t e c t("F i r s t s e n t e n c e.S e c o n d s e n t e n c e.");The result array now contains two entires. The first String is "First sentence." and the second Stringis "Second sentence." The whitespace before, between and after the input String is removed. The APIalso offers a method which simply returns the span of the sentence in the input string.S p a n s e n t e n c e s[]=s e n t e n c e D e t e c t o r.s e n t P o s D e t e c t("F i r s t s e n t e n c e.S e c o n d s e n t e n c e.");The result array again contains two entires. The first span beings at index 2 and ends at 17. Thesecond span begins at 18 and ends at 34. The utility method Span.getCoveredText can be used to createa substring which only covers the chars in the span.Sentence Detector TrainingTraining ToolOpenNLP has a command line tool which is used to train the models available from the model downloadpage on various corpora. The data must be converted to the OpenNLP Sentence Detector training format.Which is one sentence per line. An empty line indicates a document boundary. In case the documentboundary is unknown, its recommended to have an empty line every few ten sentences. Exactly like theoutput in the sample above. Usage of the tool:$b i n/o p e n n l p S e n t e n c e D e t e c t o r T r a i n e rU s a g e:o p e n n l p S e n t e n c e D e t e c t o r T r a i n e r[-a b b D i c t p a t h][-p a r a m s p a r a m s F i l e]-l a n g l a n g u a g e\[-c u t o f f n u m][-i t e r a t i o n s n u m][-e n c o d i n g c h a r s e t N a m e]-d a t a t r a i n D a t a-m o d e l m o d e l F i l eA r g u m e n t s d e s c r i p t i o n:-a b b D i c t p a t hT h e a b b r e v i a t i o n d i c t i o n a r y i n X M L f o r m a t.-p a r a m s p a r a m s F i l eT r a i n i n g p a r a m e t e r s f i l e.-l a n g l a n g u a g es p e c i f i e s t h e l a n g u a g e w h i c h i s b e i n g p r o c e s s e d.-c u t o f f n u ms p e c i f i e s t h e m i n n u m b e r o f t i m e s a f e a t u r e m u s t b e s e e n.I t i s i g n o r e d i f a p a r a m e t e r s f i l e i s p a s s e d.-i t e r a t i o n s n u ms p e c i f i e s t h e n u m b e r o f t r a i n i n g i t e r a t i o n s.I t i s i g n o r e d i f a p a r a m e t e r s f i l e i s p a s s e d.-e n c o d i n g c h a r s e t N a m es p e c i f i e s t h e e n c o d i n g w h i c h s h o u l d b e u s e d f o r r e a d i n g a n d w r i t i n g t e x t.I f n o t s p e c i f i e d t h e s y s t e m d e f a u l t w i l l b e u s e -d a t a t r a i n D a t at h e d a t a t o b e u s e d d u r i n g t r a i n i n g-m o d e l m o d e l F i l et h e o u t p u t m o d e l f i l eTo train an English sentence detector use the following command:$b i n/o p e n n l p S e n t e n c e D e t e c t o r T r a i n e r-e n c o d i n g U T F-8-l a n g e n-d a t a e n-s e n t.t r a i n-m o d e l e n-s e n t.b i nI n d e x i n g e v e n t s u s i n g c u t o f f o f5C o m p u t i n g e v e n t c o u n t s...d o n e.4883e v e n t sI n d e x i n g...d o n e.S o r t i n g a n d m e r g i n g e v e n t s...d o n e.R e d u c e d4883e v e n t s t o2945.D o n e i n d e x i n g.I n c o r p o r a t i n g i n d e x e d d a t a f o r t r a i n i n g...d o n e.N u m b e r o f E v e n t T o k e n s:2945N u m b e r o f O u t c o m e s:2N u m b e r o f P r e d i c a t e s:467...d o n e.C o m p u t i n g m o d e l p a r a m e t e r s...P e r f o r m i n g100i t e r a t i o n s.1:..l o g l i k e l i h o o d=-3384.63768267431440.389514642637722732:..l o g l i k e l i h o o d=-2191.92666885976720.93979111202129843:..l o g l i k e l i h o o d=-1645.86407715559810.96436616833913584:..l o g l i k e l i h o o d=-1340.3863037745190.97399139873028875:..l o g l i k e l i h o o d=-1148.41415485196240.9748105672742167...<s k i p p i n g a b u n c h o f i t e r a t i o n s>...95:..l o g l i k e l i h o o d=-288.255568058744360.983411836985459896:..l o g l i k e l i h o o d=-287.22836803434810.983411836985459897:..l o g l i k e l i h o o d=-286.21748303445260.983411836985459898:..l o g l i k e l i h o o d=-285.2224869810480.983411836985459899:..l o g l i k e l i h o o d=-284.242969172239160.9834118369854598following code sample shows how a model can be loaded.I n p u t S t r e a m m o d e l I n=n e w F i l e I n p u t S t r e a m("e n-t o k e n.b i n");t r y{T o k e n i z e r M o d e l m o d e l=n e w T o k e n i z e r M o d e l(m o d e l I n);}c a t c h(I O E x c e p t i o n e){e.p r i n t S t a c k T r a c e();}f i n a l l y{i f(m o d e l I n!=n u l l){t r y{m o d e l I n.c l o s e();}c a t c h(I O E x c e p t i o n e){}}}After the model is loaded the TokenizerME can be instantiated.T o k e n i z e r t o k e n i z e r=n e w T o k e n i z e r M E(m o d e l);The tokenizer offers two tokenize methods, both expect an input String object which contains theuntokenized text. If possible it should be a sentence, but depending on the training of the learnabletokenizer this is not required. The first returns an array of Strings, where each String is one token.S t r i n g t o k e n s[]=t o k e n i z e r.t o k e n i z e("A n i n p u t s a m p l e s e n t e n c e.");The output will be an array with these tokens."A n","i n p u t","s a m p l e","s e n t e n c e","."The second method, tokenizePos returns an array of Spans, each Span contain the begin and endcharacter offsets of the token in the input String.S p a n t o k e n S p a n s[]=t o k e n i z e r.t o k e n i z e P o s("A n i n p u t s a m p l e s e n t e n c e.");The tokenSpans array now contain 5 elements. To get the text for one span call Span.getCoveredTextwhich takes a span and the input text. The TokenizerME is able to output the probabilities for thedetected tokens. The getTokenProbabilities method must be called directly after one of the tokenizemethods was called.T o k e n i z e r M E t o k e n i z e r=...S t r i n g t o k e n s[]=t o k e n i z e r.t o k e n i z e(...);d o u b le t o k e n P r o b s[]=t o k e n i z e r.g e t T o k e n P r o b a b i l i t i e s();The tokenProbs array now contains one double value per token, the value is between 0 and 1, where 1 isthe highest possible probability and 0 the lowest possible probability.Tokenizer TrainingTraining ToolOpenNLP has a command line tool which is used to train the models available from the model downloadpage on various corpora. The data must be converted to the OpenNLP Tokenizer training format. Which isone sentence per line. Tokens are either separater by a whitespace or if by a special <SPLIT> tag. Thefollowing sample shows the sample from above in the correct format.P i e r r e V i n k e n<S P L I T>,61y e a r s o l d<S P L I T>,w i l l j o i n t h e b o a r d a s a n o n e x e c u t i v e d i r e c t o r N o v.29<S P L I T>.M r.V i n k e n i s c h a i r m a n o f E l s e v i e r N.V.<S P L I T>,t h e D u t c h p u b l i s h i n g g r o u p<S P L I T>.R u d o l p h A g n e w<S P L I T>,55y e a r s o l d a n d f o r m e r c h a i r m a n o f C o n s o l i d a t e d G o l d F i e l d s P L C<S P L I T>,w a s n a m e d a n o n e x e c u t i v e d i r e c t o r o f t h i s B r i t i s h i n d u s t r i a l c o n g l o m e r a t e<S P L I T>.Usage of the tool:$b i n/o p e n n l p T o k e n i z e r T r a i n e rU s a g e:o p e n n l p T o k e n i z e r T r a i n e r[-a b b D i c t p a t h][-a l p h a N u m O p t i s A l p h a N u m O p t][-p a r a m s p a r a m s F i l e]-l a n g l a n g u a g e[-c u t o f f n u m][-i t e r a t i o A r g u m e n t s d e s c r i p t i o n:-a b b D i c t p a t hT h e a b b r e v i a t i o n d i c t i o n a r y i n X M L f o r m a t.-a l p h a N u m O p t i s A l p h a N u m O p tO p t i m i z a t i o n f l a g t o s k i p a l p h a n u m e r i c t o k e n s f o r f u r t h e r t o k e n i z a t i o n-p a r a m s p a r a m s F i l eT r a i n i n g p a r a m e t e r s f i l e.b e u s es a i d->N O_O P E R A T I O N"->M E R G E_T O_R I G H TT h i s->N O_O P E R A T I O Ni s->N O_O P E R A T I O Na->N O_O P E R A T I O Nt e s t->N O_O P E R A T I O N"->M E R G E_T O_L E F T.->M E R G E_T O_L E F TThat will result in the following character sequence:H e s a i d"T h i s i s a t e s t".TODO: Add documentation about the dictionary format and how to use the API. Contributions are welcome. Detokenizing APITODO: Write documentation about the detokenizer api. Any contributions are very welcome. If you want to contribute please contact us on the mailing list or comment on the jira issue OPENNLP-216. Detokenizer DictionaryTODO: Write documentation about the detokenizer dictionary. Any contributions are very welcome. If you want to contribute please contact us on the mailing list or comment on the jira issue OPENNLP-217. Chapter 4. Name FinderTable of ContentsNamed Entity RecognitionName Finder ToolName Finder APIName Finder TrainingTraining ToolTraining APICustom Feature GenerationEvaluationEvaluation ToolEvaluation APINamed Entity Annotation GuidelinesNamed Entity RecognitionThe Name Finder can detect named entities and numbers in text. To be able to detect entities the Name Finder needs a model. The model is dependent on the language and entity type it was trained for. The OpenNLP projects offers a number of pre-trained name finder models which are trained on various freely available corpora. They can be downloaded at our model download page. To find names in raw text the text must be segmented into tokens and sentences. A detailed description is given in the sentence detector and tokenizer tutorial. Its important that the tokenization for the training data and the input text is identical.Name Finder ToolThe easiest way to try out the Name Finder is the command line tool. The tool is only intended for demonstration and testing. Download the English person model and start the Name Finder Tool with this command:$b i n/o p e n n l p T o k e n N a m e F i n d e r e n-n e r-p e r s o n.b i nThe name finder now reads a tokenized sentence per line from stdin, an empty line indicates a document boundary and resets the adaptive feature generators. Just copy this text to the terminal:P i e r r e V i n k e n,61y e a r s o l d,w i l l j o i n t h e b o a r d a s a n o n e x e c u t i v e d i r e c t o r N o v.29.M r.V i n k e n i s c h a i r m a n o f E l s e v i e r N.V.,t h e D u t c h p u b l i s h i n g g r o u p.R u d o l p h A g n e w,55y e a r s o l d a n d f o r m e r c h a i r m a n o f C o n s o l i d a t e d G o l d F i e l d s P L C,w a s n a m e da d i r e c t o r o f t h i s B r i t i s h i n d u s t r i a l c o n g l o m e r a t e.the name finder will now output the text with markup for person names:<S T A R T:p e r s o n>P i e r r e V i n k e n<E N D>,61y e a r s o l d,w i l l j o i n t h e b o a r d a s a n o n e x e c u t i v e d i r e c t o r N o v.29.M r.<S T A R T:p e r s o n>V i n k e n<E N D>i s c h a i r m a n o f E l s e v i e r N.V.,t h e D u t c h p u b l i s h i n g g r o u p.<S T A R T:p e r s o n>R u d o l p h A g n e w<E N D>,55y e a r s o l d a n d f o r m e r c h a i r m a n o f C o n s o l i d a t e d G o l d F i e l d s P L C, w a s n a m e d a d i r e c t o r o f t h i s B r i t i s h i n d u s t r i a l c o n g l o m e r a t e.names. The following lines show how to construct a custom feature generatorA d a p t i v e F e a t u r e G e n e r a t o r f e a t u r e G e n e r a t o r=n e w C a c h e d F e a t u r e G e n e r a t o r(n e w A d a p t i v e F e a t u r e G e n e r a t o r[]{n e w W i n d o w F e a t u r e G e n e r a t o r(n e w T o k e n F e a t u r e G e n e r a t o r(),2,2),n e w W i n d o w F e a t u r e G e n e r a t o r(n e w T o k e n C l a s s F e a t u r e G e n e r a t o r(t r u e),2,2),n e w O u t c o m e P r i o r F e a t u r e G e n e r a t o r(),n e w P r e v i o u s M a p F e a t u r e G e n e r a t o r(),n e w B i g r a m N a m e F e a t u r e G e n e r a t o r(),n e w S e n t e n c e F e a t u r e G e n e r a t o r(t r u e,f a l s e)});which is similar to the default feature generator. The javadoc of the feature generator classes explain what the individual feature generators do. To write a custom feature generator please implement the AdaptiveFeatureGenerator interface or if it must not be adaptive extend the FeatureGeneratorAdapter. The train method which should be used is defined asp u b l i c s t a t i c T o k e n N a m e F i n d e r M o d e l t r a i n(S t r i n g l a n g u a g e C o d e,S t r i n g t y p e,O b j e c t S t r e a m<N a m e S a m p l e>s a m p l e s,A d a p t i v e F e a t u r e G e n e r a t o r g e n e r a t o r,f i n a l M a p<S t r i n g,O b j e c t>r e s o u r c e s,i n t i t e r a t i o n s,i n t c u t o f f)t h r o w s I O E x c e p t i o nand can take feature generator as an argument. To detect names the model which was returned from the train method and the feature generator must be passed to the NameFinderME constructor.n e w N a m e F i n d e r M E(m o d e l,f e a t u r e G e n e r a t o r,N a m e F i n d e r M E.D E F A U L T_B E A M_S I Z E);Feature Generation defined by XML DescriptorOpenNLP can also use a xml descritpor file to configure the featuer generation. The descriptor file is stored inside the model after training and the feature generators are configured correctly when the name finder is instantiated. The following sample shows a xml descriptor:<g e n e r a t o r s><c a c h e><g e n e r a t o r s><w i n d o w p r e v L e n g t h="2"n e x t L e n g t h="2"><t o k e n c l a s s/></w i n d o w><w i n d o w p r e v L e n g t h="2"n e x t L e n g t h="2"><t o k e n/></w i n d o w><d e f i n i t i o n/><p r e v m a p/><b i g r a m/><s e n t e n c e b e g i n="t r u e"e n d="f a l s e"/></g e n e r a t o r s></c a c h e></g e n e r a t o r s>The root element must be generators, each sub-element adds a feature generator to the configuration. The sample xml is equivalent to the generators defined by the API above.The following table shows the supported elements:Table 4.1. Genertor elementsElement Aggregated Attributesgenerators yes nonecache yes nonecharngram no min and max specify the length of the generated character ngramsdefinition no nonedictionary no dict is the key of the dictionary resource to use, and prefix is a feature prefix stringprevmap no nonesentence no begin and end to generate begin or end features, both are optional and are boolean valuestokenclass no nonetoken no nonebigram no nonetokenpattern no nonewindow yes prevLength and nextLength must be integers ans specify the window size custom no class is the name of the feature generator class whcih will be loadedAggregated feature generators can contain other generators, like the cache or the window feature generator in the sample.EvaluationT h e u p w a r d m o v e m e n t o f g r o s s m a r g i n r e s u l t e d f r o m a m o u n t s p u r s u a n t t oa d j u s t m e n t s t o ob l i g a t i o n s t o w a r d s d e a l e r s.To be able to classify a text, the document categorizer needs a model. The classifications are requirements-specific and hence there is no pre-built model for document categorizer under OpenNLP project.Document Categorizer ToolThe easiest way to try out the document categorizer is the command line tool. The tool is only intended for demonstration and testing. The following command shows how to use the document categorizer tool.$b i n/o p e n n l p D o c c a t m o d e lThe input is read from standard input and output is written to standard output, unless they are redirected or piped. As with most components in OpenNLP, document categorizer expects input which is segmented into sentences.Document Categorizer APITo perform classification you will need a maxent model - these are encapsulated in the DoccatModel class of OpenNLP tools.First you need to grab the bytes from the serialized model on an InputStream - we'll leave it you to do that, since you were the one who serialized it to begin with. Now for the easy part:I n p u t S t r e a m i s=...D o c c a t M o d e l m=n e w D o c c a t M o d e l(i s);With the DoccatModel in hand we are just about there:S t r i n g i n p u t T e x t=...D o c u m e n t C a t e g o r i z e r ME m y C a t e g o r i z e r=n e w D o c u m e n t C a t e g o r i e r M E(m);d o u b l e[]o u t c o me s=m y C a t e g o r i z e r.c a t e g o r i z e(i n p u t T e x t);S t r i n g c a t e g o r y=m y C a t e g o r i z e r.g e t B e s t O u t c o m e();TrainingThe Document Categorizer can be trained on annotated training material. The data must be in OpenNLP Document Categorizer training format. This is one document per line, containing category and text separated by a whitespace. The following sample shows the sample from above in the required format. Here GMDecrease and GMIncrease are the categories.G M D e c r e a s e M a j o r a c q u i s i t i o n s t h a t h a v e a l o w e r g r o s s m a r g i n t h a n t h e e x i s t i n g n e t w o r k a l s o\h a d a n e g a t i v e i m p a c t o n t h e o v e r a l l g r o s s m a r g i n,b u t i t s h o u l d i m p r o v e f o l l o w i n g\t h e i m p l e m e n t a t i o n o f i t s i n t e g r a t i o n s t r a t e g i e s.G M I n c r e a s e T h e u p w a r d m o v e m e n t o f g r o s s m a r g i n r e s u l t e d f r o m a m o u n t s p u r s u a n t t o a d j u s t m e n t s\t o o b l i g a t i o n s t o w a r d s d e a l e r s.Note: The line breaks marked with a backslash are just inserted for formatting purposes and must not be included in the training data.Training ToolThe following command will train the document categorizer and write the model to en-doccat.bin:$b i n/o p e n n l p D o c c a t T r a i n e r-e n c o d i n g U T F-8-l a n g e n-d a t a e n-d o c c a t.t r a i n-m o d e l e n-d o c c a t.b i nAdditionally it is possible to specify the number of iterations, and the cutoff.Training APISo, naturally you will need some access to many pre-classified events to train your model. The class opennlp.tools.doccat.DocumentSample encapsulates a text document and its classification. DocumentSample has two constructors. Each take the text's category as one argument. The other argument can either be raw text, or an array of tokens. By default, the raw text will be split into tokens by whitespace. So, let's say your training data was contained in a text file, where the format is as described above. Then you might want to write something like this to create a collection of DocumentSamples:D o c c a t M o d e l m o d e l=n u l l;I n p u t S t r e a m d a t a I n=n u l l;t r y{d a t a I n=ne w F i l e I n p u t S t r e a m("e n-s e n t i m e n t.t r a i n");O b j e c t S t r e a m<S t r i n g>l i n e S t r e a m=n e w P l a i n T e x t B y L i n e S t r e a m(d a t a I n,"U T F-8");O b j e c t S t r e a m<D o c u m e n t S a m p l e>s a m p l e S t r e a m=n e w D o c u m e n t S a m p l e S t r e a m(l i n e S t r e a m);m o d e l=D o c u m e n t C a t e g o r i z e r M E.t r a i n("e n",s a m p l e S t r e a m);}c a t c h(I O E x c e p t i o n e){//F a i l e d t o r e a d o r p a r s e t r a i n i n g d a t a,t r a i n i n g f a i l e de.p r i n t S t a c k T r a c e();}f i n a l l y{i f(d a t a I n!=n u l l){t r y{d a t a I n.c l o s e();}c a t c h(I O E x c e p t i o n e){//N o t a n i s s u e,t r a i n i n g a l r e a d y f i n i s h e d.//T h e e x c e p t i o n s h o u l d b e l o g g e d a n d i n v e s t i g a t e d//i f p a r t o f a p r o d u c t i o n s y s t e m.e.p r i n t S t a c k T r a c e();}}}Now might be a good time to cruise over to Hulu or something, because this could take a while ifyou've got a large training set. You may see a lot of output as well. Once you're done, you can pretty quickly step to classification directly, but first we'll cover serialization. Feel free to skim.O u t p u t S t r e a m m o d e l O u t=n u l l;t r y{m o d e l O u t=n e w B u f f e r e d O u t p u t S t r e a m(n e w F i l e O u t p u t S t r e a m(m o d e l F i l e));m o d e l.s e r i a l i z e(m o d e l O u t);}c a t c h(I O E x c e p t i o n e){//F a i l e d t o s a v e m o d e le.p r i n t S t a c k T r a c e();}f i n a l l y{i f(m o d e l O u t!=n u l l){t r y{m o d e l O u t.c l o s e();}c a t c h(I O E x c e p t i o n e){//F a i l e d t o c o r r e c t l y s a v e m o d e l.//W r i t t e n m o d e l m i g h t b e i n v a l i d.e.p r i n t S t a c k T r a c e();}}}Chapter 6. Part-of-Speech TaggerTable of ContentsTaggingPOS Tagger ToolPOS Tagger APITrainingTraining ToolTraining APITag DictionaryEvaluationEvaluation ToolTaggingThe Part of Speech Tagger marks tokens with their corresponding word type based on the token itself and the context of the token. A token might have multiple pos tags depending on the token and the context. The OpenNLP POS Tagger uses a probability model to predict the correct pos tag out of the tag set. To limit the possible tags for a token a tag dictionary can be used which increases the tagging and runtime performance of the tagger.POS Tagger ToolThe easiest way to try out the POS Tagger is the command line tool. The tool is only intended for demonstration and testing. Download the english maxent pos model and start the POS Tagger Tool with this command:$b i n/o p e n n l p P O S T a g g e r e n-p o s-m a x e n t.b i nThe POS Tagger now reads a tokenized sentence per line from stdin. Copy these two sentences to the console:P i e r r e V i n k e n,61y e a r s o l d,w i l l j o i n t h e b o a r d a s a n o n e x e c u t i v e d i r e c t o r N o v.29.M r.V i n k e n i s c h a i r m a n o f E l s e v i e r N.V.,t h e D u t c h p u b l i s h i n g g r o u p.。