Relationship Extraction from Biomedical Documents using Conditional Random Fields

Relationship Extraction from Biomedical Documents usingConditional Random FieldsSaurav Sahay, Jinhan Lee, Niyant KrishnamurthiCollege of Computing, Georgia Tech.AbstractExtracting complex relationships automatically from unstructured information resources is a challenging problem. It is an important problem in this present age of abundant machine processable information as there is a need to build intelligent knowledge-aware applications for tasks such search, extraction and reasoning. We have used Conditional Random Fields (CRFs) to identify various relationships from biomedical abstracts.1. IntroductionWe look at the problem of relationship extraction as a sequence labelling and segmentation problem of observation instances. By relationship, we mean the type of association between two neighbouring entities in the domain. More explicitly, a relationship in our context is a {Concept1, Relationship, Concept2} triple. Figure 1 shows a part of the map extracted automatically from Medline abstracts related to Nuclear Cardiology.Figure 1. Map of Relationships extracted from Medline abstractsGraphical models such as Hidden Markov models (HMMs) have been most commonly applied in several bioinformatics, linguistics, modelling and recognition problem. HMMs are directed generative models that involve computation of P(y,x) whereas CRFs are undirected discriminative models that involve computation of P(y|x) (comparable to naïve bayes and logistic regression).2. Related WorkThis problem is similar to the task of named entity recognition from text and there is a lot of work in this area of language processing such as Part of Speech Tagging, Noun Phrase Chunking and Semantic Role Labelling. McCallum, Lafferty and Sutton [1-5] who invented the CRF formalism have several work using CRF to sequence and label data. Bunescu et al [6] have compared linearchain CRFs with Relational Markov Network (RMN)[7] for information extraction, the problem of identifying phrases in natural language text that refer to specific types of entities. Craven et al[8] have used relational learning methods to extract facts from text in order to construct molecular biology knowledgebases. Zhao et al [9] have described collective classification of actors, events and relationships in affiliation networks using RMNs.3. Conditional Random FieldConditional Random Fields (CRFs) are conditional probability distribution models that factorize based on an undirected model. It models the conditional distribution P(Y|X) where X is a set of input variable we observe and Y is a set of hidden variables we predict. By modeling the conditional distribution directly, it does not require to model the distribution P(X), which can includes complex dependencies. In other words, dependencies among the X do not need to be explicitly represented so that CRF can afford the use of rich global features of the input X.3.1 Linear-chain CRFA linear-chain CRF assumes a first-order Markov assumption on the dependencies among hiddenvariable y.Figure 2. Linear Chain CRFA linear-chain conditional random field takes form, where a set of real-valued feature functions isK k t k x y y f 1)},',({= and an instance-specific normalization function is )(x Z . A feature function ),,(1t t t k x y y f − could be arbitrary.∑=−=K k t t t k k x y y f x Z x y p 11)},,(exp{)(1)|(λ, ∑∑=−=y Kk t t t k k x y y f x Z 11)},,(exp{)(λ3.2 Dynamic CRFA dynamic CRF is a generalization of linear-chain CRFs that factorizes based on an undirected model whose structure and parameters are repeated. Model can have a set of hidden variables and complex interaction between hidden variables.Figure 3. Dynamic CRFA dynamic conditional random field takes form ∏∏∑∈=−=t C c K k t t t k k x y y f x Z x y p 11)},,(exp{)(1)|(λ ∑∏∏∑∈==y t C c Kk t c t k k x y f x Z 1,)},(exp{)(λInference in these models can be done using any of the inference algorithms for undirected models. Viterbi decoding is generally used for sequence labelling and marginal computation is used for parameter estimation. Exact inference is generally expensive in Dynamic models hence approximate inference techniques such as loopy Belief Propagation is applied to compute probabilities.4. MethodsWe have setup our relationship extraction task as a text classification problem where we have used rich linguistic and semantic features apart from local contextual features as feature vectors for the sentence phrases. More specifically, we have assigned features to each syntactic phrase of the sentences in biomedical abstracts. Relationships span across several phrases in a sentence. In order to capture this complex structure, we have used features from the neighbouring phrases and added them to each phrase’s set of feature.We have extracted relationships from biomedical abstracts in order to create our benchmark dataset. Our relationship extraction process is described in Figure 4. We have used NLM'sUnified Medical Language System (UMLS), a very large ontology of biomedical and health data as a dictionary for mapping biomedical terms to their Concept types and Semantic Types. We have used WordNet Lexical database for finding synonyms of verb phrases in abstracts in order to categorize them into a set of predefined relationships. The relationships are categorized as ‘affects ’, ‘causes ’, ‘exhibits ’, ‘analyzes ’, etc between the pair of concepts surrounding the verb phrases.The algorithm for relationship extraction is summarized in Figure 4[10].For each abstract AFor each sentence S in AFind occurrences of domain concept pairs in S Æ PairsFor each concept pair <C1, C2> in PairsApply verbGroup matching classify <C1,C2> into relations Æ RAdd R to Relationship Mapreturn MapFigure 4: Relation extraction algorithmExamples of some extracted and categorized concepts and relationships are as follows (Figure 5):left ventricle DIVIDED <PART_OF> regional myocardial uptakesstudy EVALUATED <MEASURES> impactprognosis population COMPRISED <CONTAINS> 16,020 consecutive patientscardioinhibitory response SHOWED <EXHIBITS> vasodepressor responseeighteen patients UNDERWENT <BRINGS_ABOUT> i-123 mibgconstant supply SUSTAIN <PREVENTS> contractile functionFigure 5: Categorized relationshipsWe have used this set of categorized relationships as labels for the phrases surrounding the concepts. For a majority of phrases, where there are no semantic relationships between surrounding concepts, we categorize them as having no relationships. Thus, we have a sparse dataset of few relationships among phrases in our training as well as test dataset.Our feature extraction algorithm is described in the following pseudo-code (Figure 6).For each phrase P in sentence SFor phrase neighborhood of {-n,+n} phrasesExtract contextual, linguistic and semantic featuresIf phrase matches in relationship R from MapLabel PhraseFeatureVector with verbGroup of RElse Label PhraseFeatureVector with type ‘None’Return PhraseFeatureVectorFigure 6: Feature extraction algorithm4.1 Choice of FeaturesThe choice of features to describe the relationship labels is extremely important to get good results for this problem. In most text classification problems, a simple ‘bag of words’ approach is taken to populate the vector space of features. These features are statistically extracted using techniques like ‘term frequency – inverse document frequency’ (TFIDF) or z-score method. These statistical features make the space of possible feature set extremely large thus requiring huge training data to come up with good decision boundaries for classification of data into the right categories.In contrast, we have used rich syntactic and semantic features for our data exploiting the rich and freely available ontologies like UMLS[11] and WordNet. We have extracted Parts of Speech information for the phrases using a Biomedical Parts of Speech tagger MedPostSKRTagger[12].The feature categories that we have used are as follows:1.Phrase2.Concept Types of phrase3.Semantic Type of phrase4. Part of Speech of phraseWe have used these features for the current as well as neighbouring phrases for capturing the rich semantics of the text to be categorized.Examples of these features are:Phrase – myocardial perfusion, Concept Type – Myocardial Perfusion, Semantic Type – Organism Attribute, Part of Speech – Noun PhrasePhrase – examined, Concept Type – Examine, Semantic Type – Finding, Part of Speech – Verb Phrase5. Experiments5.1 Recognizing a relation phraseThe aim of this experiment was to label a phrase as a relation if it defines any relationship. This binary classification experiment labelled the phrases as a ‘relation’ phrase or ‘none’. This is still a difficult classification because relationship phrases constitute less than 5% of the entire text corpus. The CRF must learn this differentiation using features which contain the POS tags, concepts, types of concepts and the actual phrases. The concepts of phrases that come in a window are specified along with their relative position. The concepts of phrases that come before the current relation phrase are coded differently than those that come after to capture the structure of concepts in a relationship.We used the Mallet CRF implementation for our tasks. The CRF performed quite well in this task, labelling most of the relation phrases correctly. Even though it had many false positives there are no false negatives and all the phrases categorized as ‘None’ had a 100% precision as seen in the table below.Label N Correct Returned P R F1None 980 966 966 1 0. 985 0.993Relation 40 40 54 0.740 1.0 0.851Testing Accuracy: 0.9861006 out of 1020 instances correctly labelledTable 1: Binary classification of relationships5.2 Segmentation of Relation PhrasesWe designed this experiment to segment each relationship triple from the entire corpus. The relationship triple consisted of simple phrases like ‘P53 gene affects cancer’. In this example‘P53 gene’ and ‘cancer’ are concepts and the experiment labels the entire phrase as a relationship.We label the relation phrases in the training data using a ‘B’ tag to indicate the start of a relationship, ‘I’ tag to indicate following phrases in the relationship and a ‘O’ tag to mark all other non relationship phrases. The model used to learn this segmentation is a factorial CRF with one of the linear CRF chains learning to label the relations and the other linear CRF to learn the segmentation.The advantage of this model which simultaneously discovers both the labels and the segmentation in comparison with an algorithm which uses the label to segment is that error does not propagate from one step to the next. The errors in the labelling step do not lead to errors in the segmentation.RPF1ReturnedLabel NCorrect(Start) 21 8 11 0.727 0.381 0.5B(Intermediate) 52 15 24 0.652 0.288 0.399I(Other) 591 582 630 0.953 0.984 0.953OTable 2: Segmentation of relationship triplesThe results of the experiment show that the system does learn to an extent to segment the relation phrases with a high precision. The low recall is mostly because of the sparse data and should improve if a larger amount of data is used to train the CRF.5.3 Relation ExtractionThe experiment uses a linear chain CRF to label each individual interesting relationship in the test corpus. The features were extracted from 100 abstracts and used for training. The features consisted of the POS tags, concepts, semantic types and the actual phrase. The noun phrase for the concepts can occur a few phrases before or after the relation verb.Label NReturnedP R F1Correct1.00.976884None 863863brings_about 2 1 1 1.0 0.5 0.6660.2750.190OTHER 21 4 8 0.5Performs 0 0 0 1.0 1.0 1.0Analyzes 2 0 0 1.0 0.0 0.0Causes 1 0 0 1.0 0.0 0.0Disrupts 0 0 0 1.0 1.0 1.0part_of 1 0 0 1.0 0.0 0.0Exhibits 2 2 2 1.0 1.0 1.0Traverses 0 0 0 1.0 1.0 1.0Uses 1 0 0 1.0 0.0 0.0location_of 0 0 0 1.0 1.0 1.0Accuracy: 0.972 Correct 870 out of 895 Joint accuracy: 0.972Table 3: Relationship labellingThe experiment extracted a few relations well from the test set. The sparseness and skew of the training data might account for it not being able to learn the low frequency relations but the relations with a larger number of examples in the training seem to work well. Another way that the efficiency of this algorithm might be hampered is if the concepts discovered in the mining stage are not general enough.6. Future WorkMany interesting experiments can be done to improve the overall efficiency of the relation discovery task of our system. Instead of giving the actual index of the relative position we can specify only if the concept came before or after a verb phrase. The NLP based relation extractor can be further improved to capture more complex relationship for training the CRF and we can use models such as skip CRF and relational markov networks to learn more complex relationships.7 ConclusionLinear chain CRF was used for labelling relations in a corpus. The experiment discovered a few relations and should perform much better with a larger training corpus. The task of binary classification of a relation and a non-relation phrase performed well with high precision and recall values. The suggestions listed in the future work can be used to further improve the experiment results.Referencesfferty, J., McCallum, A., Pereira, F.: Conditional random fields: Probabilistic models forsegmenting and labeling sequence data. In: Proc. 18th International Conf. on Machine Learning,Morgan Kaufmann, San Francisco, CA (2001) 282–2892.Dynamic Conditional Random Fields for Jointly Labeling Multiple Sequences. AndrewMcCallum, Khashayar Rohanimanesh, and Charles Sutton. In NIPS Workshop on Syntax,Semantics, and Statistics. December 2003.3.Collective Segmentation and Labeling of Distant Entities in Information Extraction. CharlesSutton and Andrew McCallum. In ICML Workshop on Statistical Relational Learning and ItsConnections to Other Fields. 2004.4.Dynamic Conditional Random Fields: Factorized Probabilistic Models for Labeling andSegmenting Sequence Data. Charles Sutton, Khashayar Rohanimanesh, and Andrew McCallum.In International Conference on Machine Learning (ICML). 2004.5.Dynamic Conditional Random Fields: Factorized Probabilistic Models for Labeling andSegmenting Sequence Data. Charles Sutton, Andrew McCallum, and Khashayar Rohanimanesh.Journal of Machine Learning Research. vol. 8. March 2007. pp. 693--723.6.Statistical Relational Learning for Natural Language Information Extraction.Razvan Bunescu and Raymond J. MooneyStatistical Relational Learning, Lise Getoor and Ben Taskar (Eds.) book, 2005.7.Benjamin Taskar, Pieter Abbeel, and D. Koller. Discriminative probabilistic models for relationaldata. In Proceedings of 18th Conference on Uncertainty in Artificial Intelligence (UAI-2002),pages 485–492, Edmonton, Canada, 2002.8.Craven, M. and Kumlien, J. 1999. Constructing Biological Knowledge Bases by ExtractingInformation from Text Sources. In Proceedings of the Seventh international Conference onintelligent Systems For Molecular Biology (August 06 - 10, 1999). T. Lengauer, R. Schneider, P.Bork, D. L. Brutlag, J. I. Glasgow, H. Mewes, and R. Zimmer, Eds. AAAI Press, 77-86.9.Bin Zhao, Prithviraj Sen, Lise Getoor, 2006. Entity and Relationship Labeling in AffiliationNetworks. ICML Workshop on Statistical Network Analysis, 200610.Domain Ontology Construction from Biomedical Text, S Sahay, B Li, EV Garcia, E Agichtein, ARam. International Conference on Artificial Intelligence (ICAI-07).11./12.L. Smith, T. Rindflesch, and W. J. Wilbur. Medpost: a part-of-speech tagger for biomedical text.Bioinformatics, 20(14), 2004.。