1 Introduction Source- and User- Adaptive Information Integration

Source-and User-Adaptive Information IntegrationSubbarao KambhampatiDepartment of Computer Science and EngineeringArizona State University,Tempe,AZ85287/i3rao@1IntroductionThe availability of structured and semi-structured scientific information sources on the web has recently lead to significant interest in scientific communities for query processing frameworks that can integrate such online sources[IBM02,MHTH01,Roo01].In the past three years,we have been involved in efforts to integrate data sources in three domains–bibliographic sources(BibFinder[NK04a,NKH03]),archaeological sources(KADIS,[Kin04])and biological sources(BioHavasu,[HK04]).Our experience in these domains lead us the the conclusion that an important impediment to success is the lack of adaptiveness in current integration frameworks to the source profiles and the user interests.The autonomous and decentralized nature of the data sources constrains the current-day mediators to operate with very little information about the structure,scope,and contents of the information sources they are trying to integrate.While some types of statistics may well be voluntarily publicized by the individual data sources,most of them need to be learned/gathered pro-actively by the mediator.Most current mediators start with just the information about how the global schemas of individual information sources map to the mediated schema.Such pared-down “source descriptions”hide a variety of types of knowledge,including the scope,profile and quality of the data stored in the sources,inter-source relations and delivery parameters,crucial to supporting effective query processing.The traditional models of the users used by the mediators are not appropriate to the multi-objective and imprecise querying needs of the scientific users.Not surprisingly,most existing integration frameworks are unable to supportflexible query processing that takes source profiles and conflicting user preferences(e.g.those between cost and coverage)into account.In our ongoing research research,we aim to develop a framework for information integration that will be adaptive to the data sources as well as user interests by being pro-active in gathering(learning)and using source statistics forflexible and efficient query processing.To concretely illustrate the challenges in adap-tive integration,we will look at one of the applications we have experience with:bibliographic data source integration.BibFinder(/bibfinder),is a publicly“fielded”computer science bib-liography mediator that we have been developing for the past year[NKH03,NK04a].BibFinder integrates several online Computer Science bibliography sources,and it currently covers CSB,DBLP,Network Bibli-ography,ACM Digital Library,ACM Guide,IEEE Xplore,ScienceDirect and CiteSeer.The global schema exported by BibFinder can be modeled in terms of just the relation:paper(title,author,conference/journal,pagenum,year,abstract,doc).Each of the individual sources may export only a subset of the global relation.For example,the source Network Bibliography only contains publications in Networks,DBLP gives more emphasis to Database related publications,while ScienceDirect has only archival journal publications.Furthermore,not all sources1may export all attributes for a particular paper–while CiteSeer always has a link to the actual document of a paper/publication it stores,while DBLP does not.On the other hand,DBLP tends to export page numbersof the original publication,while CiteSeer typically does not.Suppose we have a selection query:Select title,author,year From paper where conference/journal=SIGMOD.A naive way of answering this selection query(that is used by most existing aggregation systems)wouldbe to send it to all the data sources,wait for the results,eliminate duplicates,1and return the answers to the user.This not only leads to increased query processing time and duplicate tuple transmission(“cost consid-erations”),but also unnecessarily increases the load on the individual sources(“politeness considerations”).A more efficient and polite approach would be to direct the query only to the most relevant sources.For example,for the selection query above,DBLP,ACM DL and ACM Guide are most relevant,while Network Bibliography is much less relevant.Furthermore,since DBLP stores records of virtually all the SIGMOD papers,we can skip ACM DL if we are calling DBLP anyway.In order to judge the relative degree of relevance of a source however,BibFinder needs to know the coverage of each source with respect to the query,as well as the overlap between different sources with respect to the query.Even defining overlap between sources becomes tricky when the data sources are het-erogeneous,and include textual records.Other useful source statistics include the source latency(responsetime profiles),density(i.e.,average fraction of null values in a tuple)and dominance(an application-specific statistic–such as“cost”–that gives preference information among tuples referring to the same entity).Source statistics such as those above are not likely to be voluntarily exported by a source.Neither is it feasible for domain experts to provide such statistics in the form of annotated meta-data.We thus need to develop effective approaches for learning,maintaining and using these statistics.Gathering these statistics presents several challenges as online data sources,unlike traditional databases,are autonomous.Once gathered,the source statistics are likely to change over time,and this necessitates the need to incrementally maintain the gathered statistics.Another challenge is that the source statistics tend to interact with each other in supporting effective query processing.Given that a source like ACM Guide has a significantly broad coverage,the mediator maybe tempted to send all user queries its way.This may not be very efficient considering that ACM Guide tends to be significantly slower than some of the other sources that have a lower coverage.Adequate source selection ultimately boils down to capturing the coverage and response time preferences of the user,and using them,in concert with coverage and response time statistics.Handling the interactions between these competing objectives,using the gathered source statistics,adds a multi-objectiveflavor to query processingand introduces several new challenges,including the need to develop and use more sophisticated cost and reward models for query plans[NK01,SR86,UJ02,PY01].Finally,often the lay users of a mediator system may be unable to precisely express their query.We found this to be especially true in our experience with archaeological and biological sources,where we need to deal with a broad range of scientific users,many of who lack a specific knowledge to frame pre-cise queries.Supporting imprecise queries is thus a critical requirement of mediator systems.Traditional techniques for similarity queries over databases are not applicable here as the mediator does not have accessto internals of the data sources it integrates,and the users may not be able to articulate domain specific similarity metrics.Aditi Framework:In response to the challenges outlined above,we are developing a comprehensive frame-work for adaptive data integration that we call Aditi.Figure1shows the high-level architecture we envision 1BibFinder uses a combination of three attributes:title,author,and year as the primary key to uniquely identify a paper across sources.2Figure1:Aditi Architecture for Adaptive Information Integrationfor Aditi.There arefive important modules.The StatMiner module will mine and effectively store a variety of statistics about the data sources.In this task,it is assisted by the probing and logging module which keeps track of parts of the data sources that have been accessed by users(or have been pro-actively sampled).The monitor module is charged with incrementally revising the statistics in response to changes in data sources or user interests.The similarity miner module is aimed at extracting structural dependencies between the attributes of the data records to support imprecise queries.(In contrast to StatMiner which keeps track of source statistics,the similarity miner can be seen as keeping track of the data statistics.)The query engine module is tasked with actually supporting the user queries.It aims to(i)reduce cost of query execution by dynamically selecting the most relevant sources(ii)support multi-objective query processing by supporting tradeoffs between cost and quality of the answers and(iii)support imprecise queries.To the extent possi-ble,the query engine avoids dependence on user-supplied meta-data,and exploits instead the automatically gathered(and maintained)source and data statistics.In Summary,Aditi exploits machine learning techniques to support source-and user-adaptive informa-tion integration.Source adaptivity is supported through learning and maintenance of source profiles.It supports user adaptivity through imprecise queries,as well as making the statistics sensitive to the current user interests(as captured by the query logs).Relation to existing work:Most work on applying learning techniques to support information integra-tion has focused on the challenges in schema mapping and query reformulation(c.f[ILW+00,FALS99, YLUGM99,UF98,DL99,NLF99,TRV97,GMPQ+97,LKG99]).While reconciling heterogeneous schemas of the individual data sources is a critical challenge,our experience in integrating online bibliographic, archaeological and biological data sources to-date[NK04a,HK04,Kin04],indicates that adaptive query3processing challenges are equally important in scaling information integration systems.There has been little work on statistics gathering and source and user-profile learning in information integration scenarios. Although the utility of quantitative coverage statistics to rank the sources was explored by Florescu et.al. [FKL97],the primary aim of the effort was on the“use”of coverage statistics.In contrast,in our research, we take a comprehensive look at learning,maintaining and using a wider range of statistics to supportflexi-ble query processing.There has been some previous work on learning selectivity and response-time statistics both in multi-database literature[ZL96],and data integration literature[GRZZ00].Some of the problems re-lated to source selection have also been considered in the context of text database selection in the context of keyword queries submitted to meta-search engines[WMY00,IGS01,PF03,MYL02,CLC95,GGT99].Our proposed work differs from these efforts by considering a more comprehensive set of statistics(including overlap,latency,density,dominance),as well as their combined usage in query processing.The problem of supporting imprecise database queries has received limited attention over the years from researchers includ-ing those in fuzzy information systems[Fuzzy90],cooperative query answering[JWS81,Joshi82,Motro86] and query generalization[CCL92,Motro90].More recent efforts have focussed at supporting imprecise queries over relational databases by introducing abstract data types and extending the query processor with similarity functions[Mot98,Bin03,GS+98,AB+02].However,all these approaches require users and/or database designers to provide domain specific distance metrics and importance measures for attributes of interest.Unfortunately,such information is hard to elicit from lay users.Finally,the need forflexibility and adaptiveness of query processing in information integration scenarios is also mentioned prominently in the 2003Lowell Report on Database Research Self-Assessment[Lowell03].References[IBM02]Discovery Link.IBM./solutions/lifesciences/solutions/discoverylink.html [MHTH01]P.Mork,A.Halevy,and P.Tarczy-Hornoch.A model for data integration systems of biomedical data applied to online genetic databases.In Symposium of the American Medical Informatics Association,2001.[Roo01] D.S.Roos.Bio-informatics:Trying to swim in a sea of data.Nature,291(5507):1260–1261,February, 162001.[NK03b]U.Nambiar and S.Kambhampati.Answering Imprecise Database Queries:A Novel Approach.WIDM,2003.[HK04]T.Hernandez and S.Kambhampati Integration of Biological Sources:Current Systems and Challenges Ahead SIGMOD Record.September2004.[Kin04]K.Kintigh et.al..Enabling the sutdy of long-term Human and Social Dynamics:A Cyberinfrastructure for Archaeology.NSF Human and Social Dynamics Program.2004.(A$10K planning grant involving10PIs)[FKL97] D.Florescu,D.Koller,and ing probabilistic information in data integration.Proceeding of the International Conference on Very Large Data Bases.,1997.[ZL96]Qiang Zhu and Per-Ake Larson.Developing regression cost models for multidatabase systems.In In Proceedings of PDIS,1996.[GRZZ00]J.Gruser,L.Raschid,V.Zadorozhny,and T.Zhan.Learning response time for websources using query feedback and application in query optimization.VLDB Journal.,9(1):18–37,2000.4[WMY00]W.Wang,W.Meng,and C.Yu.Concept Hierarchy based text database categorization in a metasearch engine environment.In Proceedings of WISE,June2000.[IGS01]P.Ipeirotis,L.Gravano,M.Sahami.Probe,Count,and Classify:Categorizing Hidden Web Databases.In Proceedings of SIGMOD-01,2001.[PF03] A.L.Powell and paring the performance of collection selection algorithms.ACM Transactions on Information Systems.V ol21.No4.October2003.[MYL02]Weiyi Meng,Clement T.Yu,King-Lup Liu:Building efficient and effective metasearch engines.ACM Comput.Surv.34(1):48-89(2002)[CLC95]J.P.Callan,Z.Lu,and W.B.Croft.Searching distributed collections with inference networks.In Proceedings of ACM SIGIR Conference,pages21–28,1995.[GGT99]L.Gravano,H.Garc´ıa-Molina,and A.Tomasic.GlOSS:text-source discovery over the Internet.ACM Transactions on Database Systems,24(2):229–264,1999.[Fuzzy90]J.M.Morrissey.Imprecise Information and Uncertainty in Information Systems.ACM Transactions on Information Systems,vol8,1990.[JWS81] A.Joshi and B.Webber and I.Sag,editors.Elements of Discourse Understanding.Cambridge Uni-versity Press,1981.[Joshi82] A.Joshi.Mutual Beliefs in Question Answering Systems.In Mutual Knowledge edited by N.Smith, 1982.[Motro86] A.Motro.Extending the Relational Model to Support Goal Queries.In Proc.of First International Workshop on Expert Systems,1986.[CCL92]W.W.Chu and Q.Chen and R.Lee.A Structured Approach for Cooperative Query Answering.IEEE TKDE,1992.[Motro90] A.Motro.FLEX:A Tolerant and Cooperative User Interface to Database.IEEE TKDE,vol2(2), 231-245,1990.[Mot98] A.Motro.Vague:A user interface to relational databases that permits vague queries.ACM Transac-tions on Office Information Systems,6(3):187–214,1998.[Bin03]Micheal Ortega-Binderberger.Integrating Similarity Based Retrieval and Query Refinement in Data-bases.PhD thesis,UIUC,2003.[GS+98]R.Goldman,N.Shivakumar,S.Venkatasubramanian,and H.Garcia-Molina.Proximity search in databases.VLDB,1998.[AB+02] B.Aditya,G.Bhalotia,S.Chakrabarti,A.Hulgeri,C.Nakhe,Parag,and S.Sudarshan.BANKS: Browsing and Keyword Searching in Relational Databases.VLDB,2002.[Lowell03]The Lowell Database Research Self Assessment.June2003./Gray/Lowell[NKH03]Z.Nie and S.Kambhampati and T.Hernandez.BibFinder/Statminer:Effectively Mining and Using Coverage and Overlap Statistics in Data Integration.Proc.VLDB,2003.(Demo paper).[NK03a]Z.Nie and S.Kambhampati.Frequency-based Coverage Statistics Mining for Data Integration.IJCAI Workshop on Intelligent Information Integration.2003.5[NK04a]Z.Nie and S.Kambhampati.Frequency-based Approach for Mining Coverage Statistics in Data Integration.Proceedings of ICDE,2004.[NKN05]Z.Nie,S.Kambhampati and U.Nambiar.Effectively mining and using coverage and overlap statistics for data integration.IEEE Transactions on Knowledge and Data Engineering.2005.(To appear). [NK04b]U.Nambiar and S.Kambhampati.Providing Ranked Relevant Results for Web Database Queries.WWW(Alternate Track Papers&Poster),May17-22,2004.[NK04c]U.Nambiar and S.Kambhampati.Mining Approximate Functional Dependencies and Concept Sim-ilarities to Answer Imprecise Queries Proceedings of the Workshop on Web and Databases(WebDB)2004.[NK01]Z.Nie and S.Kambhampati.Joint optimization of cost and coverage of query plans in data integration.Proceedings of ACM CIKM,Atlanta,Georgia.,2001.[NKNV01]Z.Nie,S.Kambhampati,U.Nambiar,and S.Vaddi.Mining source coverage statistics for data inte-gration.Proceedings of Workshop on Web Information and Data Management.,2001.[NKNV02]Z.Nie,U.Nambiar,S.Vaddi and S.Kambhampati.Mining source coverage statistics for data integra-tion.Proceedings of CIKM,2002.[PY01] C.H.Papadimitriou and M.Yannakakis.Multiobjective query optimization.In Proc PODS,2001. [SR86]STEUER R.E.Multiple Criteria Optimization:Theory,Computation and Application.John Wiley, New York.1986.[UJ02]Ulrich Junker:Preference-Based Search and Multi-Criteria Optimization.AAAI/IAAI2002:34-40. [ILW+00]Z.Ives,A.Levy,D.Weld,D.Florescu,and M.Friedman.Adaptive query processing for internet applications.Data Engineering Bulletin,23(2),2000.[FALS99] D.Florescu,I.Manolescu A.Levy,and D.Suciu.Query optimization in the presence of limited access patterns.Proceedings of SIGMOD.,1999.[YLUGM99]R.Yerneni,C.Li,J.Ullman,and H.Garcia-Molina.Optimizing large join queries in mediation systems.Proceedings of ICDE.,1999.[UF98]T.Urhan and M.Franklin.Cost-based query scrambling for initial delays.In Proceedings of SIGMOD, 1998.[DL99] A.Doan and A.Levy.Efficiently ordering plans for data integration.In IJCAI Workshop on Intelligent Information Integration,Stockholm,Sweden.,1999.[NLF99] F.Naumann,U.Lesser,and J.Freytag.Quality-driven integration of heterogeneous information sys-tems.Proceedings of VLDB,1999.[TRV97]Anthony Tomasic,Louiqa Raschid,and Patrick Valduriez.A data model and query processing tech-niques for scaling access to distributed heterogeneous databases in Disco.IEEE Transactions onComputers,special issue on Distributed Computing Systems,1997.[GMPQ+97]Hector Garcia-Molina,Yannis Papakonstantinou,Dallan Quass,Anand Rajaraman,Yehoshua Sagiv, Jeffrey D.Ullman,Vasilis Vassalos,and Jennifer Widom.The TSIMMIS approach to mediation:Datamodels and languages.Journal of Intelligent Information Systems,8(2):117–132,1997.[LKG99]Eric Lambrecht,Subbarao Kambhampati,and Senthil Gnanaprakasam.Optimizing recursive infor-mation gathering plans.In Proc.IJCAI,1999.6。