Strategies for cooperative search in distributed databases

Strategies for Cooperative Search in Distributed DatabasesJosel´ıto J.Chua and Peter E.TischerSchool of Computer Science and Software EngineeringMonash University,Clayton,Victoria3800,AustraliaTel:+61-3-99055200,Fax:+61-3-99055146Email:{jjchua,pet}@.auAbstractWe consider the problem of searching a set of data repos-itories,each managed by an independent agent which is able to handle queries from users,as well as queries from peer agents.Given a query,an agent’s task is tofind a clos-est match to the query over all the distributed databases. The agents search the distributed databases cooperatively by routing queries with each other.However,it is inefficient to route all queries to all agents.In this paper,we present strategies for routing queries efficiently between agents.Keywords:Distributed query processing,query routing, peer-to-peer search1.IntroductionThe growing number of large knowledge bases man-aged by independent institutions necessitates a coopera-tive scheme for information retrieval over these distributed databases[9,10,7].Consider the problem of searching N data reposito-ries,S1,S2,...,S N,for the closest matching item to the query,q,with respect to a distance function,d(·,·).The distance function is any given metric that measures the (dis)similarity or relevance between the data items based on their attributes.The data repositories are managed by their corresponding agents,A1,A2,...,A N.Each agent handles queries from users,as well as queries from its peers.The agents cooperate by routing queries with their peers.Suppose agent A i receives a request for the closest match to the query q.Agent A i retrieves the closest match,y i,b, which the agent canfind in its database,S i.The following questions arise:1.Could the other agents provide a match better thany i,b?2.Is there a way for A i to know that y i,b is the best matchacross all the data repositories,without actually havingto search those repositories?3.To which agents should A i pass the query in its searchfor a better match?It is inefficient to route all queries to all agents.Thequestion of which peer an agent should consult regarding the query may depend on the available links and the cost ofthose links between the agents.Each agent is also likely tobe busy processing queries of their own.An agent can avoid disturbing its peers unnecessarily if it could ascertain whichpeers may have a better match than y i,b.This paper presents a number of strategies for efficient cooperative search between agents.The strategies dependon whether the databases are static or dynamic.In the dis-cussions,we shall assume that the user requires only one ofthe closest matching items to the query.In Section4,weshow how the strategies can be modified easily in situations where K-nearest matches are required.2.Static DatabasesIf the repositories are unlikely to change(static),agentscan have a list of closest matching items between each other. Let y j∈S j and y i∈S i be one of the closest matching items among any pairing of items between S j and S i.AgentA i knows that none of the items in S j can be better than y i,b if:d(y j,y i)≥2h(1) whereh=d(q,y i,b)(2) This is illustrated in Figure1.A analytic proof,based on the triangle inequality,is not hard.One such proof is provided in[2].Agent A i does necessarily need to look up a list of theclosest matching items between S i and all S j,j=i.Con-sider a minimum cost spanning tree(MCST)where the ver-tices represent the databases,S1,S2,...,S N,and the edgej iSFigure1.d(y j,y i)≥2h(b)Figure2.MCST and sub-treesweights are given by:wt(S i,S j)=miny i∈S i,y j∈S jd(y i,y j)(3)Agent A i needs to remember only the closest matching items between S i and those that are connected to S i in the MCST(Figure2-a).Every edge in the MCST partitions the vertices into two subsets(Figure2-b).If:wt(S i,S j)≥2h(4) then agent A i does not need to search the subset containing S j.The triangle inequality guarantees that none of the items in that subset can be a better match than y i,b[1].Agent A i’s search strategy can be summarised as fol-lows:function StaticDB Find Match1(q,h,y)Parametersq:queryh:distance between the query and the initial match.If the query comes from the user,then h=−1.If the query comes from another agent,then h isthe distance between the query and the bestmatch so far.y:best match so far.Its value is undefined if h=−1.1.Find y i,b=arg min yi∈S id(q,y i).Let h i,b=d(q,y i,b).2.If(h=−1)or(h i,b<h),then let h=h i,b andy=y i,b.3.For every S j which is connected to S i in the MCST,look up edge weights wt(S i,S j).Sort the edge weights in increasing order.Starting from the edge with the smallest weight,do the following for every edge:(a)If(wt(S i,S j)≥2h),then go to Step4.(b)Otherwise,request agent A j to perform a search,given the query q,and the current h and y.Lety j,b be the closest match A j canfind,and leth j,b=d(q,y j,b).(c)If(h j,b<h)then let h=h j,b and y=y j,b.4.Return h and y.end functionIn situations where it may be difficult to rely on the MCST,then the function can be modified so that Step3 is done for all S j,j=i.Without the MCST,however,the agents need a map to route the query in a way that will not route the query around in circles.One solution is to pass a list offlags that indicate which agents have already been consulted with the query q.Moreover,Step3can be done in“parallel,”in the sense that agent A i does not need to wait for a reply from an agent before it can query other agents.However,the ability to perform simultaneous queries may very between agents,de-pending on factors such as processing load and bandwidth. Let k i be the number of simulaneous queries agent A i can make.The modified strategy is as follows:function StaticDB Find Match2(q,h,y,L) Parametersq,h,y:(as before).L:index set of agents to whom the querycan be routed.Typically,L={1,2,...,N}when h=−1.However,the user maychoose to indicate a selection of databases.1.Find y i,b=arg min yi∈S i d(q,y i).Let h i,b=d(q,y i,b).2.If(h=−1)or(h i,b<h),then let h=h i,b andy=y i,b.3.Remove i from the list L.Also remove from L all jwhich yields wt(S i,S j)≥2h.4.Divide the resulting L into k i subsets,L1,...,L ki .For every subset,L t,do the following:(a)Choose j∈L t.(b)Request agent A j to perform a search,given thequery q,and the current h,y and L t.Let y j,bbe the closest match A j canfind,and let h j,b=d(q,y j,b).(c)If(h j,b<h)then let h=h j,b and y=y j,b.5.Return h and y.end function3.Dynamic DatabasesIf the databases change often,then it may be inefficient to maintain a list of closest matching pair of items between databases.Instead,each cooperating agent chooses a ref-erence item from its database,and advertises its reference item to its peers.The reference item can be chosen arbi-trarily,although reference items are usually chosen either close to the centroid[4],or far away from the centroid[8]. If agent A i wants to determine whether it should ask A j to perform a search,A ifirst requests A j for its reference item, z j,and for the value:R j=maxy j∈S jd(z j,y j)(5)Note that this information can be cached in A i,and updated only when the reference item or the database items change. It would be efficient to havefixed reference items,so that only the R j needs to be updated whenever its value changes. If:R j≤d(q,z j)−h(6) then database S j does not need to be searched(Figure3), because none of the items in S j can be closer to q than the current match[2].On the other hand,if it turns out that S j needs to be searched,then agent A j can limit the search to only those items in S j whose distance to z j is between d(q,z j)−h and d(q,z j)+h[5].In Figure4,for example,only the items in the shaded band centred at z j need to be considered in the search for a better match.If z j isfixed,then the valueofjFigure3.R j≤d(q,z j)−hjFigure4.|d(q,z j)−d(y j,s,z j)|<hd(y j,z j)can be pre-computed as soon as item y j is added to the database.Agent A i’s search strategy can be summarised as fol-lows:function DynamicDB Find Match(q,h,y,L) Parameters q,h,y and L:(as before).1.If(h=−1)then let S =S i.Otherwise,letS ={y i∈S i:|d(q,z i)−d(y i,z i)|<h}2.Find y i,b=arg min yi∈Sd(q,y i)Let h i,b=d(q,y i,b).3.If(h=−1)or(h i,b<h),then let h=h i,b andy=y i,b.4.Remove i from the list L.Also remove from L all jwhich yield R j≤d(q,z j)−h.The values of z j and R j can be obtained either from A i’s cached copy,or by requesting their values from A j.5.Divide the resulting L into k i subsets,L1,...,L ki.For every subset,L t,do the following:(a)Choose j∈L t.(b)Request agent A j to perform a search,given thequery q,and the current h,y and L t.Let y j,bbe the closest match A j canfind,and let h j,b=d(q,y j,b).(c)If(h j,b<h)then let h=h j,b and y=y j,b.6.Return h and y.end function4.ConclusionsA query may have several closest matches throughout the entire collection of databases.The strategies limit the search based on h,and yield only one of the closest matches.If a number of close matches are required,then h can be replaced byh=c·d(q,y i,b)(7) where c≥1is a scaling constant,depending on how many close matches to the query are required.The functions de-fined above can be modified easily to return a set of close matches,instead of just a single match.The technique performs best if the agents specialise in a particular group of data such that each database is themat-ically coherent.For example,the FGDC National Geospa-tial Data Clearinghouse[3]offers a gateway to a collec-tion of over250spatial data servers managed by vari-ous institutions.The databases are related thematically, but distributed spatially[6].This avoids overlaps between databases,which can affect the effectiveness of the strate-gies presented here.Data items that are repeated across repositories need to be tagged as such.Of interest to the authors are distributed case-based rea-soning applications.One of the criticisms of case-based rea-soning is that it has to lug around its knowledge base.The solution proposed in this paper allows cooperating agents to act as a large distributed case-based reasoning system.References[1]J.Chua and P.Tischer.Minimal cost spanning trees fornearest-neighbour matching.In M.Mohammadian,editor, Computational Intelligence for Modelling,Control and Au-tomation:Intelligent Image Processing,Data Analysis and Information Retrieval,pages7–12.IOS Press,1999.[2]J.J.Chua.Fast full-search equivalent nearest-neighboursearch algorithms.Master of Computing,School of Com-puter Science and Software Engineering,Faculty of Infor-mation Technology,Monash University,1999.[3]Federal Geographic Data Committee(FGDC)National Spatial Data Clearinghouse./data/data.html.[4]K.Fukunaga and P.M.Narendra.A branch and bound al-gorithm for computing k-nearest neighbors.IEEE Transac-tions on Computers,24(7):750–753,July1975.[5] C.-M.Huang,Q.Bi,G.S.Stiles,and R.W.Harris.Fast fullsearch equivalent encoding algorithms for image compres-sion using vector quantization.IEEE Transactions on Image Processing,1(3):413–416,July1992.[6]R.Leow and K.Taylor.Efficient web access to dis-tributed biological collections using a taxonomy browser.In 12th International Conference on Scientific and Statistical Database Management,pages25–38,2000.[7]M.Rio,J.Macedo,and V.Freitas.Cooperative agents indistributed indexing and retrieval.In1997IEEE Interna-tional Conference on Intelligent Processing Systems(ICIPS ’97),volume1,pages165–169,Oct.1997.[8]M.Shapiro.The choice of reference points in bestfilemunications ACM,20(5):339–343,May 1977.[9]S.Waterhouse,D.M.Doolin,G.Kan,and Y.Faybishenko.Distributed search in P2P networks.IEEE Internet Comput-ing,6(1):68–72,Jan/Feb2002.[10] B.Yu and M.P.Singh.An agent-based approach to knowl-edge management.In2002Conference on Information and Knowledge Management,pages642–644,2002.。