ZHANGYUE_1421959_Colin cooper_PremilinaryProjectReport (1)

MSc Preliminary Project ReportA Study of Hierarchical Community Structures in Social Networks with a Comparison of AlgorithmsStudent Name:Yue ZhangStudent ID:1421959Supervisor Name:Colin CooperIntroductionIn recent decades,with the wide popularity of Internet,topology has been studied further and further in this field.Clustering is one of the most significant application. As we know that,everything and every one is only about six steps away,which means if I want to know a person who I do not know before,I only need to contact with6 people.This theory is“six degrees of separation”which is stated by Stanley Milgram in1967[1].1-1six degrees of separationBecause of this famous theory social network has been concerned more and more by us.Until now many social network websites have been a part of our lives,such as Facebook,Flicker and Tweeter.This kind of social network let our world become smaller.In this cyber world every single individual is a point,the relationship between the individuals is the edge.Some points can constitute a cluster,all the clusters can constitute a entire Social network[2].However how to evaluate the relationship between the point and point,cluster and cluster is a problem need to be solved.Cluster analysis is the approach to group a set of data in such a way points in the same cluster are more similar to the others than the other points in other clusters.Actually,cluster analysis is not a algorithm,but it is always regarded as a general task to solve the problem.Facing different kind ofclusters,various algorithms can be adopted to achieve the aim.Specifically,in data mining,hierarchical clustering which also called HCA or hierarchical cluster analysis is a very important method of cluster analysis which seeks to build up a hierarchy of data.In my project I will study the importance of hierarchical community structures in social networks.Besides,I will make a comparison between different algorithms used to discover the communities.It includes four key points:1.Study and realize the existing methods of detecting hierarchical clustering in graphs In this part I will study existing algorithms which are used to solve the problem, such as greedy algorithm and G-N algorithm.2.Apply a subset of these algorithms to real social network data.In this part I will apply a subset of the algorithms I have mentioned to real problem using Matlab.3.Study and analysis the results to find out how significant the hierarchical clustering is in the social networks being studied.Discover the complexity and time-consuming of hierarchical clustering and state the importance of using this method to handle the problem.pare the hierarchical clustering algorithms and discuss any problems found-for example,how does(or should)an algorithm decide to stop digging deeper for sub-communities?BackgroundIn the process of data mining,hierarchical is a effective method.Generally speaking, strategies for hierarchical clustering fall into2types[3]:Agglomerative(move up the hierarchy):1.This belongs to“bottom up”method.To begin with,every single point locates in its cluster,and the if the distance between two points are below the criteria,they will cluster into the same cluster.The complexity of this agglomerative clustering is O(n3).In this method it will consume a long time to handle a large data set.For some cases,the optimization result of complexity is O(n2).2.Divisive(move down the hierarchy):This belongs to“top down”method.To begin with,all the point are in the same cluster,and then those point will split into different cluster based on the criteria.The complexity of this divisive clustering is O(2n)which will consume more time than agglomerative clustering.In general,it always uses a dendrogram(2-2)to show the results of hierarchical cluster analysis,and the core method of merging and splitting cluster is greedy algorithm.2-1Raw data[4]2-2dendrogram[5]The basic process of hierarchical clustering can be divide into4step:Suppose there N point which are waiting for clustering[6]:1.Initialization,every single points belongs to its own cluster,then calculate the distance between a pair of clusters to evaluate the similarity between them.2.Seek the nearest pair of cluster and merge them into one cluster so that the number of the clusters will be less.3.Recalculation the distance between the previous clusters and new clusters and evaluate the similarity between them.4.Repeat step2and3to all the point get into the same cluster,finish.The whole process is to build up a dendrogram,in this process we need to set a threshold in step2when the distance is larger than this threshold the iteration can be finished.There several metrics can be used to calculate the distance[7].Name FormulaManhattandistanceMaximumdistanceSquared EuclideandistanceEuclideandistanceMahalanobisS is the covariance matrix distanceTable1Another key point is to chose the appropriate linkage criteria in the process of hierarchical clustering in step3,there are3common method to evaluate the similarity between two clusters.The names and the formula are shown in the table-2below.Name FormulaSingle-linkageclusteringComplete-linkageclusteringAverage-linkageclusteringTable2Single Linkage:It is also called nearest-neighbor,this means adopting2nearest points which belongs two different cluster and regard the distance between these2points as the distance between their clusters.However it is easy to make a effect called“chaining”.Possibly,two clusters are far from each other,but there are two near points in each of the cluster,these two clusters may merge together.So that,it will get a loose result finally.Complete Linkage:This is the opposite of the Single Linkage.This method will make mistakes of that two clusters are near to each other,however,two points which belong to their own clusters are far from each other,it will due to these two cluster can not merge together.Group average:Compare with single-linkage and complete-linkage methods this method is the most reasonable.This will calculate the average distance in every cluster.However this is the most complex in3methods.It need duo-loop code to calculate the result[8].2-3Demonstrate of hierarchical clusteringBesides,there are another effective approach“G-N algorithm”which is applied in real social network.G-N algorithm is stated by Michelle Grivan and Mark Newman in 2002.The basic process are3steps[9]:1.Calculate the betweenness of all edges in the network.2.Remove the edge which has the highest betweenness3.Recalculate the betweenness of all edges affected by the removal4.Repeat step1and2to no edges remainhighestbetweennessremove2-4G-N algorithmBesides,I have many things to study in the future all the algorithm and application in the real world I need to study in depth,especially in terms of coding.Gantt chartReferences[1].Milgram,S.(1967)Psychol.Today2,60–67.[2].Opsahl,T.and Panzarasa,P.(2009).Clustering in weighted networks.Social Networks,31(2),pp.155-163.[3].Rokach,Lior,and Oded Maimon.(2005).Clustering methods.Data miningand knowledge discovery handbook.Springer US,pp.321-352.[4][5].Wikipedia,(2015).Hierarchical clustering.[online]Available at:/wiki/Hierarchical_clustering[Accessed27Apr.2015].[6],(2015).Hierarchical .[online]Available at:/zhubo22/article/details/24472639[Accessed27Apr.2015].[7]"The DISTANCE Procedure:Proximity Measures".SAS/STAT9.2UsersGuide.SAS Institute.Retrieved2009-04-26[8],(2015).Clustering(5):Hierarchical Clustering«Free Mind.[online]Available at:/?p=407[Accessed27Apr.2015]. [9]Girvan M.and Newman M.E.J.,Community structure in social and biologicalnetworks,A99,7821–7826(2002)。