Predicting Popular Messages in Twitter

Predicting Popular Messages in TwitterLiangjie Hong Ovidiu Dan Brian D.Davison Dept.of Computer Science and Engineering,Lehigh UniversityBethlehem,P A18015USA{lih307,ovd209,davison}@ABSTRACTSocial network services have become a viable source of informa-tion for users.In Twitter,information deemed important by the community propagates through retweets.Studying the characteris-tics of such popular messages is important for a number of tasks, such as breaking news detection,personalized message recommen-dation,viral marketing and others.This paper investigates the prob-lem of predicting the popularity of messages as measured by the number of future retweets and sheds some light on what kinds of factors influence information propagation in Twitter.We formulate the task into a classification problem and study two of its variants by investigating a wide spectrum of features based on the content of the messages,temporal information,metadata of messages and users,as well as structural properties of the users’social graph on a large scale dataset.We show that our method can successfully pre-dict messages which will attract thousands of retweets with good performance.Categories and Subject DescriptorsH.3.3[Information Storage and Retrieval]:Information Search and RetrievalGeneral TermsAlgorithms,Design,ExperimentationKeywordsInformation Diffusion,Social Media,Classification,Microblogs 1.INTRODUCTIONSocial network services such as Facebook,Myspace and Twit-ter have become important communication tools for many online users.Such websites are increasingly used for communicating breaking news,eyewitness accounts and organizing groups of ers of these services have become accustomed to receiving timely updates on important events,both of personal and global im-portance.However,theflow of information within the social graph can lead to two problems.First,even users with few friends can experience information overload due to the high volume of mes-sages.Second,users with a small number of social connections might miss important messages which do not reach them.In this second situation it would be useful to recommend interesting mes-sages to such users,which might lead them to follow new users and Copyright is held by the author/owner(s).WWW2011,March28–April1,2011,Hyderabad,India.ACM978-1-4503-0637-9/11/03.Table1:Sample tweets with high retweets RT@paramore Watch the World Premiere of Paramore’s new videofor’Brick By Boring Brick’#paramoreRT@CamaroWRX:http://bit.ly/794Edz because everyone#needsmorebradley RT@narendra:Please RT.Some recent thoughts on the empathic web.that made the Huffington Post-http://bit.ly/9WyxnTaugment their social network.Both problems require us to deter-mine the importance of messages.In this work we use“retweets”as a measure of popularity and address the problem by utilizingmachine learning techniques to predict whether and how often newmessages will be retweeted in the future.We treat the problem asa classification task.First,we train a binary classifier with positiveand negative examples of messages which will be retweeted in the future.Second,we train a multi-class classifier which predicts thevolume range of future retweets for a new message.To build theclassifiers,we investigate a wide spectrum of features to determinewhich ones can be successfully used as predictors of popularity,including the content and topical information of messages,graphstructural properties of users,temporal dynamics of retweet chains and meta-information of users.We conduct our experiments on alarge scale dataset.The results suggest that we can successfullypredict whether a message will be retweeted or not.Furthermore,we can also predict the volume of retweets with good predictiveperformance.Similar work by Suh et al.[2]studied a variety of fac-tors that might influence retweets without explicitly showing their effectiveness in a classification framework.2.PROBLEMS&FEATURESWe cast the problem of predicting the popularity of messages into two classification problems:1)a binary classification prob-lem that predicts whether or not a message will be retweeted,and,2)a multi-class classification problem that predicts the volume ofretweets a particular message will receive in the near future.Twomessages are considered identical if they share the same MD5 value.We sort all such messages by ascending time order,form-ing a chain of messages.Thefirst message in the chain is the ear-liest version of the message in our dataset and all later messagesin the same chain should contain at least one“RT”term.For thebinary classification problem,for n messages in a chain,thefirst n−1messages are considered as“positive instances”and the last one as a“negative instance”.In addition,all other messages whichare not in any chains are also considered as“negative instances”.For the multi-class classification problem,the number of messagesfollowing a specific message is treated as the number of retweetsthis message will receive.Therefore,for all messages,we assigna non-negative integer value to represent the number of subsequent retweets.It is difficult to predict the exact number of retweets aFigure1:Retweet distributionTable2:The performance of two classification tasks Methods Precision Recall F1 Retweet Before0.6850.1660.263 TF-IDF0.3100.2490.276Our Method0.9930.4350.603 Without User Retweet0.9930.3990.569Without Degree Distr.0.4730.4700.471Without Retweet Before0.6780.2500.364Previous Method Temporal User Activities Class Accu.Accu.Accu.00.99990.99990.999910.03380.11390.149020.05220.43090.434630.98960.90040.9209 particular message will receive due to the fact that the maximum volume of retweets in the test set may be much higher than in its training set.Therefore,rather than directly predicting these integer values,we relax the problem by defining several categories to rep-resent the volume of retweets and predict these categories instead. We investigate a wide range of possible features including content features,graph topological features,temporal features and meta-data features,all of which are applicable to the two classification tasks.For content features,we use TF-IDF scores as a baseline and utilize Latent Dirichlet Allocation(LDA)to obtain the topic distributions for each message.For topological features,multiple popular features are considered,including global and local struc-tures,such as PageRank,degree distribution,local clustering coef-ficient and reciprocal links.We also assume that users may track hot topics and may quickly switch topics over time.Therefore, for temporal features,we measure the time difference between the current message and the origin within the chain,the time differ-ence between the current and the previous tweet,the average time difference of consecutive messages in the same chain and the av-erage time a user’s messages get retweeted.For meta information features,we are mainly interested in whether a message has been retweeted before(Retweet Before)and how many times in the past the messages generated by a particular user have been retweeted (User Retweet).Other user activities such as the total number of messages a user produced are also considered.3.EXPERIMENTSWe run our experiments with messages collected in November and December2009,as well as the immediate social graph of the users which are active in this time period.The dataset contains 10,612,601messages and2,541,178users.For the binary classi-fication problem,we report Precision,Recall and F1score.For the multi-class classification problem,we report accuracy for each class.Instead of directly predicting the exact number of retweets, we divide the messages into different retweet volume“classes”:0:Not retweeted,1:Retweeted less than100,2:Retweeted less than10000,3:Retweeted more than10000,similarly to[1].We binned messages with retweets(size equals10)and plot the dis-tribution in Figure1,showing that a large number of messages do not receive any significant retweets(less than10)and a significant amount of messages also have more than10000retweets.Three messages with most retweets in Class3are shown in Table1as examples where thefirst one is about entertainment,the second is related to hockey and the third one is related to IT.We use Logistic Regression as the classifier,as we found it to be the most stable in preliminary experiments.We train the classifier on one week and test it on the next week and the results are the averages over four such experiments.We compare the results to two obvious baselines.One is based on whether a messages has been retweeted or not in the past,and the other uses the TF-IDF scores for terms in each message.The re-sults are shown in the upper part of Table2.For both baselines,the F1score is roughly around0.27,with relatively low Precision and Recall scores.For our method,the best performance is achieved by using“TF-IDF”,“LDA”,“degree distribution”,“Retweet Be-fore”and“User Retweet”features while all measures are much better than both baselines.In order to investigate the effective-ness of each feature,we remove individual features and see how the performance is affected.This is shown in the middle of Ta-ble2,where thefirst column shows which feature have been taken away.We observe that the performance drops significantly when “degree distribution”and“Retweet Before”are removed,implying that they contribute greatly to the overall performance.We con-jecture that in Twitter,users mainly see messages from theirfirst level friends whom they are directly following.Therefore,the like-lihood that some of a user’s followers will retweet a new message depends on how many followers the user has.Thus,the effect of in-degree is obvious to some extent.For out-degree,since many messages which will be retweeted in the future have already been retweeted,a user with limited friends may face difficulties in see-ing such messages.For multi-class classification,the lower part of Table2shows the performance achieved by using the best fea-tures in the binary classification.The results are extremely good in Class0and Class3but poor in Class1and Class2,indicating that messages only attracting a small audience might be very dif-ferent from the messages which receive huge numbers of retweets. After adding“Temporal”features,shown in the third column,the performance on Class0still holds but improvements in Class1and Class2are observed with some decrease of performance of Class 3.One explanation is that unlike popular messages which receive tens of thousands of retweets,normal messages only attract a small audience and users lose interest in them very quickly.Therefore, temporal features have a stronger effect on these messages with low and medium volume of retweets,compared to highly popular er activity features can further improve the performance marginally,shown in the fourth column in Table2. AcknowledgmentsThis material is based in part upon work supported by the NSF under Grant Number IIS-0545875.4.REFERENCES[1] E.Khabiri,C.-F.Hsu,and J.Caverlee.Analyzing and predictingcommunity preference of socially generated metadata:A case study on comments in the digg community.In ICWSM,2009.[2] B.Suh,L.Hong,P.Pirolli,and E.H.Chi.Want to be retweeted?Large scale analytics on factors impacting retweet in Twitter network.In SocialCom,2010.。