[twitter-dev] Re: how are the ten trends born?
Some of you might be interested in this: http://blog.linkibol.com/post/How-to-Build-a-Popularity-Algorithm-You-can-be-Proud-of.aspx On Fri, Oct 2, 2009 at 16:43, Nick Arnett wrote: > > > On Fri, Oct 2, 2009 at 1:00 PM, David Fisher wrote: > >> >> >> For the most part its just a frequency count of words over a short >> time period, minus stop words, filtering out usernames (notice @foo is >> never a trend) and URLs. How it combines "Wave OR Google Wave" I'm >> unsure of, and then there's some basic spam filtering in there >> additionally. > > > I hope it isn't that naive -- do you know what they're doing, or are you > speculating? > > For one thing, systems that count the unique individuals mentioning a term, > rather than just raw term counts, are far more accurate in predictive > modeling. > > Furthermore, Twitter has plenty of data to incorporate traffic and social > network analysis to further improve this "buzz" analysis. > > FYI, I've been doing social network buzz analytics for about ten years and > have some patents in that area (which don't belong to me, but to > Nielsen/Buzzmetrics). > > Nick > > -- Abraham Williams | Community Evangelist | http://web608.org Hacker | http://abrah.am | http://twitter.com/abraham Project | http://fireeagle.labs.poseurtech.com This email is: [ ] blogable [x] ask first [ ] private.
[twitter-dev] Re: how are the ten trends born?
On Fri, Oct 2, 2009 at 1:00 PM, David Fisher wrote: > > > For the most part its just a frequency count of words over a short > time period, minus stop words, filtering out usernames (notice @foo is > never a trend) and URLs. How it combines "Wave OR Google Wave" I'm > unsure of, and then there's some basic spam filtering in there > additionally. I hope it isn't that naive -- do you know what they're doing, or are you speculating? For one thing, systems that count the unique individuals mentioning a term, rather than just raw term counts, are far more accurate in predictive modeling. Furthermore, Twitter has plenty of data to incorporate traffic and social network analysis to further improve this "buzz" analysis. FYI, I've been doing social network buzz analytics for about ten years and have some patents in that area (which don't belong to me, but to Nielsen/Buzzmetrics). Nick
[twitter-dev] Re: how are the ten trends born?
It's definitely not going to be based on sheer volume, but rather delta based on some averages. You need to filter out natural language too, as that can be all over the place. Although it's a different (and also secret) algorithm, take a look at http://twitscoop.com and watch their trend cloud change, might help give some ideas for algorithms etc. On Oct 2, 9:00 pm, David Fisher wrote: > It's pretty simple, but with a few twists. > > First of all, remember that everything that Twitter does is done with > simplicity and efficiency in mind. > > For the most part its just a frequency count of words over a short > time period, minus stop words, filtering out usernames (notice @foo is > never a trend) and URLs. How it combines "Wave OR Google Wave" I'm > unsure of, and then there's some basic spam filtering in there > additionally. > > I was theorizing that some of it is based on accelleration of words > above their standard volume. Apple for example is always talked about > a great deal, but isn't always trending. Sometimes it has a greater > volume than other trending words, but it doesn't trend. Yet some > things stick around for a long time like IranElection. I need to dig > into this more. I almost had it fully modeled at one point, then lost > the code (damn you version control mistakes) > > dave > > On Oct 2, 10:54 am, Nigel Cannings > wrote: > > > > > @secretbear did it first in the halcyon days of the PubSub Firehose... > > I'd ask him > > > == > > > Why not encrypt the mail you send me? You never know who's looking. > > If you use Firefox, why not use the FireGPG plugin to make it easy > > (http://getfiregpg.org) > > > Get my key from:http://keyserver.pgp.com/ > > > == > > > On Thu, Oct 1, 2009 at 4:20 PM, Martin Dudek > > wrote: > > > > Good morning > > > > wonder if somebody knows how twitter determines the ten trends it > > > declares every five minutes? Is this a pure word/phrase frequency > > > algorithm or some more complexity behind. > > > > Thanks > > > > martin
[twitter-dev] Re: how are the ten trends born?
I am doing research on Twitter trends and defining them in general, in order to develop better algorithms. So this is very timely and instead of starting a new thread, perhaps we can expand it a bit. There's quite a few services doing twitter trends already, and most are doing it with a simple pragmatic approach as defined by David above. I wonder if the operators of those services could say a few words here about their approaches. If we keep an overview of the field and develop ways to compare and evaluate trend-tracking algorithms, it will become science and will work better. Cheers, Alexy
[twitter-dev] Re: how are the ten trends born?
It's pretty simple, but with a few twists. First of all, remember that everything that Twitter does is done with simplicity and efficiency in mind. For the most part its just a frequency count of words over a short time period, minus stop words, filtering out usernames (notice @foo is never a trend) and URLs. How it combines "Wave OR Google Wave" I'm unsure of, and then there's some basic spam filtering in there additionally. I was theorizing that some of it is based on accelleration of words above their standard volume. Apple for example is always talked about a great deal, but isn't always trending. Sometimes it has a greater volume than other trending words, but it doesn't trend. Yet some things stick around for a long time like IranElection. I need to dig into this more. I almost had it fully modeled at one point, then lost the code (damn you version control mistakes) dave On Oct 2, 10:54 am, Nigel Cannings wrote: > @secretbear did it first in the halcyon days of the PubSub Firehose... > I'd ask him > > == > > Why not encrypt the mail you send me? You never know who's looking. > If you use Firefox, why not use the FireGPG plugin to make it easy > (http://getfiregpg.org) > > Get my key from:http://keyserver.pgp.com/ > > == > > > > On Thu, Oct 1, 2009 at 4:20 PM, Martin Dudek > wrote: > > > Good morning > > > wonder if somebody knows how twitter determines the ten trends it > > declares every five minutes? Is this a pure word/phrase frequency > > algorithm or some more complexity behind. > > > Thanks > > > martin
[twitter-dev] Re: how are the ten trends born?
@secretbear did it first in the halcyon days of the PubSub Firehose... I'd ask him == Why not encrypt the mail you send me? You never know who's looking. If you use Firefox, why not use the FireGPG plugin to make it easy (http://getfiregpg.org) Get my key from: http://keyserver.pgp.com/ == On Thu, Oct 1, 2009 at 4:20 PM, Martin Dudek wrote: > > Good morning > > wonder if somebody knows how twitter determines the ten trends it > declares every five minutes? Is this a pure word/phrase frequency > algorithm or some more complexity behind. > > Thanks > > martin >
[twitter-dev] Re: how are the ten trends born?
On Thu, Oct 1, 2009 at 10:05 PM, Martin Dudek wrote: > > just curious ... That can be the most difficult and dangerous problem of all! Nick
[twitter-dev] Re: how are the ten trends born?
just curious ... On Oct 2, 9:55 am, Nick Arnett wrote: > On Thu, Oct 1, 2009 at 8:20 AM, Martin Dudek wrote: > > > > > Good morning > > > wonder if somebody knows how twitter determines the ten trends it > > declares every five minutes? Is this a pure word/phrase frequency > > algorithm or some more complexity behind. > > I wouldn't expect an answer to that. I'd bet quit a bit of money that > Twitter considers the algorithm to be a trade secret. If they disclosed it, > they'd be making it easier for people to manipulate the rankings. > > What problem are you trying to solve? > > Nick
[twitter-dev] Re: how are the ten trends born?
On Thu, Oct 1, 2009 at 8:20 AM, Martin Dudek wrote: > > Good morning > > wonder if somebody knows how twitter determines the ten trends it > declares every five minutes? Is this a pure word/phrase frequency > algorithm or some more complexity behind. I wouldn't expect an answer to that. I'd bet quit a bit of money that Twitter considers the algorithm to be a trade secret. If they disclosed it, they'd be making it easier for people to manipulate the rankings. What problem are you trying to solve? Nick
[twitter-dev] Re: how are the ten trends born?
it's MAGIC! On Thu, Oct 1, 2009 at 09:20, Martin Dudek wrote: > > Good morning > > wonder if somebody knows how twitter determines the ten trends it > declares every five minutes? Is this a pure word/phrase frequency > algorithm or some more complexity behind. > > Thanks > > martin > -- Internets. Serious business.
