[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
So it appears that most (if not all) of the trending bots have been removed from search results as of 2 days ago, nice. I have also noticed that the referral traffic from my bot links have dropped about 85% in the last two days. Not complaining; I'm all for this change. Just noticing out loud. -Chad
[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
oh noes... now even al3x is a trending bot: http://twitter.com/al3x/status/1302097888 On Mon, Mar 9, 2009 at 3:07 PM, TjL wrote: > > The more I think about this, the more I realize that there really > ought to be a "logged in" version of Twitter Search. > > Not that you would HAVE to login, but IF you were logged in: > > People you have BLOCKED would not appear. > > People who have private accounts you follow WOULD appear. > > That way you could just block bots and have them excluded from results. > > Personal choice, FTW. > > Now it just needs to be implemented :-) > > TjL >
[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
The more I think about this, the more I realize that there really ought to be a "logged in" version of Twitter Search. Not that you would HAVE to login, but IF you were logged in: People you have BLOCKED would not appear. People who have private accounts you follow WOULD appear. That way you could just block bots and have them excluded from results. Personal choice, FTW. Now it just needs to be implemented :-) TjL
[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
On Mon, Mar 9, 2009 at 6:26 AM, Paul Kinlan wrote: > On the topic of bots, http://www.itsabot.com works pretty well most of the > time. My list is now updating live: http://www.twurlednews.com/twitter-bots/ Nick
[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
On the topic of bots, http://www.itsabot.com works pretty well most of the time. Paul 2009/3/9 TjL > > On Sun, Mar 8, 2009 at 11:20 PM, Chad Etzel wrote: > > > > On Sun, Mar 8, 2009 at 9:04 PM, TjL wrote: > >> > >> On Sun, Mar 8, 2009 at 7:37 PM, Cameron Kaiser > wrote: > >>> > IMO, "trend bots" should have to be registered with Twitter (they say > what they are going to use their API access for, right?) and should > excluded from Twitter search. > >>> > >>> How do you enforce bots registering as bots, however? > >> > >> Well, revoking API whitelisting for any that don't register properly > >> would be a good first step. > > > > Huh? Bots don't need any sort of whitelisting to exist or function. > > It's trivial to create and run one. It won't be so trivial once OAuth > > hits, but I'm sure it won't be much of a barrier. > > Ah. Well. My mistake. > > Thanks > > TjL >
[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
On Sun, Mar 8, 2009 at 11:20 PM, Chad Etzel wrote: > > On Sun, Mar 8, 2009 at 9:04 PM, TjL wrote: >> >> On Sun, Mar 8, 2009 at 7:37 PM, Cameron Kaiser wrote: >>> IMO, "trend bots" should have to be registered with Twitter (they say what they are going to use their API access for, right?) and should excluded from Twitter search. >>> >>> How do you enforce bots registering as bots, however? >> >> Well, revoking API whitelisting for any that don't register properly >> would be a good first step. > > Huh? Bots don't need any sort of whitelisting to exist or function. > It's trivial to create and run one. It won't be so trivial once OAuth > hits, but I'm sure it won't be much of a barrier. Ah. Well. My mistake. Thanks TjL
[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
On Sun, Mar 8, 2009 at 9:04 PM, TjL wrote: > > On Sun, Mar 8, 2009 at 7:37 PM, Cameron Kaiser wrote: >> >>> IMO, "trend bots" should have to be registered with Twitter (they say >>> what they are going to use their API access for, right?) and should >>> excluded from Twitter search. >> >> How do you enforce bots registering as bots, however? > > Well, revoking API whitelisting for any that don't register properly > would be a good first step. Huh? Bots don't need any sort of whitelisting to exist or function. It's trivial to create and run one. It won't be so trivial once OAuth hits, but I'm sure it won't be much of a barrier. -chad
[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
On Sun, Mar 8, 2009 at 7:37 PM, Cameron Kaiser wrote: > >> IMO, "trend bots" should have to be registered with Twitter (they say >> what they are going to use their API access for, right?) and should >> excluded from Twitter search. > > How do you enforce bots registering as bots, however? Well, revoking API whitelisting for any that don't register properly would be a good first step. Just a checkbox/radio button on the API whitelisting form should do. That will deal with any new ones. As for existing ones, well, just a matter of watching the Trending Topics and ID'ing trending bots. Add a banner on search.twitter.com which links to a blog post on the Twitter blog for more information. TjL
[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
> IMO, "trend bots" should have to be registered with Twitter (they say > what they are going to use their API access for, right?) and should > excluded from Twitter search. How do you enforce bots registering as bots, however? -- personal: http://www.cameronkaiser.com/ -- Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckai...@floodgap.com -- In defeat, unbeatable; in victory, unbearable. -- Churchill, on Montgomery -
[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
Here's the latest example of bots drowning out actual posts: http://tntluoma.com/temp/TrendBots.png (screenshot) Of 11 visible results: 4 are actual content (although 2 are ReTweets, which is another issue altogether, but at least they are humans) 7 are bots. Almost 2x as many bots as actual results. IMO, "trend bots" should have to be registered with Twitter (they say what they are going to use their API access for, right?) and should excluded from Twitter search. If people want to FOLLOW one of the Trend bots, fine, but they are choking the usefulness of the trending topics list and with nothing but echoing noise. It's getting as bad as trying to go to Google to find hotel information. TjL
[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
On Fri, Mar 6, 2009 at 12:05 PM, Chad Etzel wrote: > > Just make sure not to feed @RobotTweeters to itself... you may rip a > blackhole in the tweet/space continuum! I endeavor to never cross the streams. Nick
[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
On Fri, Mar 6, 2009 at 2:56 PM, Nick Arnett wrote: > I created @RoboTweeters this morning. I'll probably start feeding it screen > names and ids of the ones I find, since that's quite simple. > Nick Just make sure not to feed @RobotTweeters to itself... you may rip a blackhole in the tweet/space continuum!
[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
On Fri, Mar 6, 2009 at 11:49 AM, Cameron Kaiser wrote: > > > I agree, most ppl probably won't abide by any guidelines that they > > have to 'voluntarily' follow in order to identify themselves at bots. > > It's pretty darn easy to tell if something is a trend bot or not... > > especially with the username :) Matt even said they've identified > > them (uh oh, i'm on some kind of twitter watchlist but who watches > > the watchlist?) > > Dibs on @drmanhattanbot. I created @RoboTweeters this morning. I'll probably start feeding it screen names and ids of the ones I find, since that's quite simple. Nick
[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
> I agree, most ppl probably won't abide by any guidelines that they > have to 'voluntarily' follow in order to identify themselves at bots. > It's pretty darn easy to tell if something is a trend bot or not... > especially with the username :) Matt even said they've identified > them (uh oh, i'm on some kind of twitter watchlist but who watches > the watchlist?) Dibs on @drmanhattanbot. -- personal: http://www.cameronkaiser.com/ -- Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckai...@floodgap.com -- All things are possible, except skiing through a revolving door. ---
[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
On Thu, Mar 5, 2009 at 8:09 PM, TjL wrote: > > Specifically > > 1) There are WAY to many "trending topic" bots which fill search > results with useless clutter > > 2) I'd love to see a "trending topics" list that does NOT include hash > tags, you know, to find out what ordinary people are talking about :-) > > I know this is the wrong place for it (sorry) but I'm not sure where else > to go. Wrong place? I don't think so... I've got code that's doing a fairly job of identifying robots -- essentially any user who posts an unusually high number of tweets in a short time and doesn't follow many people. (My goal is to analyze tweets by real people, not automated systems.) The code is relatively simple -- it looks at how many updates the user has made over time, short-term and long-term. My long-term period is 12 days, but that misses the new ones that seem to pop up every day, so I also look at the last 12 hours. The long-term one also looks at the follower count. If the user is following less than 100 people, that seems to be a strong clue that it's a bot. I'm sure that I can refine the rules further, but it hasn't been a priority, since this seems to be working reasonably well. There are a bit more than 1,800 in the list, which isn't just for trending bots, but all sorts of automated users. I've added it as a blog page here: http://www.twurlednews.com/twitter-bots/ Grab everything between the pre tags and you'll have a CSV list of screen names and ids. I assume it would be more useful as a plain CSV file. If there's interest in that, I'll make it available and keep it updated. The current update schedule for TwURLed News is 15 minutes, so it should stay fairly fresh. Other than an API, which I don't really have time to create right now, is there another form this data could take that would be useful? I could create a Twitter user and send a tweet every time I identify a new possible bot. But if this is sufficient, I have other things I'd like to work on... ;-) I've been thinking about adding a page that lists them, which would give a filter list that you could use to remove them from results. One of these days, I might even put that in an API. This is something that's been done for years for user-agents on the web, so that advertisers and others can isolate robots from web analytics.
[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
I agree, most ppl probably won't abide by any guidelines that they have to 'voluntarily' follow in order to identify themselves at bots. It's pretty darn easy to tell if something is a trend bot or not... especially with the username :) Matt even said they've identified them (uh oh, i'm on some kind of twitter watchlist but who watches the watchlist?) If twitter themselves ever incorporate auto-updating search results like the special election pages, my bot and its links would pretty much be rendered useless D: -Chad On Fri, Mar 6, 2009 at 11:53 AM, Peter Denton wrote: > I am skeptical that bot devs, (outside of the integrious Jazzy Chad), will > do anything to encourage segregation, as it would probably lead to a nuking > list at some point. I would say this has to be done programatically, with a > "secret sauce" that is known to twitter only. As search is more and more > the golden goose apparent, gaming will be enemy number 1. > - Show quoted text - > > On Fri, Mar 6, 2009 at 8:45 AM, Nick Arnett wrote: >> >> >> On Fri, Mar 6, 2009 at 8:34 AM, Matt Sanford wrote: >>> >>> Hi there, >>> We've talked about this among the search folks a few times. We >>> exclude a bunch of bots and things from influencing trends but then they >>> still get displayed. I just opened a ticket for someone to fix that so we >>> can exclude the trend bots using a parameter or search operator. >>> As far as if this is the correct place for search or not, I think it >>> is. If other Twitter API developers disagree please let me know and I'll >>> start a second group. From my perspective keeping up with one is easier for >>> me to manage … and we're planning to merge the APIs in the next version of >>> the API. >> >> It would be terrific if users could self-identify as bots and that data >> became part of the user profile. Although I'm sure that many people would >> not bother, we'd at least know that some of them definitively are bots. My >> bots self-identify in their description, which people seem to appreciate. >> Hmm. Maybe it would be far easier to simply encourage a hashtag in the >> description - how about #bot? That's something we could do now, without >> Twitter having to make any code changes. Thoughts? >> Nick >
[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
I am skeptical that bot devs, (outside of the integrious Jazzy Chad), will do anything to encourage segregation, as it would probably lead to a nuking list at some point. I would say this has to be done programatically, with a "secret sauce" that is known to twitter only. As search is more and more the golden goose apparent, gaming will be enemy number 1. On Fri, Mar 6, 2009 at 8:45 AM, Nick Arnett wrote: > > > On Fri, Mar 6, 2009 at 8:34 AM, Matt Sanford wrote: > >> Hi there, >> We've talked about this among the search folks a few times. We exclude >> a bunch of bots and things from influencing trends but then they still get >> displayed. I just opened a ticket for someone to fix that so we can exclude >> the trend bots using a parameter or search operator. >> >> As far as if this is the correct place for search or not, I think it >> is. If other Twitter API developers disagree please let me know and I'll >> start a second group. From my perspective keeping up with one is easier for >> me to manage … and we're planning to merge the APIs in the next version of >> the API. >> > > It would be terrific if users could self-identify as bots and that data > became part of the user profile. Although I'm sure that many people would > not bother, we'd at least know that some of them definitively are bots. My > bots self-identify in their description, which people seem to appreciate. > > Hmm. Maybe it would be far easier to simply encourage a hashtag in the > description - how about #bot? That's something we could do now, without > Twitter having to make any code changes. Thoughts? > > Nick >
[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
On Fri, Mar 6, 2009 at 8:34 AM, Matt Sanford wrote: > Hi there, > We've talked about this among the search folks a few times. We exclude > a bunch of bots and things from influencing trends but then they still get > displayed. I just opened a ticket for someone to fix that so we can exclude > the trend bots using a parameter or search operator. > > As far as if this is the correct place for search or not, I think it > is. If other Twitter API developers disagree please let me know and I'll > start a second group. From my perspective keeping up with one is easier for > me to manage … and we're planning to merge the APIs in the next version of > the API. > It would be terrific if users could self-identify as bots and that data became part of the user profile. Although I'm sure that many people would not bother, we'd at least know that some of them definitively are bots. My bots self-identify in their description, which people seem to appreciate. Hmm. Maybe it would be far easier to simply encourage a hashtag in the description - how about #bot? That's something we could do now, without Twitter having to make any code changes. Thoughts? Nick
[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
Hi there, We've talked about this among the search folks a few times. We exclude a bunch of bots and things from influencing trends but then they still get displayed. I just opened a ticket for someone to fix that so we can exclude the trend bots using a parameter or search operator. As far as if this is the correct place for search or not, I think it is. If other Twitter API developers disagree please let me know and I'll start a second group. From my perspective keeping up with one is easier for me to manage … and we're planning to merge the APIs in the next version of the API. Thanks; — Matt Sanford On Mar 6, 2009, at 08:25 AM, TjL wrote: On Fri, Mar 6, 2009 at 11:11 AM, Doug Williams wrote: In your experience, do trending bots have a disproportionate participation in the search results for trending topics? Have you done any analysis like that? I'm not Chad :-) but if you click on any of the Trending Topics and watch for any length of time you'll see scads of trending topic bots popping up. I think the most I counted at one point was like 12 out of the top 20 results. It's insane. TjL
[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
what I think is also important is that the bots are then contributing to the trending itself, and eliminating other legitimate topics from trends. On Fri, Mar 6, 2009 at 8:25 AM, TjL wrote: > > On Fri, Mar 6, 2009 at 11:11 AM, Doug Williams wrote: > > In your experience, do trending bots have a disproportionate > > participation in the search results for trending topics? Have you done > > any analysis like that? > > I'm not Chad :-) but if you click on any of the Trending Topics and > watch for any length of time you'll see scads of trending topic bots > popping up. > > I think the most I counted at one point was like 12 out of the top 20 > results. > > It's insane. > > TjL >
[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
Well, it's kind of a weird feedback loop. Say you are following a trending bot (many many people do, a surprising number to me). As soon as you see a tweet from your favorite trending bot, you click the link and head over to see the results Well, all the other bots are tweeting at about the same time, so as soon as a new trend appears you get a dozen or so trend-bot tweets appearing in the results you just loaded up. I will admit this can be semi-annoying. Disproportionate? I guess it depends on how many results your browser loads by default. Mine is always set to 100, so I can scroll by the bots pretty quickly, but if people are only seeing 25 at a time, they'd have to click "Next" or "Older" to get past the bots. Like I said in my blog post, once you are actually searching for a trend, you don't need a dozen things telling you it's a trend again.. you're already there! Some bots are "worse offenders" than others and just spew all the trends every 5 minutes or retweet people (randomly it seems) that match the trend (not naming names, I'm sure you can figure them out). As a means of driving traffic they are very effective (at least the one I run seems to be). A little over 50% of the traffic to tweetgrid.com/search comes from links posted by my bot. I am not sure if the effectiveness can be attributed to the mere fact that the bot exists, or because it has some useful information attached (e.g. #trend has risen to the #3 trend! ). Very few of the bots seem to talk about the rank of the trend, but mine does, so it has some added value. I think this has helped my bot, and it also means that it gets retweeted quite a bit (another big surprise to me). In all honesty, I started my bot because one of my competitors convinced one of the existing trend bots to link to their site instead of search.twitter.com. I launched my bot in defense. A long, meandering answer to a short question. I am somewhat conflicted on the issue since I run one of these bots, but I will admit I find the greasemonkey script to blow them away quite nice. How's that for a definite maybe? -Chad On Fri, Mar 6, 2009 at 11:11 AM, Doug Williams wrote: > > Chad, > In your experience, do trending bots have a disproportionate > participation in the search results for trending topics? Have you done > any analysis like that? > > Doug Williams > @dougw > - Show quoted text - > On Fri, Mar 6, 2009 at 10:43 AM, Burhan TANWEER wrote: >> I agree with him. Search trends are not available in xml format. I will >> appreciate, if twitter can provide search trends in xml and so that i can >> update my social search engine ExploreWWW.com with search trends in real >> time. >> >> Thanks >> Burhan >> >> On Thu, Mar 5, 2009 at 11:09 PM, TjL wrote: >>> >>> Specifically >>> >>> 1) There are WAY to many "trending topic" bots which fill search >>> results with useless clutter >>> >>> 2) I'd love to see a "trending topics" list that does NOT include hash >>> tags, you know, to find out what ordinary people are talking about :-) >>> >>> I know this is the wrong place for it (sorry) but I'm not sure where else >>> to go. >>> >>> TjL >> >> >> >> -- >> Sincerely, >> >> Burhan Tanweer >> www.explorewww.com >> expl...@explorewww.com >> >> > > > > -- > Doug Williams > > do...@igudo.com > http://www.igudo.com >
[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
On Fri, Mar 6, 2009 at 11:11 AM, Doug Williams wrote: > In your experience, do trending bots have a disproportionate > participation in the search results for trending topics? Have you done > any analysis like that? I'm not Chad :-) but if you click on any of the Trending Topics and watch for any length of time you'll see scads of trending topic bots popping up. I think the most I counted at one point was like 12 out of the top 20 results. It's insane. TjL
[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
Chad, In your experience, do trending bots have a disproportionate participation in the search results for trending topics? Have you done any analysis like that? Doug Williams @dougw On Fri, Mar 6, 2009 at 10:43 AM, Burhan TANWEER wrote: > I agree with him. Search trends are not available in xml format. I will > appreciate, if twitter can provide search trends in xml and so that i can > update my social search engine ExploreWWW.com with search trends in real > time. > > Thanks > Burhan > > On Thu, Mar 5, 2009 at 11:09 PM, TjL wrote: >> >> Specifically >> >> 1) There are WAY to many "trending topic" bots which fill search >> results with useless clutter >> >> 2) I'd love to see a "trending topics" list that does NOT include hash >> tags, you know, to find out what ordinary people are talking about :-) >> >> I know this is the wrong place for it (sorry) but I'm not sure where else >> to go. >> >> TjL > > > > -- > Sincerely, > > Burhan Tanweer > www.explorewww.com > expl...@explorewww.com > > -- Doug Williams do...@igudo.com http://www.igudo.com
[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
I agree with him. Search trends are not available in xml format. I will appreciate, if twitter can provide search trends in xml and so that i can update my social search engine ExploreWWW.com with search trends in real time. Thanks Burhan On Thu, Mar 5, 2009 at 11:09 PM, TjL wrote: > > Specifically > > 1) There are WAY to many "trending topic" bots which fill search > results with useless clutter > > 2) I'd love to see a "trending topics" list that does NOT include hash > tags, you know, to find out what ordinary people are talking about :-) > > I know this is the wrong place for it (sorry) but I'm not sure where else > to go. > > TjL > -- Sincerely, Burhan Tanweer www.explorewww.com expl...@explorewww.com
[twitter-dev] Re: OT - where's the proper place to talk about search.twitter.com?
1) I agree, so I wrote a greasemonkey script to zap them from http://search.twitter.com/ results: http://jazzychad.com/stuff/?p=37 (disclosure: i also run one of the trending bots... i discuss this in the post) 2) http://twitscoop.com to the rescue! I have been told that they are hoping to create an API for their site (right now they have special mojo with tweetdeck integration). -Chad On Thu, Mar 5, 2009 at 11:09 PM, TjL wrote: > > Specifically > > 1) There are WAY to many "trending topic" bots which fill search > results with useless clutter > > 2) I'd love to see a "trending topics" list that does NOT include hash > tags, you know, to find out what ordinary people are talking about :-) > > I know this is the wrong place for it (sorry) but I'm not sure where else to > go. > > TjL >
