[twitter-dev] Re: how are the ten trends born?

[David Fisher]

It's pretty simple, but with a few twists.

First of all, remember that everything that Twitter does is done with
simplicity and efficiency in mind.

For the most part its just a frequency count of words over a short
time period, minus stop words, filtering out usernames (notice @foo is
never a trend) and URLs. How it combines "Wave OR Google Wave" I'm
unsure of, and then there's some basic spam filtering in there
additionally.

I was theorizing that some of it is based on accelleration of words
above their standard volume. Apple for example is always talked about
a great deal, but isn't always trending. Sometimes it has a greater
volume than other trending words, but it doesn't trend. Yet some
things stick around for a long time like IranElection. I need to dig
into this more. I almost had it fully modeled at one point, then lost
the code (damn you version control mistakes)

dave

On Oct 2, 10:54 am, Nigel Cannings 
wrote:
> @secretbear did it first in the halcyon days of the PubSub Firehose...
>  I'd ask him
>
> ==
>
> Why not encrypt the mail you send me?  You never know who's looking.
> If you use Firefox, why not use the FireGPG plugin to make it easy
> (http://getfiregpg.org)
>
> Get my key from:http://keyserver.pgp.com/
>
> ==
>
>
>
> On Thu, Oct 1, 2009 at 4:20 PM, Martin Dudek  
> wrote:
>
> > Good morning
>
> > wonder if somebody knows how twitter determines the ten trends it
> > declares every five minutes? Is this a pure word/phrase frequency
> > algorithm or some more complexity behind.
>
> > Thanks
>
> > martin

[twitter-dev] Re: Best way to implement caching of searches (with multithreading)?

[David Fisher]

Really a database is the way to go. Any modern database should allow
you to check if a value is in there before inserting, so the same
tweet won't go in there twice. Additionally, not every user search has
to use the up to the minute results. They can go back just a little in
time (30 seconds or so) to have batches in case there are multiple
people searching for the same thing.

On Oct 2, 7:36 am, Bjoern  wrote:
> Hi,
>
> just wondering about a best practice thing. Suppose I show results of
> specific Twitter searches on a web site. How would I go about caching
> the searches?
>
> The naive approach seems to be to first check in my own database, then
> do a twitter search with the since_id parameter to only get results I
> don't already have. Then store the results from twitter in the
> database, too, and return the merged results to the web site.
>
> The problem I see is that if multiple user run the same search on my
> web site, threading issues might occur (as each user starts a separate
> thread on my server). Not only could multiple twitter searches with
> the same since_id be executed (maybe forgivable), but trouble starts
> when said results are to be inserted in my local database. Different
> threads could attempt to insert the same messages into my database.
>
> One simple solution I could imagine: just use the message ids from
> twitter as the primary key in my local database. That way, multiple
> threads saving the same message would just overwrite the message with
> itself. I actually wonder if that is a common solution - to use the
> twitter ids as primary keys (also for users, direct messages...). I
> have kind of arrived at the opinion that this would be the way of
> least resistance, although I feel a bit uneasy about it.
>
> An alternative that came to my mind might be to have single threaded
> background jobs do the copying of the search results from twitter to
> my database, and only show the results from my cache to the web site.
> This would cause some lag in the time the search results would appear,
> but it would not be too bad. However, if I have a lot of different
> searches, it would become infeasible to update all of them
> periodically. It would become necessary to only trigger an update when
> a user does the search. At that point things might get overly
> complicated: presumably I would need some kind of Ajax solution to
> trigger the "caching" with the first request, show a spinner while the
> updating of my local db is going on, and then show the results from my
> local cache/db. The trickiest part being to prevent the starting of
> multiple update tasks for the same search.
>
> All in all the simple solution might be the better way to go?
>
> Would be interested in hearing your opinions, experiences and
> solutions!
>
> Thanks!
>
> Björn

[twitter-dev] Additional Favorites methods?

[David Fisher]

I'm hoping to do some research and build some neat tools around favorites in
Twitter (which while generally underutilized, when they are used they are
awesome and very useful).
Yet it seems the only way to get the knowledge that a tweet has been
favorited is via asking for a given user's favorites. There is no way (that
I'm aware of) to easily find all of the users that have favorited a tweet,
or get a timeline of recently favorited tweets. All of this is VERY api
intensive (and somewhat unneededly so).

If there was a way to ask Twitter about the favorites associated with a
single tweet even, that would be somewhat awesome and very useful for me.

Thoughts?

-David Fisher
Web Ecology Project
http://webecologyproject.org

[twitter-dev] Re: About the oneforty application directory

[David Fisher]

I'll vouch for Laura as being an upright person, and all of the other
oneforty people I've met are as well. They aren't going to try to
screw anyone over and I have faith that whatever they come up with
will be fair for developers and the community both.

david

[twitter-dev] Re: Search API Query Limitation (140 characters?)

[David Fisher]

If you need to search specific users why don't you use the Shadow API
and grab all of their tweets and then search them locally?

On Sep 28, 3:14 pm, Chad Etzel  wrote:
> Hello,
>
> The limit is indeed 140 and most likely won't be going up any time
> soon. The reason for the limit is for performance reasons. In order to
> do timely queries we don't allow for longer/arbitrary queries which
> could be very complex.
>
> -Chad
>
>
>
> On Mon, Sep 28, 2009 at 3:08 PM, zapnap  wrote:
>
> > (this could be overcome, I suppose, by performing multiple queries,
> > but that isn't much of a solution if you want to use the stock twitter
> > js search widget, etc)
>
> > On Sep 27, 11:37 am, zapnap  wrote:
> >> Search API queries appear to be limited to 140 characters. I mean,
> >> that's cute and all, but it's also rather limiting. In my particular
> >> case I was hoping to construct a search for a particular term from a
> >> group of N Twitter users only (instead of searching everyone).
>
> >> Is there another way to achieve this? Are there any plans to allow
> >> longer / more complex search queries?
>
> >> Thanks!
>
> >> ..nap

[twitter-dev] Re: master thesis related to Twitter

[David Fisher]

A small group of us (mainly Harvard students, but others as well) had
similar questions a few months back and we've started digging into the
research pretty heavily (also doing contracting/consulting). Our group
is called the Web Ecology Project and we've released a handful of
academic (but accessible and market relevant) papers about events and
influence on Twitter. We also released some source code for
identifying the language of Tweets using the Google API and python.

We have a research relations arm and I've forwarded your contact
information to our research coordinator Dharmishta who will probably
be in touch soon.

It might be worth looking over our papers. When you do, feel free to
email me (or any of us) and ask questions. We've done some semantic
analysis of tweets for our paper on the death of michael jackson and
the response on Twitter using the ANEW method/dataset and NLTK.

Our work can be found at http://webecologyproject.org and all of our
stuff is released under a Creative Commons license so feel free to
quote us and use bits where you need to.

One of the hardest initial things that we encountered with Twitter was
gathering and storing of useful and meaningful data but now we've
gotten mostly past those issues and we are now starting to mine and
analyze other social networks as well.

Thanks,

David Fisher
Web Ecology Project

On Sep 27, 8:11 pm, Stefna  wrote:
> Thank you all for the feedback.
>
> My main motivation for posting this thread was to gather some loose
> ideas. One man’s trash is another man’s treasure, posting a link costs
> nothing but saved a lot of mine time.
>
> "strict formed data" - 140 chars, #tag, @username, RT etc. - that's
> why there are so many sites presenting graphs, charts, trends,
> tendencies etc.
>
> "first pick was vaguely to analyse the semantic meaning of tweets" -
> like take a tag, take all the words with tweets with this tag and on
> that base find tweets that might me related to the topic. Or finding
> and measuring tags that are in tweets with aforementioned one.
>
> I should keep in mind that my task is to present a satisfactory
> dissertation, not cure for cancer. And probably by the time I finish
> my work someone else will ship similar software independently.
> Nevertheless by creating it I will learn a lot of stuff about Twitter
> itself not to mention improving programming skills.
>
> On 27 Wrz, 20:38, Mitchel Berberich  wrote:
>
> > Careful! Stefna was talking about semantic meaning. Not
> > "syntactic" ...
> > But I think you're right - Stefna, please us tell a bit more about the
> > context.
> > And - what do you think of when you say "strict formed data"?
> > What exactly do you want to achieve?
> > Maybe your promoter should tell you in more detail, what he expects
> > from you?
>
> > On Sep 27, 6:44 am, Nalin Savara  wrote:
>
> > > Good luck buddy.. Btw I'm curious, What exactly are you referring to
> > > or focussing on when you say 'syntactic meaning of tweets' ?
>
> > > I mean I'd appreciate a clarification on 'syntactic meaning to whom ?'
> > > and 'syntactic meaning of tweets in what context ?'
>
> > > Just wondering..
>
> > > Best Regards,
> > > Nalin
>
> > > On 9/25/09, Stefna  wrote:
>
> > > > I've submitted a ticket with following content:
> > > > *** *** ***
> > > > I am a 23 years old student of informatics at AGH Universtity of
> > > > Science and Technology in Cracow (Poland). Due to a rapid development,
> > > > strict formed data and accessible API I would like to designate my
> > > > master thesis to the Twitter related topic. My promoter is the PhD at
> > > > the Department of Computer Linguistics and our first pick was vaguely
> > > > to analyse the semantic meaning of tweets.
>
> > > > Do you have suggestions about the dissertation topic?
> > > > Do you have any pending requests or prospect features you want to
> > > > develop?
>
> > > > I will browse known issues, I will think thoroughly about the topic
> > > > but still - your suggestion might be very helpful. Even the shortest
> > > > one (like "good luck") will encourage me to more intensive research.
> > > > *** *** ***
> > > > Does anyone have any suggestions? My ticket has a six-digit number so
> > > > I'm afraid I won't get any answer :)
>
> > > > I'll probably ask for help during my work so I subscribe to this group
> > > > anyway.
>
> > > > Thanks in advance!
>
> > > --
> > > Sent from Gmail for mobile | mobile.google.com- Hide quoted text -
>
> > > - Show quoted text -

[twitter-dev] Re: SERIOUS Problem With Cursors In JSON Followers/Friends Ids

[David Fisher]

I'm sorry, but the problem isn't Twitter- its your language and JSON
parser. Outputting everything as a string, when it clearly should be a
number, is inefficient and crazy.

Saying that startups can't afford 64 bit processors in systems is
crazy. Most startups I know are running on EC2 or have fairly new
hardware. I bought a killer 64-bit quad xeon server for less than
$1,500 for our startup and its rocking. If your startup doesn't have
$1,500 for a primary capital computing expense that's another problem
you have there.

You either need to run on a newer system or use a language that can
properly handle 64-bit numbers. C, Python, Ruby, Scala, Erlang, C#,
etc none of them have problems with 64-bit ints.

Not a Twitter problem. It's a programming issue on your end and
unfortunately I can't help as I don't know PHP in depth.

-dave

On Sep 24, 3:29 pm, Dewald Pretorius  wrote:
> That magical maximum number appears to be 1 (1.0E+12).
>
> So, for tweet ids we still have a bit of breathing space.
>
> Dewald
>
> On Sep 24, 4:18 pm, Dewald Pretorius  wrote:
>
>
>
> > Clearly PHP_INT_MAX plays no role in json_decode.
>
> > There must be some other mystical maximum number above which it
> > represents the number as float in the decoded data.
>
> > Dewald

[twitter-dev] Re: Getting screen_name from id without gazillion API calls?

[David Fisher]

I've done this this way:

Every time I get a user's data- I store it in my database. Doesn't
matter if its from gardenhose or a REST method. I track various
versions of it and keep all changes (good data warehousing practice).

Then, when I crawl a user's list of friends I ask the database if we
already have the user (by id). If we've already got them, then I don't
waste an API call in asking for it again (as I only have so many). If
we don't have them, then I ask Twitter for it. If the next person I go
through has all the same friends, then I never make an additional REST
API requests.

I also do this when accessing the Search API, to make sure I have all
user profiles (at least in some stage). Unfortunately I have to do
this one by screen_name/from_user because the Search API returns
different user id numbers (Which I also store so I can match them if I
must).

I've got nearly 8M twitter user profiles this way so far and its
always growing.

-dave

On Sep 8, 12:41 pm, dizid  wrote:
> Owkaye,
>
> Well, Twitter would not need to change the system (with id's an
> screenname's) because there is a user advantage to being able to
> change your screenname.
>
> But, maybe the Twitter API could return both ID and screen_name with
> only 1 call to the API.
> Programmers can then choose to work with ID and/or screen_name and the
> API calls would still be limited. Only the data returned would be (a
> little) more...
>
> But, as it stands, there seems to be no other way to get screen_names
> (for friends and or followers) without doing 1 API call for each
> person (ID) on the returndata, which amounts to alot of API calls.
>
> On Sep 5, 5:09 pm, owkaye  wrote:
>
> > The ideal solution is for Twitter to "change the system" and
> > allow each account to have only one screen name, all the
> > time, forever, with no changes.  Then a separate "id" value
> > is not required because all account identification will be
> > done by the original screen name.
>
> > REST and SEARCH would finally be consistent.  No extra calls
> > to figure out who the user really is.  Users would complain
> > until they got used to the fact that they cannot change
> > their screen names on a whim anymore, but they will learn to
> > deal with it soon enough.
>
> > Email doesn't just let you change your address whenever you
> > feel like it, and I see no reason why Twitter should allow
> > screen name changes either ... except that it takes more
> > work to standardize the system in this way than to continue
> > with what already exists.
>
> > But with only the screen name as each unique account
> > identifier things would certainly be much simpler.  Many
> > fewer requests to the server.  Less data storage.  And being
> > that Twitter is supposed to be simple this seems like a goal
> > worth pursuing, at least from my point of view.
>
> > Owkaye
>
> > > >> When i request friends (or followers) from the Twitter
> > > >> API i want to get the screen_name's based on the id's.
>
> > > >> I use users/show for this, inputting the id and
> > > >> getting back de screen_name.
> > > >> This costs ALOT of API calls and i run into the API
> > > >> rate limit fast, especially with many friends.
>
> > > >> Is there a better way of getting screen_names for
> > > >> friends / followers?
> > > >> ( Better, meaning in fewer API calls.)

[twitter-dev] Re: Search API limits

[David Fisher]

The rpp defaults to 15 or something if you don't specify it. Sounds
like you need to mess around and play with things a bit more.

The key to max search results isn't in paging or rpp, but in max_id.

Be careful what you ask for. Retrieval of everything available can
take a long time (hours)

dave fisher
http://webecologyproject.org

On Sep 4, 10:16 am, owkaye  wrote:
> How can I retrieve the maximum number of tweets in a search?
>
> Can rpp be set to more than 100?
>
> What if I do not send a rpp value, does twitter default to
> returning more than 100 per page?
>
> Owkaye

[twitter-dev] Re: appended to API works in progress

[David Fisher]

lul. That's like asking the Duke Nukem Forever team when they'll
release.
"When its done"

While I love the twitter dev team, I'm not sure how hard they are able
to stick to deadlines in their development, so its probably just
better to say, "the future"

On Aug 24, 10:15 am, "METROmilwaukee ..." 
wrote:
> It’s great to see new API items are getting marked up. I would find it very
> helpful if [COMING SOON] was replaced by [MM-DD-] when the new item is
> officially released presuming why to do so is obvious marking the milestone.

[twitter-dev] Re: Twitter's "Tweet" Trademark Torpedoed

[David Fisher]

This is good news. No one will be getting sued over Tweet then.

Yet, keep in mind that Twitter probably *could * shut off access to
the API to any company they choose, as its their playground and their
rules. Not that they'll start doing that at all however.

On Aug 19, 1:57 pm, Sam Johnston  wrote:
> [refer to the article itself for the inline links - @samj]
>
> Twitter's "Tweet" Trademark 
> Torpedoedhttp://samj.net/2009/08/twitters-tweet-trademark-torpedoed.html
>
> Last month Twitter founder Biz Stone announced in a blog post (May The
> Tweets Be With You) that they "have applied to trademark Tweet because
> it is clearly attached to Twitter from a brand perspective".  This
> understandably caused widespread upset as the word "tweet" has been
> used generically by users for some time as well as in any number of
> product names by independent software vendors. Here's some samples
> from the resulting media storm:
>
>     * CNET News: Is Twitter freaking out over 'tweet' trademark?
>     * TechExpert: Twitter Trying to Trademark "Tweet"
>     * LA Times: Will Twitter trademark 'tweet' before it’s
> genericized?
>     * PC Magazine: Twitter Trying to Trademark 'Tweet'
>     * TechCrunch: Twitter Grows “Uncomfortable” With The Use Of The
> Word Tweet In Applications
>     * TechCrunch: Twitter To Developers: “Tweet” Your Heart Out, But
> Don’t “Twitter” It
>     * Bloomberg: Twitter Lays Claim to ‘Tweet’ Trademark in Bid to
> Protect Brand
>
> What they failed to mention though was that according to USPTO records
> (#77715815) not only had they actually applied some months before (on
> 16 April 2009) but that their application had been refused that very
> same day (1 July 2009).
>
> According to documents from the Trademark Document Retrieval system,
> their lawyers (Fenwick & West LLP) were notified of the rejection by
> email to tradema...@fenwick.com that day. The USPTO had explained that
> "marks in prior-filed pending applications may present a bar to
> registration of applicant’s mark. [...] If the marks in the referenced
> applications register, applicant’s mark may be refused registration
> under Trademark Act Section 2(d) because of a likelihood of confusion
> between the two marks", referencing and attaching not one, not two but
> three separate trademark applications:
>
>     * #77695071 for TWEETMARKS (pending receipt of Statement of Use)
>     * #77697186 for COTWEET (pending clarification)
>     * #77701645 for TWEETPHOTO (pending transfer to Supplemental
> Register)
>
> Now I may not be a lawyer (I did play a role in overturning Dell's
> "cloud computing" and Psion's "Netbook" trademarks) but given all
> three of the marks identified look like proceeding to registration (it
> only takes one to rain on their parade), it's my non-expert opinion
> that Twitter has a snowflake's chance in hell of securing a monopoly
> over the word "Tweet".
>
> That's too bad for Twitter but it's great news for the rest of the
> community as it's one less tool for locking in Twitter's rapidly
> growing microblogging monopoly. People do use the word "tweet"
> generically (including with non-Twitter services) and if Twitter, Inc.
> were successful in removing it from the public lexicon then we could
> all suffer in the long run.
>
> In any case it is neither serious nor safe for one company to become
> the "pulse of the planet" and that is why I will be following up with
> a series of posts as to how distributed social networking can be made
> a reality through open standards (if that stuff is of interest to you
> then subscribe and/or follow me for updates). I've also got some
> interesting things in the pipeline in relation to standards and
> trademarks in general so watch this space.
>
> Anyway it just goes to show that with trademarks you need to "use it
> or lose it". The "propagation delay" of the media has dropped from
> months at the outset to near real-time today so companies need to move
> fast to protect their marks or lose them forever. As for whether the 1
> July post was a scramble to protect the mark on receipt of the USPTO's
> denial, whether the USPTO was acting in response to it, or whether it
> was just a coincidence and particularly bad timing I don't know. I
> don't really care either as the result is the same, but I would like
> to believe that the USPTO is becoming more responsive to the needs of
> the community (after all, they revoked Dell's cloud computing
> trademark in the days following the uproar, despite having already
> issued a "Notice of Allowance" offering it to them).

[twitter-dev] Re: Do My Customers Have a Twitter Account?

[David Fisher]

Sounds like something you should be able to do in an email to them or
with a message on your website.

On Aug 19, 1:29 pm, arawajy  wrote:
> I want to invite them to follow the company on Twitter.
>
> On Aug 19, 8:25 pm, Andrew Badera  wrote:
>
> > On Wed, Aug 19, 2009 at 11:07 AM, arawajy wrote:
>
> > > Dear Developers,
> > > I have a list of 400,000 e-mail addresses of my clients. I want to
> > > know "Is it possible to develop a script to check if they have a
> > > twitter account or not?". I will then want to generate 2 separate
> > > lists based upon the result; one for the twitter users and one for the
> > > non-twitter users. I want to only invite the users and create a custom
> > > invitation message. Is it possible to check if the e-mail address's
> > > owner is a twitter user or not? provide details please.
> > > Thanks and Regards,
> > > Mahmoud
>
> > If they're already your "clients" then what are you inviting them to?
>
> > ∞ Andy Badera
> > ∞ This email is: [ ] bloggable [x] ask first [ ] private
> > ∞ Google me:http://www.google.com/search?q=(andrew+badera)+OR+(andy+badera)

[twitter-dev] Re: legal issues - is tweet an official verb in the US language?

[David Fisher]

Unless someone here is a lawyer, we should probably avoid legal
debate- consult with each our own counsels, and move on to doing what
we do best (coding).
I find these debates are often filled with FUD, misinformation,
speculation, a misunderstanding of law, etc

The easiest way to get around it is to not use Twitter based words in
your company/product name. Otherwise, just do what Biz said and use
Tweet. It seems he gave the thumbs up on Tweet.

dave fisher

On Aug 18, 5:27 pm, Bill Kocik  wrote:
> On Aug 17, 8:06 am, Nicole Simon  wrote:
>
> > Question: is to tweet an official word in the english language
> > both american and english? as in widely used?
>
> > does the US and UK trademark system reject such applications?
>
> Microsoft has a registered trademark on Windows. Apple Computer has a
> registered trademark on Apple.

[twitter-dev] Re: Cease & Desist from Twitter

[David Fisher]

How are some of you failing to see the difference between "Powered by
Twitter" being something they want you to do and "http://
TwitterApplication.com" is something they don't want you to do?

Why don't they want the latter? Because someone with the email of
"adultsexdatin...@googlemail.com" registered the domain. Not exactly
the type of company that Twitter wants to associate itself with. Yet,
for applications and sites that DO comply with the ToS, they want an
attribution and link back to their site. Aren't some of you self
proclaimed SEO/Marketing experts? Everyone wants links back to their
site, including Twitter.

Making a logo downloadable doesn't mean either that they want you to
use it, or their font on your website when doing your own branding.

Some people here are confused

dave

[twitter-dev] Re: Cease & Desist from Twitter

[David Fisher]

I am wondering if this is a case of their legal department getting a
bit heavyhanded and running loose.

What they asked of you seemed fairly reasonable however, and the name
of your application doesn't seem to be the issue.

I'm glad you didn't think you were being sued :)

It seems that Twitter should develop better UI guidelines for
applications using their API, since it seems that there are many
potential trademark concerns that their legal department is now
bringing up. Happens with every big company at some point I guess- and
Twitter is now getting 'big' and probably is just around the corner
from hiring their first full time attorney or something at this rate.

Hope it resolves well. Sounds like you're handling it well.

-dave

[twitter-dev] Re: FW: Twitter is Suing me!!!

[David Fisher]

Jeremy,

The problem with your logic is that you don't feel that a company can
set a ToS for how they want users to use their service. They can.

There are legitimate and non-legitimate uses of Twitter. This guy
screwed up, and overreacted. Case closed. Twitter's got him on the
naming issue and the ToS issue (which they can change any time they
like).

[twitter-dev] Re: FW: Twitter is Suing me!!!

[David Fisher]

@Vincent

No. Do you not understand that Trademark infringements occur between
things that could be mistaken for each other or in the same industry,
diluting a brand?

A Disney film from the 1940's has what to do with a 3rd part
application for a 2006-present social network?
There is a CLEAR connection between MyTwitterButler and Twitter.
MyTwitterButler is using Twitter as the basis for their business and
clearly in the same brand space.

This isn't uncomfortable for Twitter or MyTwitterButler.

Were you asleep for the whole Apple Computers v Apple Music thing? As
long as they weren't in similar spaces (computers weren't supposed to
make sounds/music initially) everything was ok, but then once Apple
got into music, the stuff got nuts and went to court. No Apple (fruit)
companies were sued however or sent notices.

You guys are so stupid this makes my head hurt.

david

On Aug 12, 3:18 pm, Vincent Wright 
wrote:
> None of us actually know how this might turn out but, even recognizing
> that this could become an uncomfortable matter for MyTwitterButler
> and/or Twitter, I nonetheless, decided to ask the question regarding
> the 1942 movie Bambi:
> "Would Twitter Sue Bambi For Being Twitterpated? :-) 
> ":http://mylinkingpowerforum.ning.com/group/twitterpated/forum/topics/w...
>
> Thanks, and Keep STRONG!!
> Vincent Wright
> Director Of Community
> MyLinkingPowerForum.ning.com |

[twitter-dev] Re: Following a large number of users

[David Fisher]

Streaming methods are your friend. Start with the follow method, and
email the API team about getting Shadow. If your use is legit (likely)
then they'll give you Shadow. If/when you outstrip Shadow they will
likely upgrade you then.

The only caveat to streaming is that if you miss some time (downtime,
etc) then you'll miss updates.

You can also use the REST api to grab updates from non-protected
users. Get your account whitelisted and you'll be able to pull 20K
queries per hour. Not a bad way to catch up if you miss some of the
streaming.

Sounds definitely possible.

dave
webecologyproject.org

On Aug 12, 10:10 am, murphy  wrote:
> We are building a suite of applications for Facebook Pages and one of
> the features is integration of user's twitter status updates into
> their facebook page feed. Thus the need to get a lot of users' twitter
> statuses at reasonable intervals.
> Looking at the API documentation there seems to be 3 ways to do that:
> 1. Use the streaming API. The 'follow' method looks good but allows
> only 200 users to be followed. How hard is it to get approved for the
> 'shadow' method?
> 2. Have a bot follow the users' accounts and get their updates with
> 'statuses/friends_timeline'. That allows up to 200 statuses to be
> retrieved every 30 seconds (to stay within the 150 calls per hour
> limit). This will probably do until we get a number of users ( I'd say
> more than 20 000). After that there will be updates cut off because of
> the 200 statuses per call limit.
> 3. Use the search API.
> e.g.http://search.twitter.com/search.json?q=from:user1OR
> from:user2...
> Now the query can be only 140 characters long and with the rate limit
> for search API calls, this approach will probably not work well (if at
> all).
>
> Any suggestions and comments how to approach this are appreciated.
>
> Slav Ivanov

[twitter-dev] Re: FW: Twitter is Suing me!!!

[David Fisher]

Let's not get into a semantic argument of spam.

Spam, noise, trash followers, whatever you want to call them. They are
all annoying and not what Twitter or most users want in their
service.

I'm unsure if a company HAS to pursue every trademark infringement to
hold their trademark. Otherwise you could in theory make a company go
bankrupt quickly just by setting up a bunch of shill companies using
similar names and force them to pursue each fully in court. Also there
is no obligation for Twitter to constantly search, identify and sue/
C&D/defend against every company worldwide that uses the name Twitter.
Just seems silly. IANAL however and could be wrong.

dave

[twitter-dev] Re: Trademark infringement

[David Fisher]

My guess is that Twitter will hold both Tweet and Twitter as IP, but
allow general use of Tweet. It seems to really make sense to defend
the name of their company always, and Tweet only when it is misused.

-dave

On Aug 12, 11:43 am, Dewald Pretorius  wrote:
> Nick,
>
> TweetLater.com has been using tweet as a verb since April 2008.
>
> Dewald
>
> On Aug 12, 12:21 pm, Nick Arnett  wrote:
>
> > In case anybody wants some decent facts on this issue, Wikipedia has a
> > pretty good article.
>
> >http://en.wikipedia.org/wiki/Trademark_infringement
>
> > I'm not a lawyer, but I published a book on IP for developers a number of
> > years ago and learned a lot in the process.
>
> > In the end, it is extraordinarily unlikely that anybody could possibly fail
> > to infringe when using "Twitter" and/or their logo in a product or service
> > name that is based on Twitter.  That inevitably could cause confusion about
> > whether or not it is from the company Twitter or not.
>
> > Infringement has nothing to do with whether or not the infringement is
> > connected with a "good" or "bad" use, such as spamming, etc.
>
> > It will be interesting to see what they do with "tweet."  Many attorneys
> > advise that if you want to preserve rights in a mark, you have to use it
> > only as a proper adjective.  Are they going to grant everybody permission to
> > use "tweet" but only that way?  I'll send you a Tweet(tm) message?  Seems
> > unlikely.  Tweet has become a noun and a verb, which I suspect means it is
> > fated to become genericized, if it hasn't already.
>
> > Nick

[twitter-dev] Re: FW: Twitter is Suing me!!!

[David Fisher]

The OP should have first gotten a lawyer to look over it, instead of
freaking out hysterically here.

He's not being sued. Twitter does own the name Twitter and can
selectively choose to sue/C&D (or not sue) anyone they like who
infringes on it. You defense being "other people have registered the
domain names too!" isn't any better one than you'll get with saying,
"But other people use Bittorrent too!"

Someone up in the thread said, "No one can now use the Twitter API"
because it says you can't use Twitter's name in your application, but
they want you to put "Powered by Twitter" on the site. That's just
FUD.

Come on guys, put on your big boy pants and lets show a bit of
business acumen and maturity here. Twitter isn't draconian, shutting
down the developer community, etc. That's just silly.

[twitter-dev] Re: Search API Limits lowered?

[David Fisher]

I don't have a domain to point back to. I'm doing data-mining and
analysis on a server that isn't public.
I have set the User-Agent to something unique (I thought you were
saying to change it for every request?).
Yet I'm still getting rate limited and told to back off a lot. Ryan S
said it might still have to do something with the DDoS attack, which
is still going on apparently.

dave

On Aug 11, 4:13 pm, Chad Etzel  wrote:
> The referrer is not as important as the user-agent. You can also put
> your URL in the user-agent instead.
> -Chad
>
> On Tue, Aug 11, 2009 at 4:09 PM, Larry Wright wrote:
> > In addition to setting a unique user-agent, I believe it was requested that
> > we set a referrer header that pointed back to a domain.
>
> > On Tue, Aug 11, 2009 at 9:30 AM, David Fisher  wrote:
>
> >> While i haven't done scientific testing of this, I was able to run up
> >> to 3-4 instances of my search script prior at a time before it told me
> >> to enhance my calm. Now I'm barely able to run one without hitting the
> >> limit. I can put delays in my code to slow it down, but I'm wondering
> >> if this is just a symptom of the aftermath of the DDoS attack or
> >> something else? My server has a dedicated IP and no one else runs code
> >> from it, so it isn't other people on my IP hitting the Search API.
>
> >> Maybe I need to talk about Search API whitelisting...
>
> >> dave
> >>http://webecologyproject.org
>
> > --
> > Larry Wright

[twitter-dev] Re: Following Churn: Specific guidance needed

[David Fisher]

Follower churn wouldn't exist, but getting hundreds of spam emails
(about being followed) would still exist.
I've got over 12,000 emails in my inbox about being followed on
Twitter. Dozens of those are from the same users. Some weeks the same
users unfollow and refollow me nonstop to try to get me to fall for
it. I don't. I've got only ~2000 followers because most of them were
just spammers churning through trying to follow me. Its annoying. Yes,
many users fall for it. Many big users auto-follow back because they
want the numbers (don't tell me everyone that Scoble followed back
initially was a real user).

I personally don't think you should be able to follow more than 25
people per day, but that's just me.

-david

On Aug 11, 1:55 pm, Kevin Mesiab  wrote:
> This entire debate focuses on the wrong side of the coin.
> Follow churn exists as a side effect of the improper Twitter culture of
> reciprocating follows blindly.
>
> If users paid due diligence to those they follow and only followed those
> people who demonstrate some value to them, follower churn would not exist.
>  Period.
>
>
>
> On Tue, Aug 11, 2009 at 7:51 AM, owkaye  wrote:
>
> > > > Would be very helpful to know the definition of "quick"
> > > > as relates to following churn suspensions.
>
> > > As Cameron pointed out earlier, as soon as they do that,
> > > the following churners will adjust their methods to be
> > > just inside that definition of OK.
>
> > This seems like a really short-sighted reason for NOT
> > clarifying what's acceptable and what's not.
>
> > If it's acceptable then who cares if the churners adjust
> > their methods?  At least everyone will know how to avoid
> > problems for a change, right?
>
> --
> Kevin Mesiab
> CEO, Mesiab Labs 
> L.L.C.http://twitter.com/kmesiabhttp://mesiablabs.comhttp://retweet.com

[twitter-dev] Re: Search API Limits lowered?

[David Fisher]

The user agent for each search request is the same. I'm using the Ruby
Twitter API wrapper, so sending anything else with search requests
isn't possible unless that is now deprecated.

dave

On Aug 11, 10:36 am, Andrew Badera  wrote:
> On Tue, Aug 11, 2009 at 10:30 AM, David Fisher wrote:
>
> > While i haven't done scientific testing of this, I was able to run up
> > to 3-4 instances of my search script prior at a time before it told me
> > to enhance my calm. Now I'm barely able to run one without hitting the
> > limit. I can put delays in my code to slow it down, but I'm wondering
> > if this is just a symptom of the aftermath of the DDoS attack or
> > something else? My server has a dedicated IP and no one else runs code
> > from it, so it isn't other people on my IP hitting the Search API.
>
> > Maybe I need to talk about Search API whitelisting...
>
> > dave
> >http://webecologyproject.org
>
> Are you sending a unique client ID header with each request, per
> previous Search API throttling conversations? (Not sure that it
> matters, that seemed pretty fuzzy when discussed ...)
>
> ∞ Andy Badera
> ∞ This email is: [ ] bloggable [x] ask first [ ] private
> ∞ Google me:http://www.google.com/search?q=(andrew+badera)+OR+(andy+badera)

[twitter-dev] Search API Limits lowered?

[David Fisher]

While i haven't done scientific testing of this, I was able to run up
to 3-4 instances of my search script prior at a time before it told me
to enhance my calm. Now I'm barely able to run one without hitting the
limit. I can put delays in my code to slow it down, but I'm wondering
if this is just a symptom of the aftermath of the DDoS attack or
something else? My server has a dedicated IP and no one else runs code
from it, so it isn't other people on my IP hitting the Search API.

Maybe I need to talk about Search API whitelisting...


dave
http://webecologyproject.org

[twitter-dev] Re: Following Churn: Specific guidance needed

[David Fisher]

IDOLPeeps,

I feel you're being overly alarmist and haven't painted the situation
properly.

You can unfollow anyone you want. The issue is a quick follow and then
unfollowing if not reciprocated. You're *supposed* to follow someone
because you want to hear what they are saying, not because you want
them to hear what you say. People were churning in attempts to gather
mass numbers of followers. A lot of the MLM types did a good job at
this and soon had 20, 40, or even 60K followers. Now they are clamping
down.

Comparing the situation/rules to communist regimes or dictatorships is
a straw man argument and uncalled for. Surely given another post or
two and this would be a comparison to Hitler, fulfilling Godwin's law
of the internetz.

I'm speculating, but I have to wonder more about your friends that
were banned. Did they do exactly as Scoble (were they active members
for over two years and offering massive community contribution?)? Were
they following people prior to unfollowing? I know Scoble isn't going
on massive following kicks. Scoble's intentions were also made very
clear. Did your friends make their intentions clear? Had they been
following and unfollowing people a lot?

The other day I unfollowed about 30% of the people I was following. No
ban. They aren't playing favorites. They are looking at multiple
factors. If you're acting like a spammer, then you'll be treated like
one.

Churning and spam following is being cracked down on. Instead, grow
slowly, naturally and contribute to the community.

-David
http://WebEcologyProject.org


On Aug 11, 8:34 am, Dewald Pretorius  wrote:
> I follow a simple principle in TweetLater.
>
> Where Twitter rules are clearly spelled out, such as for spam, I give
> the users a hammer and caution them, "Carefully read the rules because
> you can drive a nail into wood, but you can also smash your thumb to a
> pulp with this thing."
>
> Where Twitter rules are nebulous, such as for unfollow, I do not give
> them a tool, because they wouldn't know (and neither would I) how to
> use it properly and safely.
>
> Dewald
>
> On Aug 11, 8:42 am, Dewald Pretorius  wrote:
>
> > On Aug 11, 3:11 am, TFT Media  wrote:
>
> > > For its auto-follow, tweetlater.com specifically states: "[w]e have
> > > limits in place to ensure that your daily following remains well
> > > within the limits imposed by Twitter."  So you are presumably touching
> > > the rate limit then going back -1, -2, -3, or whatever.
>
> > You pulled that snippet from my feature that finds potential new
> > friends based on user-selected keywords. That snippet simply means
> > that the user can pause that feature when her account reaches the
> > point where she would not be able to follow more people anyway. And
> > no, my system absolutely does NOT then unfollow to "make room for
> > more".
>
> > Here is what I very publicly said about bulk unfollow:
>
> >http://bit.ly/JM3as
>
> > Dewald

[twitter-dev] Re: 404 for statuses/user_timeline of non-existing users?

[David Fisher]

Yep. In my experience calling a user that doesn't exist does yield a
404. You'll have to handle the errors on this.
I guess 404 does make sense here, as the user is not found.

dave

On Aug 11, 5:40 am, Carlo Zottmann  wrote:
> Hi there,
>
> I'm doing statuses/user_timeline requests via JS, and sometimes due to
> faulty user input, I query statuses/user_timeline for nonexisting
> users. The statuses/user_timeline API endpoint returns correct data
> (for example, `callbackFunction({"request":"\/statuses\/user_timeline\/
> nonexistingexample.json?count=200&callback=
> callbackFunction","error":"Not found"})`), but unfortunately it sends
> a 404 status code as well.
>
> Neither Firefox nor Safari consider these documents as valid and
> refuse to execute the code within.
>
> Is this the correct behaviour or an API bug?
>
> Thanks,
> Carlohttp://TwerpScan.com

[twitter-dev] Re: NO API CONNECTION, BUT STILL FULLY BLOCKED

[David Fisher]

Yea, and we all threatened to go to Pownce.
That didn't go so well (seeing that Pownce is now dead)

dave

On Aug 9, 3:05 pm, Dossy Shiobara  wrote:
> On 8/9/09 1:53 PM, David Fisher wrote:
>
> > Anyone that was around in 2006/2007 knows that Twitter was excessively
> > unstable then, and people didn't whine/complain as much.
>
> Sure we did, there were just fewer of us at the time.  Weren't you
> around in 2006/2007?  Perhaps you have a short memory retention.
>
> --
> Dossy Shiobara              | do...@panoptic.com |http://dossy.org/
> Panoptic Computer Network   |http://panoptic.com/
>    "He realized the fastest way to change is to laugh at your own
>      folly -- then you can let go and quickly move on." (p. 70)

[twitter-dev] Re: OK Seriously People

[David Fisher]

> It is like a surreal tech soap opera on this list though :-) Hey maybe I
> could pitch this to Fox .
>
> All the best
> Neil
>

DO IT! Maybe Twitter would have made a good reality show. Lots of
super-dramatic angles and queues pointing at sysadmins typing at a
console. :)

The mailing list would be like the random room they walk into to talk
to the camera.

dave

[twitter-dev] Re: Can Twitter team Respond?

[David Fisher]

It will be fixed when it is. Stop complaining. We're all in the same
boat. The ETA is "when its done".

dave

On Aug 9, 1:08 pm, freefall  wrote:
> Decentralisation.
>
> On Aug 9, 5:47 pm, Dewald Pretorius  wrote:
>
> > I wonder how many times this weekend has Chad heard, "FO, we're busy,"
> > when he tried to get a status update for us.
>
> > On Aug 9, 1:42 pm, Jesse Stay  wrote:
>
> > > Good luck.
>
> > > On Sun, Aug 9, 2009 at 6:01 AM, Jacob  wrote:
>
> > > > I have thousands of people coming to my application only to get
> > > > "Twitter Api is not responding!" can Twitter team help me?
> > > > I'm aware of the API difficulties, but, I need some answers or maybe a
> > > > date when the problem will be fixed.
>
> > > > Thanks in advance
> > > > Jacob
> > > >http://twitteranalyzer.com-Hide quoted text -
>
> > - Show quoted text -

[twitter-dev] Re: OK Seriously People

[David Fisher]

A few of you are acting like real children and a few of you still have
your heads screwed on right.

I'm confident they are doing everything they can. Chill and enjoy your
weekend. They'll get it sorted out.

What did you guys do in 2007? Twitter was down all the time then. Your
blood pressure must have been through the roof with weekly visits to a
shrink if you responded this way every time it went down.

dave

On Aug 9, 1:48 pm, Neil Ellis  wrote:
> Nice story Adam, however the band are actually trying to run a  
> business, not doing this for love/free. I can assure you the investors  
> in Twitter will be looking to turn profit. Of course if the band are  
> laid up then the danger is the hotdog man (and all his customers) will  
> go to another band that are still playing and have fans.
>
> That's why I'm 100% confident all that can be done is being done, cos  
> plenty of people at Twitter will know how fickle a user base can be.
>
> Good luck guys, I know what these situations are like and it's hard on  
> you all - I actually hope you guys are getting some rest because it  
> doesn't sound like this is a 100 yard sprint.
>
> I also hope someone is making sure the ops/devs aren't reading this  
> list (or getting emails etc) - stress doesn't help productivity in my  
> experience. Knowing what is at stake does.
>
> Again good luck chaps, I know how the trenches feel :-)
>
> And of course it does suck for the rest of us too, alas that is  
> business.
>
> ATB
> Neil
>
> On 9 Aug 2009, at 18:24, Adam Cloud wrote:
>
> > ***Scenario***
>
> > A band broadcasts their music on a radio station all the time, and  
> > people are able to freely tune into it, or go buy their music. They  
> > go and play in a city park for free every day just because it's a  
> > much nicer experience for the listener then to be just sitting at  
> > home listening on their radio.
>
> > You as an up and coming entrepreneur go buy a hotdog & drink stand  
> > and setup camp in that park to make some cash off of the flow of  
> > people who come to see this free event every day. You being there,  
> > giving the ability for people to eat & drink without leaving the  
> > park allows for more of this bands songs to be heard, in effect  
> > increasing the chance that their music might be purchased. So you're  
> > essentially helping them, by taking advantage of them for your  
> > business.
>
> > The band gets in a car crash, and alot of equipment is damaged to  
> > the point of not being able to be used, along with their main source  
> > of transportation. The band starts working to find and replace all  
> > that is damaged in their equipment and for their car.
>
> > Now you can imagine that little hotdog stand guy standing on their  
> > doorstep while they recover yelling profanities and how they should  
> > be skipping the shipping company who's delivering their parts and  
> > get their parts themselves to save time. Yelling that they shouldn't  
> > be sleeping, they should be working on their band van right now to  
> > make sure it can take them back to the park so he can make some  
> > money. "People aren't coming to my stand anymore!!! They're going to  
> > fast food restaurants and going home. WTF i sold my wife for this  
> > stand!!!"
>
> > Now of course, this little hotdog stand man may not have really sold  
> > his wife, depending upon which one of you people who are still up in  
> > arms about this was put in his place, but i think you get my points.
>
> > The band could easily move to a venue that has their own hotdog/
> > drink stand making your services not necessary, but instead of doing  
> > that and capitalizing on the profit they could get from that,  
> > they're still planning on going back to the same park they do their  
> > free shows at, and allowing you to continue earning your money.
>
> > And this concludes storytime. :)
>
> > Happy sunday! (Relax!)
>
> > On Sun, Aug 9, 2009 at 10:58 AM, Terry Jones  wrote:
>
> > > "Stuart" == Stuart   writes:
> > Stuart> * I can't believe you lot don't realise that constantly  
> > demanding
> > Stuart> status updates, while certainly important to you, is little  
> > more
> > Stuart> than a distraction for those who are actually fighting the  
> > good
> > Stuart> fight.
>
> > I woke up this morning with the thought that the Twitter mailing  
> > list has
> > now become part of the DDoS.
>
> > What percentage of the people complaining loudly and increasing the  
> > general
> > stress/pressure level are actually bots?  :-)
>
> > Terry

[twitter-dev] Re: The silence is deafening....

[David Fisher]

I'm disgusted by you guys. Were any of you around in 2007 when Twitter
was down nonstop? Did you bitch and moan as much then? You're acting
like children.

The reason they are still facing issues is because of the DDoS. Oh,
and they are still also coping with scaling overall and astronomical
growth. A DDoS that is shifting and trying to thwart them isn't easy
to stand up to- moreso if you were just able to handle the load of
your standard business traffic anyway.

Its very clear that many of you have NEVER worked in a startup or
technology environment. Its no like everyone from Biz down to the
receptionist can focus all their magical energy on Twitter at the same
time to make it work.

I've worked for a startup before when our service/site has gone down.
There were one, maybe two people in the company who had the ability,
authorization or knowledge to do it. The rest of us were as frustrated
as hell, but even if one of us was the cause (a bad query to the
database or something) that didn't mean that we could all instantly
fix it. Companies have different roles for people and it doesn't help
at all for EVERYONE to SSH to the same box and try to muck with the
settings to fix things.

All hands are on deck that need to be. Everyone else at Twitter can't
do anything about it, but is stressing out too.

Chad not working today isn't a sign of them not working on the stuff.
He just joined the team, and isn't in the engineering/operations
department. Him flying out to California to stand and stare at the
others working on the issue won't help. Different people have
different roles and his role isn't to fix this stuff.

If something happens at your office, does everyone from the CEO to the
receptionist jump on the issue all at once? No, they can't. It doesn't
make sense. Stop bitching and complaining. You're making yourselves
look unprofessional and bad.

-David Fisher
Web Ecology Project
http://webecologyproject.org

On Aug 9, 2:51 am, chinaski007  wrote:
> And, by the way, if you're a deckhand on a submarine going down, you
> think you would go to a movie because it's your time off, or do
> whatever you can to help out?
>
> On Aug 8, 11:47 pm, chinaski007  wrote:
>
> > Part of it is DDOS response, part of it is API issues... as one of the
> > updates most recently noted with the "known issues" list.
>
> > On Aug 8, 11:42 pm, Bill Kocik  wrote:
>
> > > On Aug 9, 2:28 am, chinaski007  wrote:
>
> > > > You're wrong.
>
> > > > If you check the tweets of the other main Twitter developers, you will
> > > > see that they are doing sushi, rock concerts, weddings, watching
> > > > movies on Saturday afternoon, etc.  And while "getting married" is
> > > > certainly a legitimate excuse, some of the other activities, during
> > > > this major upheaval, are certainly questionable!
>
> > > Oh...I get it, you think that the developers are the right people to
> > > handle a DDoS attack. I'm afraid I have some bad news for you: you're
> > > wrong.
>
> > > And because not *all* the developers are sweating it out at Twitter
> > > HQ, that means no one's working on it. I can't argue with that logic.
> > > Seriously, I can't.

[twitter-dev] Re: NO API CONNECTION, BUT STILL FULLY BLOCKED

[David Fisher]

They haven't overreacted. If you think you can do better, then apply
for a job with them. Have you tried your account from multiple IP
blocks?

They have faced a crushing attack. They are working on it and they
have worked hard to communicate with us the status of things.

Anyone that was around in 2006/2007 knows that Twitter was excessively
unstable then, and people didn't whine/complain as much.  They've
gotten better but they don't have the capacity to serve traffic and
handle a DDoS attack at the same time.

Your ticket will be dealt with and they are working to resolve overall
issues before they address specific people.

dave fisher/@tibbon
webecologyproject.org

On Aug 9, 10:30 am, chuckblakeman  wrote:
> For what it's worth.  I am a Twitter use who does not use TweetDeck,
> TweetLater, or any other 3rd party software doing API calls to
> Twitter.  I use Twitter directly without any connection to other API
> calls.
>
> Since Wednesday I have not been able to use any Twitter functions and
> as of Friday evening I was no longer even able to sign in and watch
> the Tweet stream helplessly.  Login function has been disabled since
> then.
>
> So they are not just blocking majore APIs - for some reason my direct
> Twitter account has been blocked by Twitter as well.  I've sent two
> tickets and one email to support over the last three days with no
> response.
>
> They have clearly over-reacted to this issue.

[twitter-dev] Re: "You have been rate limited. Enhance your calm."

[David Fisher]

Yea I'm using John's. Have to pull out the packet sniffer soon

On Aug 7, 3:32 pm, Larry Wright  wrote:
> You can use Wireshark or any other packet sniffer to determine whether your
> client is following redirects. I'm not sure what ruby twitter client you're
> using, but if it's John Nunemaker's, I believe it does follow redirects.
> Larry Wright/@larrywright
>
>
>
> On Fri, Aug 7, 2009 at 2:03 PM, David Fisher  wrote:
>
> > I can't be sure if my client is following redirects. Probably not. I'm
> > just using the Ruby Twitter Gem which haven't been updated for a month
> > or so I think
>
> > dave
>
> > On Aug 7, 1:15 pm, lucasnicolato  wrote:
> > > im having the same problem. im just lucky my app is still in test.
>
> > > RT @twitter Due to defense measures some Twitter clients are unable to
> > > communicate with our API, and many users are unable to tweet via SMS.
>
> > > I think we can only wait for twitter to normalize de api.
>
> > > On 7 ago, 12:20, diddy  wrote:
>
> > > > Hi,
>
> > > > I use the Twitter search api, e.g: -
>
> > > >http://search.twitter.com/search.rss?q=iphone
>
> > > > and I now get: -
>
> > > > "You have been rate limited. Enhance your calm."
>
> > > > I rely on this for my application.  Anything I can do to stop it?
>
> > > > Thanks!
>
> --
> Larry Wright

[twitter-dev] Re: "You have been rate limited. Enhance your calm."

[David Fisher]

I can't be sure if my client is following redirects. Probably not. I'm
just using the Ruby Twitter Gem which haven't been updated for a month
or so I think

dave

On Aug 7, 1:15 pm, lucasnicolato  wrote:
> im having the same problem. im just lucky my app is still in test.
>
> RT @twitter Due to defense measures some Twitter clients are unable to
> communicate with our API, and many users are unable to tweet via SMS.
>
> I think we can only wait for twitter to normalize de api.
>
> On 7 ago, 12:20, diddy  wrote:
>
> > Hi,
>
> > I use the Twitter search api, e.g: -
>
> >http://search.twitter.com/search.rss?q=iphone
>
> > and I now get: -
>
> > "You have been rate limited. Enhance your calm."
>
> > I rely on this for my application.  Anything I can do to stop it?
>
> > Thanks!

[twitter-dev] Re: "You have been rate limited. Enhance your calm."

[David Fisher]

If you're getting that then you're getting more than me.

I'm just doing:

require 'rubygems'
gem 'twitter'
require 'twitter'

Twitter::Search.new('foo').each do |r|
  puts r.inspect
end

And I get only this back now:

/var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty/request.rb:56:in
`setup_raw_request': undefined method `request_uri' for # (NoMethodError)
from /var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty/request.rb:
39:in `perform'
from /var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty/request.rb:
99:in `handle_response'
from /var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty/request.rb:
40:in `perform'
from /var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty.rb:156:in
`perform_request'
from /var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty.rb:122:in
`get'
from /var/lib/gems/1.8/gems/twitter-0.6.13/lib/twitter/search.rb:
100:in `fetch'
from /var/lib/gems/1.8/gems/twitter-0.6.13/lib/twitter/search.rb:
108:in `each'
from searchtest.rb:5

My search is totally borked. Its just the API though I think, give it
some time


On Aug 7, 11:20 am, diddy  wrote:
> Hi,
>
> I use the Twitter search api, e.g: -
>
> http://search.twitter.com/search.rss?q=iphone
>
> and I now get: -
>
> "You have been rate limited. Enhance your calm."
>
> I rely on this for my application.  Anything I can do to stop it?
>
> Thanks!

[twitter-dev] Re: Current Twitter site status

[David Fisher]

Is the Search API being effected? I thought at first that I had messed
up my code, but I rolled back pretty far and I'm still getting really
odd errors

/var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty/request.rb:56:in
`setup_raw_request': undefined method `request_uri' for # (NoMethodError)
from /var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty/request.rb:
39:in `perform'
from /var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty/request.rb:
99:in `handle_response'
from /var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty/request.rb:
40:in `perform'
from /var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty.rb:156:in
`perform_request'
from /var/lib/gems/1.8/gems/httparty-0.4.3/lib/httparty.rb:122:in
`get'
from /var/lib/gems/1.8/gems/twitter-0.6.13/lib/twitter/search.rb:
100:in `fetch'
from /var/lib/gems/1.8/gems/twitter-0.6.13/lib/twitter/search.rb:
108:in `each'


On Aug 6, 11:45 am, Stuart  wrote:
> 2009/8/6 Chad Etzel :
>
>
>
> > Some of you may already be aware that the main Twitter site is under a
> > DDoS attack.  Please keep a close eye onhttp://status.twitter.com/
> > and this list for details and updates.
>
> "Encountered seemingly neverending redirects" - that can't be helping!!
>
> http://titsup.net/http://twitter.com/
>
> -Stuart
>
> --http://stut.net/

[twitter-dev] Re: API only shows messages from last 7 days

[David Fisher]

I don't think that adding more people to the staff at Twitter is the
solution. In one startup I saw a thing posted on the refrigerator that
had the adage, "Adding more people to a project already behind
schedule will only slow it down more". Surely for support and customer
service issues having more people on the team to deal with growth is
good, but I doubt throwing more programmers at it will help fix most
issues. It just never seems to work that way.

While many startups do tend toward younger employees (I personally
think because being younger normally means that you can work a lot
with minimal life impact), I'm sure that someone with a strong
background would be able to get a job at Twitter if they were local to
the company (or willing to move).

A lot of this surely comes down to priorities inside the company.
While Doug and Team want to support us developers as much as possible,
much of our initial 'value' that we've offered in helping push twitter
to the masses has already happened. We aren't the core business
strategy, and with a fixed amount of resources and focus they aren't
working to push mainly for developer access, but for standard user
access. This 100% makes sense. Users are what is going to make twitter
happen, not 3rd party developers. They want to provide a stable
experience on both fronts, but users come first.

In my private discussions with some team members, I've gotten the
sense that they have good stuff in the pipeline for us and that they
are working hard to make it happen. However we're only a small part of
the overall strategy of a quickly growing company that is still
dealing with massive growing pains which is no fault of theirs and
something they are dealing with as best they can.

david

On Jul 28, 1:46 pm, owkaye  wrote:
> I'm sure others feel the same way Dave, but it looks and
> feels like Twitter is moving in the opposite direction.  
>
> The load on a server to extract a big dataset once a month
> would be minimal, and both you and I can see the value in
> this approach. But I'm not sure the folks at Twitter do, or
> if they do maybe they just don't have the people who can
> (and will) get things like this implemented.  Is a shortage
> of competent staff the cause of this type of problem?
>
> Even though I have the capabilities I do not have the
> 'resume' to get a job there and help them deal with some of
> this stuff, nor do I have the contacts within the Twitter
> organization to put a good word in for me and help me get
> hired so I could do good things for them.
>
> I'm 52 years old too, and my age seems to be a negative to
> most of the Web 2.x companies hiring these days.  This is
> kind of a shame considering that people like me frequently
> have broader-based experience and insights that are
> sometimes lacking in younger people, and because of this we
> can add a lot more value in the areas of planning and
> structural development than people half our age.  Our coding
> skills are honed after so many years of experience too, not
> to mention the thousands of code snippets we have collected
> over the years to contribute to making us even faster.
>
> But since jobs like this are basically not open to me and
> many other folks my age, my alternative is to remain self-
> employed and try to build something on top of their existing
> available source data and API's ... and then deal with the
> issues and frustrations created when building a service on
> top of a 'moving target' that sometimes seems to be moving
> in funny directions.
>
> I hear about Twitter having lots of money to work with, and
> I'm probably wrong here but it almost seems like there's too
> little of this money being dedicated to paying new talent
> with long term views of some of these issues, and who will
> implement wise policies to help support and encourage rapid
> growth in the areas that are lacking.  But once again this
> might just be due to a shortage of the right staff.
>
> Obviously we cannot do anything from the outside except
> point out these issues and ask questions, or beg and plead
> for changes, but it sure would be great if a few of us could
> actually get in there as employees and implement a couple of
> the new features we really need -- such as a new Historical
> Search API for example.  Then developers like you and I
> could proceed with some of our plans now, instead of months
> or years from now ... or possibly never.  I would love to
> lead a team on a project like this, or even be one of its
> members, but until it happens I'll focus on building my own
> little space in the Twitter universe and continue to hope
> for the best.
>
> :)
>
> Owkaye
>
>
>
> > I would do anything (including paying good amounts of
> > money) to be able to purchase access to older datasets
> > that I could transfer to my database through non-rest-api
> > methods. I'm envisioning being able to download a CSV or
> > SQL file that I could merge with my database easily, but
> > only have to ma

[twitter-dev] Re: API only shows messages from last 7 days

[David Fisher]

I would do anything (including paying good amounts of money) to be
able to purchase access to older datasets that I could transfer to my
database through non-rest-api methods. I'm envisioning being able to
download a CSV or SQL file that I could merge with my database easily,
but only have to make a single request to the server to get a month of
data. I'd sign agreements and pay money for such.

dave

On Jul 28, 12:03 pm, owkaye  wrote:
> I agree with you Dave.  I have several thought about new
> services based on searching Twitter's historical data.  
> Unfortunately my ideas appear to be getting less and less
> practical.
>
> Twitter claims to have all its data stored in disk-based
> databases from what I understand ... yet without access to
> this data it is worthless.  
>
> It seems to me they could allow searches of this historical
> data via a new "History API" then let us cache the results
> on our own servers.  Most of the services I've conceived
> would do this infrequently -- never in real time -- and
> would not impact their existing "cached" server data because
> this historical data would exist on separate data storage
> servers ... theoretically anyways.
>
> Owkaye
>
>
>
> > I am a bit concerned. I remember at one point it being
> > between 30-45 days. Now it seems to be getting smaller by
> > about 1-day per month. Last month it was closer to 10
> > days.
>
> > Is it basically going to keep getting smaller and smaller
> > until we get V2 of the API, or will we be forced to all
> > use only streaming services and then locally cache
> > everything that we'd want to search for any time period?
>
> > I know there are a LOT of problems inherent in the
> > massive scaling out of Twitter, and this is just a
> > symptom of them- but at the same time I can only imagine
> > how unusable Google would be if you only had a 7-day
> > window to Search in, and couldn't get any content made
> > prior to that. Very worried about this soon being a 2-3
> > day window.
>
> > dave

[twitter-dev] Re: API only shows messages from last 7 days

[David Fisher]

I am a bit concerned. I remember at one point it being between 30-45
days. Now it seems to be getting smaller by about 1-day per month.
Last month it was closer to 10 days.

Is it basically going to keep getting smaller and smaller until we get
V2 of the API, or will we be forced to all use only streaming services
and then locally cache everything that we'd want to search for any
time period?

I know there are a LOT of problems inherent in the massive scaling out
of Twitter, and this is just a symptom of them- but at the same time I
can only imagine how unusable Google would be if you only had a 7-day
window to Search in, and couldn't get any content made prior to that.
Very worried about this soon being a 2-3 day window.

dave

On Jul 26, 4:11 am, Flashing Moose  wrote:
> Hmm, then i can't use the API for this project, thx for replies guys.
>
> On 26 jul, 04:10, Dewald Pretorius  wrote:
>
>
>
> > I believe the tweet retention in Twitter Search has always been 7
> > days.
>
> > On Jul 25, 1:18 pm, Flashing Moose  wrote:
>
> > > Hello, having some trouble with the API because only the messages from
> > > the last 7 days show up:
>
> > > example:
> > > feed://search.twitter.com/search.atom?q=from%3AstimulusHome
>
> > > Yes, there must be older posts in there... but how do i get to them?
>
> > > I read about the Operator Limits:
> > > filter:links operator:
> > > results are limited to 7 days
> > > source: operator:
> > > results are limited to 7 days
> > > queries must contain at least one keyword parameter with the source:
> > > operator
> > > lang= operator:
> > > results are limited to 7 days
> > > location operator:
> > > results are limited to 7 days
>
> > > but i'm not using filter, source, language or location do i?
>
> > > regards, Moose

[twitter-dev] Re: New Twist To Follow Terms Violations

[David Fisher]

"What gives Twitter the right to dictate who you want to follow or
not?"

Its their service. They can dictate what they want. Their playground,
their rules. The ToS clearly says they can alter their terms at any
time and if you don't want to comply you can leave.

That being said, this is to prevent people who are massively following
and unfollowing (as I see happen dozens of times daily on my account).
You aren't going to get banned for not following someone back, or for
unfollowing someone that bothers you.

dave

On Jul 24, 2:15 pm, Joshua Perry  wrote:
> Think about a bot who just bulk follows random people, it then would
> kept track of users who didn't blindly or automatically follow back and
> dump them quickly and try following another batch of users so that it
> wouldn't bust it's follow ratio limit. Using this strategy a bot could
> eventually build a very large following/followers list for someone while
> still keeping it's ratio within the boundaries set.
>
> I believe that the second part of that term is to protect against this
> scenario.
>
>
>
> Vision Jinx wrote:
> > What?
>
> > Re: "as well as following and unfollowing those who don't follow back,
> > are both violations of our terms of service."
>
> > What gives Twitter the right to dictate who you want to follow or not?
> > That is like Gmail saying you can't remove contacts from your contacts
> > list. When I signed up it suggested a list of people to follow but I
> > didn't find the tweets interesting so I un-followed them (they didn't
> > follow me back, but that was not the reason I un-followed them). I
> > should have the right to decide who I want to follow or not unless
> > Twitter is under a communist regime? Is there also a term that if
> > someone posts a link I have to click it also?
>
> > I also followed iGoogle for a while but didn't find the tweets that
> > interesting so I un-followed them, they never followed me back, so if
> > Twitter wants to delete my account (for TOS violations) then fine go a
> > head, do so right now then, but I feel it is my right to decide who I
> > do and do not want to follow and that will not change. They need to
> > post a message when you sign up that you are not allowed to un-follow
> > people. Why is there even that option then?
>
> > Regards,
> > Vision Jinx
> > @visionjinx
> > (In case Twitter wants to delete my account for feeling I have the
> > right to decide who I follow, fine then do it now.) I also, un-
> > followed someone because they kept posting the same tweets over again
> > so who's the bigger offender there then?
>
> > On Jul 24, 10:22 am, Dewald Pretorius  wrote:
>
> >> On Twitter's new site,http://business.twitter.com, under the heading
> >> Best Practices, the following is listed as a spamming practice:
>
> >> "Following churn: Following and unfollowing the same people
> >> repeatedly, as well as following and unfollowing those who don't
> >> follow back, are both violations of our terms of service."
>
> >> Take note devs, the "...unfollowing those who don't follow back..."
> >> statement is posing a risk for any of your apps that do bulk unfollow.
>
> >> On that point, I would like to get clear guidance from Twitter whether
> >> unfollowing someone who has stopped following you, i.e., unfollowed
> >> you first, would also constitute a violation of Twitter terms.

[twitter-dev] LOLCODE API Wrapper

[David Fisher]

I can't seem to find a LOLCODE (http://en.wikipedia.org/wiki/LOLCODE)
API wrapper for Twitter. I was hoping to write my next Twitter
application in LOLCODE, but its very hard to do so without one.

Can anyone help me get started with a basic program? My code so far
isn't working

HAI
CAN HAS STDIO?
MAKE CONNECT TWITTER NAO
I HAS A VAR TWEET
IM IN YR LOOP
   UP VAR!!1
   GET UR TWEETS
   VISIBLE TWEETS
IM OUTTA YR LOOP
KTHXBYE

Thoughts? An API wrapper would make all this much easier.

David

[twitter-dev] Re: Twitter is not making money

[David Fisher]

Show me these killer companies doing great NLP with social networks. I
find the ones that are doing stuff right now themselves are far behind
the curve and not really pushing stuff to the edge. They are often
marketing companies that have hired one NLP guy (and underpaid them)
and are just pushing the marketing side. I have yet to see anything
truly revolutionary come from most of these monitoring companies yet
and they are all too narrow focused. Plus, none of them have the VC
funding to really expand and grow (and not many people are getting new
funding these days)

-David

On Jul 18, 3:29 am, "M. Edward (Ed) Borasky"  wrote:
> On Jul 16, 1:14 pm, Stuart  wrote:
>
> > Twitter have a business plan, we're just not worthy enough to know all
> > the details. What we know so far is that they're planning to launch a
> > premium account type with a bunch of tools to aid brand and engagement
> > tracking.
>
> I've got news for you ... Twitter itself is woefully behind the curve
> on monitoring / marketing / analytics technologies. Third parties are
> springing up daily with offerings in this area, many of them involving
> cutting-edge natural language processing. Twitter could obviously
> invest in these areas, but I'm not sure why they would, rather than
> focusing on stability, scalability and security of the underlying
> platform and messaging systems.
>
> Perhaps one way to "monetize" Twitter would be to implement a per-
> follower charge, say, free up to 2000 followers, then a small monthly
> fee up to 10,000, a larger fee up to 100,000 and so on. I haven't seen
> follower count distribution data recently, but I'd say that people
> with more than 2000 followers are rare and are probably using Twitter
> as a "push marketing / sales" platform in some sense. Of course, I'm
> at around 3500 followers at the moment, so I would be paying a monthly
> fee and would need to justify it as a business expense (or block about
> 1500 people, which isn't out of the question) ;-)

[twitter-dev] Re: Filter Profanity

[David Fisher]

Why don't you just do the filtering on your end? Twitter's API's job
is to give you data- what you do with filtering it on your end should
be up to you...

On Jul 17, 12:43 pm, Steve Brunton  wrote:
> On Fri, Jul 17, 2009 at 12:14 PM, Cameron Kaiser wrote:
>
> > (IDNSOWFT)
> > The problem with this is that it puts Twitter in the position of having to
> > a be a de facto content censor. Besides people having varying ideas of what
> > constitutes offensive, it also possibly subjects them to legal consequences.
>
> As a funny example of the "varying ideas" topic. Here where I work (hi
> Doug) we had the Big List of Bad Terms from one of our other business
> units that was supposed to be the "master" list of words that we were
> to filter on. The one that always gave us humour in meetings was
> "downblouse" .. Yes "downblouse" was on the restricted list and
> supposed to be filtered out. YMMV as to what should or should not be
> filtered and should really be something that the consumer side deals
> with. Personally I'm attempting to increase my ranking on Cursebird,
> but that's just me.
>
> -steve

[twitter-dev] Re: Twitter is not making money

[David Fisher]

Don't feed the trolls. Name a Venture backed startup that DOESNT run
on VC for the first few years. Ok? Right. Stupid topic. Lets move on
and talk about things that matter (ie. development)

On Jul 16, 4:34 pm, Andrew Badera  wrote:
> On Thu, Jul 16, 2009 at 4:26 PM, Nick Arnett  wrote:
>
> > On Thu, Jul 16, 2009 at 1:14 PM, Stuart  wrote:
>
> >> Twitter have a business plan, we're just not worthy enough to know all
> >> the details. What we know so far is that they're planning to launch a
> >> premium account type with a bunch of tools to aid brand and engagement
> >> tracking. If Twitter can maintain their popularity big business will
> >> pay a small fortune to be able to measure the effectiveness of the way
> >> they're using their accounts.
>
> > At the risk of really deviating from developer talk... We know this?  Who's
> > "we" and how do we know this?
>
> > I have a hard time seeing how analysis of Twitter alone would compete with
> > existing services that monitor brands in conversations across many
> > platforms.  I started one of the first companies to do that, ten years ago,
> > which is quite a head start... and it is now owned by one of the biggest
> > brand monitoring companies on the planet.  Lots of competition has come
> > along since then.
>
> > Anyway, this was fun, but it's not about developing code as such, so I'll
> > shut up.  Maybe this is a conversation for that non-platform-specific social
> > media developer community I was wondering about... ;-)
>
> > Nick
>
> "We" is anyone who have paid attention to the twitter dev list and blogs and
> scuttlebutt. Biz has talked about the premium accounts. Contextual
> advertising has been brought up several times. Search-driven monetization
> has also been brought up.
>
> Who's "we?" Anyone with their eyes and ears open.
>
> Thanks-
> - Andy Badera
> - and...@badera.us
> - Google me:http://www.google.com/search?q=andrew+badera
> - This email is: [ ] bloggable [x] ask first [ ] private

[twitter-dev] Re: How to insure that all tweets are retrieved in a search?

[David Fisher]

Err, the Search API isn't limited to 150 requests per hour. It's much
higher than that. Much, but not unlimited.

As John said, read into the Search API more, and check into the
Streaming API as well.

It is certainly possible to get more than 1,500 results for a term,
but not by using simple paging. I've been able to pull 2M+ results for
a query before. It took 8 hours or so, but it worked. Read up more on
the Search API and you should be able to figure it out.

-David Fisher
http://WebecologyProject.org

On Jul 9, 8:16 pm, John Kalucki  wrote:
> First, I wouldn't expect that thousands are going to post your promo
> code
> per minute. That doesn't seem realistic.
>
> Second, in addition to the Search API, which is quite liberal, you can
> use
> the /track method on the Streaming API, which will return all keyword
> matches up to a certain limit with no other rate limiting. Contact us
> if the
> default limits are an issue.
>
> -John Kalucki
> Services, Twitter Inc.
>
> On Jul 9, 3:51 pm, owkaye  wrote:
>
>
>
> > > You are correct, you have to do 15 requests.  However,
> > > you can cache the results in your end, so when you come
> > > back, you are only getting the new stuff.
>
> > Thanks Scott.  I'm storing the results in a database on my server but
> > that doesn't stop the search from retrieving the same results
> > repetitively, because the search string/terms are still the same.
>
> > My problem is going to occur when thousands of people start tweeting
> > my promo codes every minute and I'm not able to retrieve all those
> > tweets because of the search API limitations.
>
> > If I'm limited to retrieving 1500 tweets every 6 minutes and people
> > post 1000 tweets every minute I need some way of retrieving the
> > missing 4500 tweets -- but apparently Twitter doesn't offer anything
> > even remotely close to this capability -- so I can see where it has a
> > long way to go before it's ready to support the kind of search
> > capabilities I need.
>
> > > Twitter has pretty good date handling, so you specify
> > > your last date, and pull forward from there.  You may
> > > even be able to get the last id of the last tweet you
> > > pulled, and just tell it to get you all the new ones.
>
> > Yep, that's what I'm doing ... pulling from the records I haven't
> > already retrieved based on the since_id value.
>
> > But when the new tweets total more than 1500 in a short time, the
> > excess tweets will get lost and there's no way to retrieve them --
> > unless I run my searches from multiple servers to avoid Twitter's ip
> > address limits -- and doing this would be a real kludge that I'm not
> > tempted to bother with.
>
> > > > I'm building an app that uses the atom search API to retrieve recent
> > > > posts which contain a specific keyword.  The API docs say:
>
> > > > "Clients may request up to 1,500 statuses via the page and rpp
> > > > parameters for the search method."
>
> > > > But this 1500 hits per search cannot be done in a single request
> > > > because of the "rpp" limit.  Instead I have to perform 15 sequential
> > > > requests in order to get only 100 items returned on each page ... for
> > > > a total of 1500 items.
>
> > > > This is certainly a good way to increase the server load, since 15
> > > > connections at 100 results each takes far more server resources than 1
> > > > connection returring all 1500 results.  Therefore I'm wondering if I'm
> > > > misunderstanding something here, or if this is really the only way I
> > > > can get the maximum of 1500 items via atom search?
>
> > > --
> > > Scott * If you contact me off list replace talklists@ with scott@ *

[twitter-dev] Re: Twitter User List

[David Fisher]

Thanks for the replies Doug.

btw, as far as I can tell the ~ 52M integer values for user ids aren't
fully sequential, with some amount missing in between, which I'm
assuming is intentionally done to make it slightly more difficult to
know the exact user growth trends/patterns (seems valid to me) or for
some other system reason. There's definitely some gaps in there FYI.

dave

On Jun 29, 2:04 pm, Doug Williams  wrote:
> Please do not scrape our site. We have processes in place that will
> automatically block your spiders.
>
> If you feel that you have a compelling need for vast amounts of data please
> email the API team [1] with a detailed description of your needs and the
> value you hope to create and let's have a conversation.
>
> 1.http://apiwiki.twitter.com/Support
>
> Thanks,
> Doug
>
>
>
> On Mon, Jun 29, 2009 at 10:30 AM, Scott Haneda  wrote:
> > I dint think this is a function of a workaround. This is a function of
> > Twitter having a good policy in place to prevent abuse.
>
> > You can do what you want by incrementally querrying the API. The API limits
> > will make it take too long. Even with multiple accounts it will be months
> > before you get a final list. Even then, I'm not sure you could keep on top
> > of new user registrations.
>
> > Having acces to this data could only be used for nefarious efforts. What
> > you want would be a spammers dream.
>
> > I think you would be better and faster to build a crawl farm and crawl all
> > links on Twitter.com and parse the users out, bypassing the API.
>
> > Even with the API, as you add new records, those records you just added
> > will expire, delete, get banned, blocked etc. There is no way you could ever
> > have a reconciled system.
>
> > Consider if each username is an average 10 bytes. You have 520,000,000
> > bytes to download of just username data. Let's double that for http overhead
> > and other misc data that will come over the wire. 1 billion bytes.
>
> > That's a strongly conservative terrabyte of data that you would have to
> > download once a day and reconcile against the previous day. A terrabyte of
> > just usernames.
>
> > Then you have all the CPU that you will need, network lag, time to insert
> > into your data source.
>
> > This is not something that can be worked around. This is simply a
> > limitation of scale, one that can not be overcome. You need a direct link to
> > twitters data sources, ideally from within their data center to reduce
> > network lag. This probably will not be approved :)
> > --
> > Scott
> > Iphone says hello.
>
> > On Jun 29, 2009, at 9:06 AM, Arunachalam  wrote:
>
> > Even if i have my account whitelisted, which have 20,000 request per hour,
> > i need to run for many days which is not feasible. Any other workaround.
>
> > Any other way to get rid of these request limit.
>
> > Cheers,
> > Arunachalam
>
> > On Mon, Jun 29, 2009 at 7:01 PM, Abraham Williams < <4bra...@gmail.com>
> > 4bra...@gmail.com> wrote:
>
> >> There has been over 5200 profiles created. You could just start at
> >> 1 and count up. Might take you a while though.
>
> >> Abraham
>
> >> On Mon, Jun 29, 2009 at 07:55, Arunachalam< 
> >> arunachala...@gmail.com> wrote:
> >> > Any idea how to implement the same using php / any other language.
> >> > Im confused abt the implementation.
>
> >> > Cheers,
> >> > Arunachalam
>
> >> > On Mon, Jun 29, 2009 at 5:57 PM, Cameron Kaiser <
> >> spec...@floodgap.com>
> >> > wrote:
>
> >> >> > I am looking to find the entire twitter user list ids.
>
> >> >> > Social graph method provides the way to fetch the friends and
> >> followers
> >> >> > id,
> >> >> > thorough which we can access the profile of the person using user
> >> method
> >> >> > -
> >> >> > show. But this requires a code to be written to recursively crawl the
> >> >> > list
> >> >> > from any starting id and appending the followers and friends id of
> >> the
> >> >> > person without duplicating.
>
> >> >> > Do we have any other API to get entire list. If not, any other ways
> >> >> > apart
> >> >> > from crawling to get the entire list.
>
> >> >> No, and no, there are no other ways.
>
> >> >> --
> >> >>  personal:
> >> >> http://www.cameronkaiser.com/--
> >> >>  Cameron Kaiser * Floodgap Systems * 
> >>www.floodgap.com*
> >> >> ckai...@floodgap.com
> >> >> -- Careful with that Axe, Eugene. -- Pink Floyd
> >> >> ---
>
> >> --
> >> Abraham Williams | Community Evangelist | 
> >>http://web608.org
> >> Hacker | http://abrah.am| 
> >>http://twitter.com/abraham
> >> Project | 
> >>http://fireeagle.labs.poseurtech.com
> >> This email is: [ ] blogable [x] ask first [ ] private.

[twitter-dev] Re: getting the initial tweet on a subject

[David Fisher]

If its within the window that twitter offers via search, you can go
back via a mixture of using the max_id, a rpp of 100, and paging.

However there is almost no way to verify that the oldest one is truely
the first. What you really need is Firehose, and then constantly log
everything to be able to find the original one. Unfortunately, it
seems few of us will ever get this type of access.

The best thing you can do, is to use the Track Streaming API to
constantly catch all tweets for things that are trending, refresh the
list of what's trending frequently, and the on any change, so a search
back for the past 10 days for all the tweets for that term. Without
Firehose, I can't think of a better way.

-dave/ @tibbon

On Jun 19, 12:41 am, Cameron Kaiser  wrote:
> > I am wondering if it is possible for me to get the first ever tweet on
> > a trending topic?
>
> You could try to page back arbitrarily far but there is no specific method
> for this, and it won't work for extremely fat or popular topics.
>
> --
>  personal:http://www.cameronkaiser.com/--
>   Cameron Kaiser * Floodgap Systems *www.floodgap.com* ckai...@floodgap.com
> -- Michael Jackson is [reverse] Pinocchio: more lies, less nose. -- Vanity 
> Fair

[twitter-dev] Re: airline accident case study

[David Fisher]

Topics don't just trend because its something 'important'. Now if it
was of significantly larger volume than another topics (like the
iphone's launch today), then that is rather interesting, but from what
I can tell its mostly the most popular things floating to the top
generally, plus some spam-filtering. I haven't figured out the
exacting mechanisim for when something hits trending, but its not
rocket science either.

Maybe I'm missing your point

-...@tibbon

On Jun 19, 2:35 am, Bjoern  wrote:
> On Jun 19, 7:00 am, AJ  wrote:
>
> > This case study shows the difference between various trending
> > applications. A good real time semantic analysis is the key that makes
> > the difference, I think.
>
> Maybe I misunderstood, but isn't the more likely explanation that the
> topic simply wasn't trending?
>
> Björn

[twitter-dev] Re: Twitpocalypse and yajl

[David Fisher]

Has anyone had any luck getting Brian Mario's Yajl gem working?
http://github.com/brianmario/yajl-ruby/tree/master

It seems to be breaking for me, but my code was working prior really
well. I tried to patch it, and rebuild the gem, but I was getting
errors installing.

This is critical for me to fix asap.

Thanks,

David/@tibbon

On Jun 14, 6:22 pm, Matt Sanford  wrote:
> Hi there,
>
>      Someone emailed me directly to mention that the gist linked to  
> still has a hard coded string length (strtoul is using 10) that would  
> stop working on the newly supported long numbers (up to 20 digits).  
> I'm not quite sure why they emailed me directly but I'll withhold  
> their name in case it was not an accident.
>
> Thanks;
>   – Matt Sanford / @mzsanford
>       Twitter Dev
>
> On Jun 12, 2009, at 5:30 PM, Steve Streza wrote:
>
>
>
> > For those who didn't survive to the post-twitpocalyptic world thanks
> > to YAJL, give this patch a shot.
>
> >http://gist.github.com/129033

[twitter-dev] Re: Is the spritzer supposed to be as fast as the datamining feed?

[David Fisher]

Gardenhose is around what... 1.5M tweets/day? I think that's about
right. Yes, that is a LOT of data and going through it then takes a
lot. My server started crying a bit when I switched up to gardenhose.
I'm going to have to get a bigger one if I ever can consume the
firehose

On Jun 11, 5:04 pm, "M. Edward (Ed) Borasky"  wrote:
> I clocked spritzer (JSON feed) at something like 29,000 tweets per
> hour between 10 and 11 AM PDT the other day. That may in fact be more
> data than I can practically use.
>
> On May 25, 11:12 pm, John Kalucki  wrote:
>
> > The sampling rates of the Streaming API and the REST API public
> > methods are independent, and any similarity in proportions is
> > basically happenstance.
>
> > Currently, Gardenhose will average three timesSpritzer, with some
> > deviation. The rate of all statuses varies significantly throughout
> > the day and week, so be sure you are comparing the exact same time
> > periods.
>
> > -John Kalucki
> > Services, Twitter Inc.
>
> > On May 25, 7:20 pm, Twittledee  wrote:
>
> > > Hello:
> > >  In an answer to a different question in this group, Doug said that
> > > thespritzerwas supposed to be as fast as the datamining feed. Is
> > > that right? We only get 5.3 msg/sec on avg. for thespritzer. If this
> > > is right, then we are only getting half the speed we should be ... and
> > > I will have to debug our code. We get 18.7 msg/sec avg. on the
> > > gardenhose, so I thought everything was right/okay. Can others state
> > > their rates for these two streams?
>
> > > Thanks

[twitter-dev] Re: how are people collecting spritzer/gardenhose?

[David Fisher]

I'm just using a realtime json parser in Ruby written as a native C
extension (http://github.com/brianmario/yajl-ruby/tree/master)
It's really simple to use and well documented.

I'm just storing everything in a Postgres database, and then using
other scripts to query it. Note: using gardenhose at least you get a
LOT of data fast. In just a few days, I have a 4GB+ database now or so

On Jun 11, 5:10 pm, "M. Edward (Ed) Borasky"  wrote:
> Right now, I'm collecting spritzer data with a simple shell script
> "curl  | bzip2 -c > .bz2". A cron
> job checks every minute and restarts the script if it crashes. The
> rest is simple ETL. :)

[twitter-dev] Re: New Public Streaming API Resource - Follow

[David Fisher]

If I have Gardenhose access, do I automatically have access to Shadow
or Birddog or do I need to send in a separate application/contract to
gain access?

Thanks,

David

On Jun 11, 12:09 pm, John Kalucki  wrote:
> Let me restate: If you are trying to follow many prolific users via
> the REST API, then, yes, it just might be mathematically possible to
> have the REST API rate limit prevent you from seeing all of your
> timeline. In other words, if you follow 2000 accounts that each post
> every 5 minutes, that's 400 messages per minute. If you can poll for
> your timeline once a minute and only get 300 messages, you've lost 100
> messages...  Are there 2000 accounts that each post every 5 minutes?
> Probably not. Would anyone want to read this? Unlikely. But, for
> certain applications that need this access, there's the Streaming
> API...
>
> The Streaming API does not use oAuth, rather Basic Auth. The Streaming
> API is primarily intended for Server to Server communication, although
> desktop applications can work too. Desktop developers, or anyone
> intending to form a per-end-user connection to the Streaming API
> should contact us to hash out loading and scaling issues before
> launching into something unsupportable on what is still an Alpha
> service.
>
> Any Twitter account ("AnyTwitterUser") can request to the /follow and /
> spritzer methods. Each account should only log into the Streaming API
> once and maintain the connection for as long as practical. Forming
> excessive connections from the same account or from the same IP
> address will lead to temporary banning. Under normal legitimate use,
> connections should not be denied, and we endeavor to adjust our
> algorithms to balance legitimate access against abuse.
>
> If your service needs to follow more than 200 users, you should apply
> for the /shadow method that allows 2000 accounts. If you need to
> follow more than 2000 accounts, let's discuss. Perhaps we can raise
> this limit somewhat to allow more applications to operate.
>
> I hope I've answered your questions. Keep them coming!
>
> -John Kalucki
> Services, Twitter, Inc.
>
> On Jun 11, 8:15 am, AE  wrote:
>
> > On Jun 10, 6:51 pm, John Kalucki  wrote:
>
> > > I haven't done the math, but I don't think a single prolific user
> > > could tweet more than you can consume. Now, if you are trying to
> > > consume an inordinate number of users, then yes, you'll have a
> > > problem. TheStreamingAPIlimits the number of users you can follow,
> > > but not how many statuses you receive from those users.
>
> > Thank you John,
>
> > Question is what is that "inordinate number of users"?
> > I mean each twitter user who has given the 3rd party app
> > an oAuth approval can follow up to 200 users no? -- (this is where I
> > am
> > getting lost)
>
> > In that case the 3rd party site can request to follow
> > (say the site has 1000 users and uses 1 IP address)
> > 1000 * 200 users??
>
> > curl -d @followinghttp://stream.twitter.com/follow.json-
> > uAnyTwitterUser:Password.
>
> > The problem I have above is "AnyTwitterUser" on the curl request. It
> > is my understanding
> > a signle IP can only follow 200 users with 1 twitter account. Cos I am
> > not allowed to make
> > multiple curl request?
>
> > I am sorry if I am confusing everyone.
>
> > > -John Kalucki
> > > Services, Twitter Inc.
>
> > > On Jun 10, 9:26 am, AE  wrote:
>
> > > > Hi John
>
> > > > Questions regardingfollow.
>
> > > > On May 13, 10:50 pm, John Kalucki  wrote:
>
> > > > > I'll attempt to answer these questions, but I can only do so with some
> > > > > speculation and humble ignorance.
>
> > > > > 1) OAuth allows clients to authenticate with the Twitter RESTAPIvia
> > > > > third-party services. These services should not also need to interact
> > > > > with theStreamingAPIon a per client basis. Instead, the service
> > > > > should establish a single query that satisfies all clients' needs.
> > > > > This may not be practical in all cases, but I suspect we can
> > > > > approximate the desired behavior with the current set of primitives.
>
> > > > How about IP Rate Limit? I mean if the people Ifollowthey tweet more
> > > > then the limit... what happens.. There is no such rate limit 
> > > > inStreamingAPI?
> > > > Cos you mention RESTAPIabove.. not sure I understand it.
>
> > > > Thanks again.
>
> > > > > 2) There are no immediate plans to support HTTPS, mainly because we're
> > > > > not really trying to keep the data private. Also, and I am probably
> > > > > totally wrong here, I don't think we use HTTPS on the main WWW site or
> > > > > on the RESTAPI, so this doesn't make things much worse than they
> > > > > already are. A possible workaround would be for sensitive service to
> > > > > create an account just forstreaming. Should the password be
> > > > > compromised, there's only a denial of service risk and no further
> > > > > risk.
>
> > > > > -John
>
> > > > > On May 13, 11:18 am, Marco Kaiser  wrote:
>
> > > > > > John

[twitter-dev] Re: Twitter conference

[David Fisher]

They had one the other week:
http://twtrcon.com/

-David Fisher
@tibbon

On Jun 8, 2:26 pm, Emrah  wrote:
> Hi everybody,
>
> Is there any planned Twitter Meetup in the near future? Is there any
> official schedule available online?
> I heard about #140conf but I was imagining something more developers
> oriented. Something to discuss possible improvements, add-ons or similar
> stuffs about Twitter and Twitter apps.
>
> I was actually imagining planning a Web or Phone conference between
> Twitter admins, developers and perhaps some interesting users?
> In both cases, the conference could be recorded as a podcast to be
> redistributed if necessary.
> Mixing both telephone and Web conferencing for users who are not able to
> join the Web meeting is possible.
>
> About the Web meeting, it would be possible to support 2 or 3 concurrent
> video streams but the number of participants would be unlimited.
> Regarding the telephone conferencing, it could be possible to integrate
> Skype, VoIP, regular phone lines and a callback function for users who
> can't make international phone calls.
>
> What do you think?
>
> Emrah

[twitter-dev] Re: What is current search time range limit?

[David Fisher]

Can't you get more than 1500 by using Max_id and ID together and
creatively?

-David
@tibbon

On Jun 6, 2:41 pm, Doug Williams  wrote:
> See [1] for information on replication. Sometimes there is a lag in the
> replication of the database across multiple machines.
>
> 1.http://en.wikipedia.org/wiki/Replication_(computer_science)
>
> Thanks,
> Doug
>
> On Sat, Jun 6, 2009 at 8:24 AM,  wrote:
>
> > What do you mean by "replication lag"? Is this related to repeated
> > tweets in the search results?

[twitter-dev] Re: random sampling of users....do we know anything about user id range?

[David Fisher]

I'm hoping that Twitter counts "users" when reporting their numbers,
and not "accounts". The reason being that I've signed up probably... 5
accounts myself (main, API testing, business, etc, etc). I'm not sure
how many the average user signs up, but it's definitely on average
more than 1.

I'm wondering if IDs are sequential for users afterall. Not using
system generated primary keys if I remember right puts more strain on
the system as it has to check uniqueness and generate a number (not
hard, but still), and most of Twitter is all about scaling and speed
as I see it.

Otherwise it seems on creating a new user, they are taking the last
ID, and adding an artibrary number to it (1d20?) for the next user
ID.

45M seems like a lot of users, but I could see there being that many
"accounts". perhaps

On Jun 4, 11:44 am, Nick Arnett  wrote:
> On Wed, Jun 3, 2009 at 7:13 PM, TechRavingMad wrote:
>
>
>
> > There are a little over 44.5 million twitter IDs as of right now
> > (10:10pm cst 6/3/9) with what seems to be about 10 being added every
> > second.
>
> However, Twitter has been quite clear about not saying if status IDs
> correspond to the actual number of statuses, so I'd guess that they're
> equally circumspect about whether or not the number of user IDs corresponds
> to the number of users.  In other words, we can be sure there are not more
> than 44.5 million users, but we don't know how much lower the actual number
> is.  We don't know if all IDs have been used... and even Twitter doesn't
> know how many of those IDs belong to the same users.
>
> I would think that if one wants a random sample of users, one would have to
> propose a selection method and ask Twitter if there's any reason that it
> would introduce a selection bias... and hope that they are willing to reply.
>
> Seems to me that the biggest problem would be to include "quiet" users,
> since only those who post in public become visible.
>
> NIck

[twitter-dev] Re: Forgive Forgive Forgive - XML 2 CSV XSL

[David Fisher]

http://snippets.dzone.com/posts/show/3701

Gets it done in Ruby. I haven't tested it, but it looks right and
simple

On Jun 4, 4:04 pm, Mark  wrote:
> I would like to take the follower .xml output and convert to csv.
>
> From googling I believe I need to create an XSL or XSD and then use
> MSXML
>
> example:
>
> MSXSL C:\twitterout\adventuregirl69.txt C:\Users\mark\Documents
> \twitteratom2txt.xsl  –o out.xml
>
> I don't know xsl trying to learn from examples, does anyone have
> any .XSLs they created for Twitter's API responses?
>
> Would like to create flat file output from xml response.
>
> Goal is to tie a zip code to a twitter profile like @geofollow