Re: [twitter-dev] Streaming API time drifting problem and possible solutions

2010-07-08 Thread Pascal Jürgens
Larry, moreover, I assume you checked I/O and CPU load. But even if that's not the issue, you should absolutely check if you have simplejson with c extension installed. The python included version is 1.9 which is decidedly slower than the new 2.x branch. You might see json decoding load drop by

Re: [twitter-dev] Streaming API time drifting problem and possible solutions

2010-07-08 Thread Pascal Jürgens
Larry, have you decoupled the processing code from tweepy's StreamListener, for example using a Queue.Queue oder some message queue server? Pascal On Jul 8, 2010, at 17:31 , Larry Zhang wrote: > Hi everyone, > > I have a program calling the statuses/sample method of a garden hose > of the Str

Re: [twitter-dev] Streaming API time drifting problem and possible solutions

2010-07-08 Thread John Kalucki
Absolutely do not reset the connection and reconnect. Connections should be long-lived on the Streaming API. This is almost certainly a problem with the read throughput of your client, or, less likely, with bandwidth from your system. Run curl(1) from the same system and grep for the date field. I

[twitter-dev] Streaming API time drifting problem and possible solutions

2010-07-08 Thread Larry Zhang
Hi everyone, I have a program calling the statuses/sample method of a garden hose of the Streaming API, and I am experiencing the following problem: the timestamps of the tweets that I downloaded constantly drift behind real-time, the time drift keeps increasing until it reaches around 25 minutes,