Thanks for the responses. That's unfortunately not the case. We've gone massively concurrent with this (we've wrapped our own concurrent queueing around twitter4j, which it looks like Yusuke has just updated to add his own concurrent approach to this...thx), and falling behind on the stream is definitely not the problem.
Bandwidth is not the problem. I still don't understand why, right after reconnecting to the twitter firehose, the stream just plain ends. It's not happening after some time of possibly falling behind...it happens right away. To clarify, this is NOT an issue all the time. And since I posted this the other day, the problem has gone away as mysteriously as it arrived. But it does happen from time to time. When everything else on my app's end is steady-state, the flying fickle finger of blame points to twitter... :-) Thanks, Dan On Fri, Dec 17, 2010 at 1:57 PM, Matt Harris <thematthar...@twitter.com>wrote: > Hey Dan, > > If you fall too far behind when receiving the stream we will disconnect > you. Check the timestamp of the Tweets being received to the time on your > computer. If the times are drifting further apart you are falling behind. > > The most common reasons for falling behind are: > 1. You are attempting to process the stream in the same code that consumes > them - instead of running a queuing system. > 2. Your connection is being used by other processes reducing your available > bandwidth. > > As Tom suggested, run through the Streaming Documentation linked to from > http://dev.twitter.com/pages/streaming_api and make sure you implement the > suggestions. > > Best, > @themattharris > Developer Advocate, Twitter > http://twitter.com/themattharris > > > On Fri, Dec 17, 2010 at 8:52 AM, Dan Checkoway <dchecko...@gmail.com>wrote: > >> Our app is using twitter4j 2.1.9-SNAPSHOT against the twitter firehose, >> and lately we are having problems with the stream "just ending" out of the >> blue. It happens several times a day lately with no rhyme or reason. >> Here's an example of what we see: >> >> Stream closed.TwitterException{exceptionCode=[a3652dee-000a1c7a >> a3652dee-000a1c35], statusCode=-1, retryAfter=0, rateLimitStatus=null, >> version=2.1.9-SNAPSHOT} >> at >> twitter4j.AbstractStreamImplementation.handleNextElement(AbstractStreamImplementation.java:149) >> at twitter4j.StatusStreamImpl.next(StatusStreamImpl.java:74) >> at >> twitter4j.TwitterStream$TwitterStreamConsumer.run(TwitterStream.java:687) >> Caused by: java.io.IOException: the end of the stream has been reached >> at >> twitter4j.AbstractStreamImplementation.handleNextElement(AbstractStreamImplementation.java:80) >> ... 2 more >> >> We call cleanUp and then reconnect to the firehose, and then the same >> thing happens again. Eventually we get temporarily rate limited due to the >> reconnect attempts. >> >> I'm wondering if anybody else out there has seen this issue on the >> firehose, or if anybody has a suggestion on how to avoid or work around it? >> Is the likely cause on twitter's side, or could something on the client side >> be causing this? >> >> Thanks, >> Dan >> >> -- >> Twitter developer documentation and resources: http://dev.twitter.com/doc >> API updates via Twitter: http://twitter.com/twitterapi >> Issues/Enhancements Tracker: >> http://code.google.com/p/twitter-api/issues/list >> Change your membership to this group: >> http://groups.google.com/group/twitter-development-talk >> > > -- > Twitter developer documentation and resources: http://dev.twitter.com/doc > API updates via Twitter: http://twitter.com/twitterapi > Issues/Enhancements Tracker: > http://code.google.com/p/twitter-api/issues/list > Change your membership to this group: > http://groups.google.com/group/twitter-development-talk > -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
