[twitter-dev] Streaming API question..
hi there, I am using the streaming API (the statuses/filter), and I get a lot of tweets in spanish. I wanted to know if there is a way to get results only in English? I tried to use the geo-location of the USA only, but it didn't help much. anybody? thanks, Omri -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Streaming API question..
Hi Omri, Sorry, there's no option currently to filter by language. Taylor On Mon, Sep 27, 2010 at 1:50 AM, omri omridek...@gmail.com wrote: hi there, I am using the streaming API (the statuses/filter), and I get a lot of tweets in spanish. I wanted to know if there is a way to get results only in English? I tried to use the geo-location of the USA only, but it didn't help much. anybody? thanks, Omri -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Streaming API question
* rob robert.bag...@gmail.com [100219 08:56]: Has anyone else ran into an issue where over time the Streaming API just stops sending results? Yes. I'm seeing the same thing. I've set up a 45 second timeout. The following entries were extracted from the application log. I'm currently following 100 users, so periods of inactivity are not unusual. Receipt of keep alive packets are indicated by ping. If there's no activity for 45 seconds, I drop the connection and reconnect. Times are PST. @semifor 2010/02/09 14:39:44 connecting 2010/02/09 14:40:29 timeout 2010/02/09 14:40:29 Waiting 0 seconds before reconnecting... -- 2010/02/09 16:09:10 ping 2010/02/09 16:09:40 ping 2010/02/09 16:10:25 timeout 2010/02/09 16:10:25 Waiting 0 seconds before reconnecting... 2010/02/09 16:10:25 connecting 2010/02/09 16:11:10 timeout 2010/02/09 16:11:10 Waiting 0 seconds before reconnecting... -- 2010/02/11 08:54:04 8968466186: @BarackObama. Gd morning MR President. OMG u twi 2010/02/11 08:54:49 timeout 2010/02/11 08:54:49 Waiting 0 seconds before reconnecting... 2010/02/11 08:54:49 connecting 2010/02/11 08:55:34 timeout 2010/02/11 08:55:34 Waiting 0 seconds before reconnecting... -- 2010/02/12 08:50:40 ping 2010/02/12 08:51:10 ping 2010/02/12 08:51:55 timeout 2010/02/12 08:51:55 Waiting 0 seconds before reconnecting... -- 2010/02/12 11:45:05 ping 2010/02/12 11:45:35 ping 2010/02/12 11:46:20 timeout 2010/02/12 11:46:20 Waiting 0 seconds before reconnecting... 2010/02/12 11:46:20 connecting 2010/02/12 11:47:05 timeout 2010/02/12 11:47:05 Waiting 0 seconds before reconnecting... -- 2010/02/16 07:11:25 9188373420: @alexpriest oh Alex, you're a fool. Haha. :D Dri 2010/02/16 07:11:55 ping 2010/02/16 07:12:40 timeout 2010/02/16 07:12:40 Waiting 0 seconds before reconnecting... -- 2010/02/16 08:38:49 Waiting 0 seconds before reconnecting... 2010/02/16 08:38:49 connecting 2010/02/16 08:39:34 timeout 2010/02/16 08:39:34 Waiting 0 seconds before reconnecting... -- 2010/02/17 07:28:03 9238057534: @BarackObama President B. Pls explain this to us 2010/02/17 07:28:48 timeout 2010/02/17 07:28:48 Waiting 0 seconds before reconnecting... -- 2010/02/18 09:24:21 9292626812: @SuzieLin no problem! So far can't complain, jus 2010/02/18 09:24:50 9292645527: @BarackObama Another terrorism actack? How many 2010/02/18 09:25:35 timeout 2010/02/18 09:25:35 Waiting 0 seconds before reconnecting... -- 2010/02/19 09:36:16 9344797252: @BarackObama We The People want the #PNHP at 2010/02/19 09:37:01 timeout 2010/02/19 09:37:01 Waiting 0 seconds before reconnecting... 2010/02/19 09:37:01 connecting 2010/02/19 09:37:46 timeout 2010/02/19 09:37:46 Waiting 0 seconds before reconnecting...
Re: [twitter-dev] Streaming API question
A 45 second period of inactivity is not unusual when following just 100, or even 100,000 users. The keep-alive newlines are only sent once every 10 minutes. You should not reconnect so aggressively. -John Kalucki http://twitter.com/jkalucki Infrastructure, Twitter Inc. On Sat, Feb 20, 2010 at 7:59 PM, Marc Mims marc.m...@gmail.com wrote: * rob robert.bag...@gmail.com [100219 08:56]: Has anyone else ran into an issue where over time the Streaming API just stops sending results? Yes. I'm seeing the same thing. I've set up a 45 second timeout. The following entries were extracted from the application log. I'm currently following 100 users, so periods of inactivity are not unusual. Receipt of keep alive packets are indicated by ping. If there's no activity for 45 seconds, I drop the connection and reconnect. Times are PST. @semifor 2010/02/09 14:39:44 connecting 2010/02/09 14:40:29 timeout 2010/02/09 14:40:29 Waiting 0 seconds before reconnecting... -- 2010/02/09 16:09:10 ping 2010/02/09 16:09:40 ping 2010/02/09 16:10:25 timeout 2010/02/09 16:10:25 Waiting 0 seconds before reconnecting... 2010/02/09 16:10:25 connecting 2010/02/09 16:11:10 timeout 2010/02/09 16:11:10 Waiting 0 seconds before reconnecting... -- 2010/02/11 08:54:04 8968466186: @BarackObama. Gd morning MR President. OMG u twi 2010/02/11 08:54:49 timeout 2010/02/11 08:54:49 Waiting 0 seconds before reconnecting... 2010/02/11 08:54:49 connecting 2010/02/11 08:55:34 timeout 2010/02/11 08:55:34 Waiting 0 seconds before reconnecting... -- 2010/02/12 08:50:40 ping 2010/02/12 08:51:10 ping 2010/02/12 08:51:55 timeout 2010/02/12 08:51:55 Waiting 0 seconds before reconnecting... -- 2010/02/12 11:45:05 ping 2010/02/12 11:45:35 ping 2010/02/12 11:46:20 timeout 2010/02/12 11:46:20 Waiting 0 seconds before reconnecting... 2010/02/12 11:46:20 connecting 2010/02/12 11:47:05 timeout 2010/02/12 11:47:05 Waiting 0 seconds before reconnecting... -- 2010/02/16 07:11:25 9188373420: @alexpriest oh Alex, you're a fool. Haha. :D Dri 2010/02/16 07:11:55 ping 2010/02/16 07:12:40 timeout 2010/02/16 07:12:40 Waiting 0 seconds before reconnecting... -- 2010/02/16 08:38:49 Waiting 0 seconds before reconnecting... 2010/02/16 08:38:49 connecting 2010/02/16 08:39:34 timeout 2010/02/16 08:39:34 Waiting 0 seconds before reconnecting... -- 2010/02/17 07:28:03 9238057534: @BarackObama President B. Pls explain this to us 2010/02/17 07:28:48 timeout 2010/02/17 07:28:48 Waiting 0 seconds before reconnecting... -- 2010/02/18 09:24:21 9292626812: @SuzieLin no problem! So far can't complain, jus 2010/02/18 09:24:50 9292645527: @BarackObama Another terrorism actack? How many 2010/02/18 09:25:35 timeout 2010/02/18 09:25:35 Waiting 0 seconds before reconnecting... -- 2010/02/19 09:36:16 9344797252: @BarackObama We The People want the #PNHP at 2010/02/19 09:37:01 timeout 2010/02/19 09:37:01 Waiting 0 seconds before reconnecting... 2010/02/19 09:37:01 connecting 2010/02/19 09:37:46 timeout 2010/02/19 09:37:46 Waiting 0 seconds before reconnecting...
Re: [twitter-dev] Streaming API question
* John Kalucki j...@twitter.com [100220 20:24]: A 45 second period of inactivity is not unusual when following just 100, or even 100,000 users. The keep-alive newlines are only sent once every 10 minutes. You should not reconnect so aggressively. I can certainly set the time out to 10 minutes. I'm seeing newlines in the stream every 30 seconds, except for rare occasions. I understood those to be keep-alive packets. Apparently they are not and should not be relied on? @semifor
Re: [twitter-dev] Streaming API question
Arg. This is what I get for not checking the configuration each time. Yes, it's currently set to send a newline every 30 seconds. On Sat, Feb 20, 2010 at 8:45 PM, Marc Mims marc.m...@gmail.com wrote: * John Kalucki j...@twitter.com [100220 20:24]: A 45 second period of inactivity is not unusual when following just 100, or even 100,000 users. The keep-alive newlines are only sent once every 10 minutes. You should not reconnect so aggressively. I can certainly set the time out to 10 minutes. I'm seeing newlines in the stream every 30 seconds, except for rare occasions. I understood those to be keep-alive packets. Apparently they are not and should not be relied on? @semifor
Re: [twitter-dev] Streaming API question
* John Kalucki j...@twitter.com [100220 21:02]: Arg. This is what I get for not checking the configuration each time. Yes, it's currently set to send a newline every 30 seconds. Ok. Sorry to drag this out, but what, then, is an appropriate timeout value for the application? @semifor
Re: [twitter-dev] Streaming API question
60 or 90 seconds seems reasonable, but your code should also detect a socket close immediately and reconnect immediately. The common case for a connection drop -- a server restart -- should cause your socket to close, the client to detect the closure, and reconnect, all within about a second. On Sat, Feb 20, 2010 at 9:09 PM, Marc Mims marc.m...@gmail.com wrote: * John Kalucki j...@twitter.com [100220 21:02]: Arg. This is what I get for not checking the configuration each time. Yes, it's currently set to send a newline every 30 seconds. Ok. Sorry to drag this out, but what, then, is an appropriate timeout value for the application? @semifor
[twitter-dev] Streaming API question
Has anyone else ran into an issue where over time the Streaming API just stops sending results? We are using a Ruby library to connect (twitter-stream) which uses EventMachine to open a persistent connection to the API (we are tracking and following). The library properly handles reconnection (from dropped connections) and the various error conditions. All works well for a period of time (8+ hours in some cases, sometimes a full day) after that the connection does no get dropped but no data gets sent. (At least that's what is seems as EventMachine feels its still connected) If we just drop the connection and reconnect all is well and the data starts to flow again. (Which we could do but that seems like a hack) Anyone else have this issue or should I dive into the EventMachine code and see if there is an issue in there? Thanks in advance, Rob
Re: [twitter-dev] Streaming API question
Me too! Exact same case. I am using tweepy. On Fri, Feb 19, 2010 at 10:36 AM, rob robert.bag...@gmail.com wrote: Has anyone else ran into an issue where over time the Streaming API just stops sending results? We are using a Ruby library to connect (twitter-stream) which uses EventMachine to open a persistent connection to the API (we are tracking and following). The library properly handles reconnection (from dropped connections) and the various error conditions. All works well for a period of time (8+ hours in some cases, sometimes a full day) after that the connection does no get dropped but no data gets sent. (At least that's what is seems as EventMachine feels its still connected) If we just drop the connection and reconnect all is well and the data starts to flow again. (Which we could do but that seems like a hack) Anyone else have this issue or should I dive into the EventMachine code and see if there is an issue in there? Thanks in advance, Rob -- Regards, Atul Kulkarni
Re: [twitter-dev] Streaming API question
Hi, Yup, I saw it the last couple of weeks, this week has been considerably better. I use the delimited stream so I do read(entry size), read(entry), repeat ... I just put a 30 second timeout on the read operation (this is all in python) and if a read fails I close the stream and reconnect. This seems to work quite well ... depending on the chunk size you are reading you could probably lower the timeout if you are afraid of losing data. ttyl Dima On 19-Feb-10, at 7:36 AM, rob wrote: Has anyone else ran into an issue where over time the Streaming API just stops sending results? We are using a Ruby library to connect (twitter-stream) which uses EventMachine to open a persistent connection to the API (we are tracking and following). The library properly handles reconnection (from dropped connections) and the various error conditions. All works well for a period of time (8+ hours in some cases, sometimes a full day) after that the connection does no get dropped but no data gets sent. (At least that's what is seems as EventMachine feels its still connected) If we just drop the connection and reconnect all is well and the data starts to flow again. (Which we could do but that seems like a hack) Anyone else have this issue or should I dive into the EventMachine code and see if there is an issue in there? Thanks in advance, Rob -- ddbrod...@gmail.com The price of reliability is the pursuit of the utmost simplicity. It is a price which the very rich find the most hard to pay. (Sir Antony Hoare, 1980)
Re: [twitter-dev] Streaming API question
This shouldn't be happening, and having developers build these sorts of workarounds saddens me. It is possible that the server side is holding dead connections open, but I doubt it -- as I've a considerable amount of data to the contrary. I suspect that the socket code does not detect a close, driven by either a TCP Close or a TCP Reset. I've run connections over the public internet with close monitoring and rarely noticed a timeout. If you point the same client at a file of streaming data on a web server, does the client detect the end of file at the correct point? -John Kalucki http://twitter.com/jkalucki Infrastructure, Twitter Inc. On Fri, Feb 19, 2010 at 9:16 AM, Dima Brodsky ddbrod...@gmail.com wrote: Hi, Yup, I saw it the last couple of weeks, this week has been considerably better. I use the delimited stream so I do read(entry size), read(entry), repeat ... I just put a 30 second timeout on the read operation (this is all in python) and if a read fails I close the stream and reconnect. This seems to work quite well ... depending on the chunk size you are reading you could probably lower the timeout if you are afraid of losing data. ttyl Dima On 19-Feb-10, at 7:36 AM, rob wrote: Has anyone else ran into an issue where over time the Streaming API just stops sending results? We are using a Ruby library to connect (twitter-stream) which uses EventMachine to open a persistent connection to the API (we are tracking and following). The library properly handles reconnection (from dropped connections) and the various error conditions. All works well for a period of time (8+ hours in some cases, sometimes a full day) after that the connection does no get dropped but no data gets sent. (At least that's what is seems as EventMachine feels its still connected) If we just drop the connection and reconnect all is well and the data starts to flow again. (Which we could do but that seems like a hack) Anyone else have this issue or should I dive into the EventMachine code and see if there is an issue in there? Thanks in advance, Rob -- ddbrod...@gmail.com The price of reliability is the pursuit of the utmost simplicity. It is a price which the very rich find the most hard to pay. (Sir Antony Hoare, 1980)
