Re: [twitter-dev] twurl with streaming API?
Matt, Thanks for the --no-ssl trick, much appreciated! That was staring me right in the face with the --help output and I still managed not to see it. No worries, I'm only trying to use twurl for a test. We do use twitter4j for live firehose access. Thanks, Dan On Sun, Dec 26, 2010 at 10:54 AM, Matt Harris thematthar...@twitter.comwrote: Hi Dan, The Track/Sample Streaming API doesn't support SSL which twurl enables by default. You need to instruct twurl to not use SSL when making requests. For example: twurl --no-ssl -t -H stream.twitter.com /1/statuses/sample.json This is ok for testing and debugging but you want to look at something like twitter4j or phirehose as a programatic way to consume the Streaming API. Best, @themattharris Developer Advocate, Twitter http://twitter.com/themattharris On Sun, Dec 26, 2010 at 12:42 AM, dcheckoway dchecko...@gmail.com wrote: After authorizing with twurl I tried this: $ twurl -t -H stream.twitter.com /1/statuses/sample.json opening connection to stream.twitter.com... ...and it just sits there. Anybody know if there's a way to use twurl with the streaming API? I can use curl no problem with basic auth, but that's pretty lame and short-lived. It would be nice if twurl supported the streaming API. If it does, can you clue me in? Thanks, Dan -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] 4294967295
Cool, I appreciate the response. I forgot to mention, although you guys probably know this by now...originally it was just the firehose on which we saw those funky values, but lately we've been seeing them in the wild as well. Thanks again, Taylor. Dan On Wed, Dec 22, 2010 at 10:56 PM, Taylor Singletary taylorsinglet...@twitter.com wrote: It's going to be a little bit of time before we can totally prevent these values from occurring. Right now, you should probably just consider this value as unknown rather than necessarily null, 0, or otherwise. The team responsible for the low level component causing the bug has a fix planned, but it can't be applied until a few more dependencies are resolved. Thanks, Taylor On Wed, Dec 22, 2010 at 2:59 PM, Dan Checkoway dchecko...@gmail.comwrote: I just wanted to follow up on this, because the issue continues to happen, and it gets more and more interesting. We've now been seeing user.listed_count coming back as 4294967293 on occasion. So just to recap, we have now seen these values in the user.listed_count field: 4294967295 (a.k.a. unsigned -1) 4294967294 (a.k.a. unsigned -2) 4294967293 (a.k.a. unsigned -3) twitter4j has worked around this issue no problem, but I'm more than just a bit curious what these values represent. Should -1 and -2 and -3 be treated to mean anything other than we don't know what the listed count is? What happens if/when -4 starts popping out? I realize this is pretty low priority, but it's still a bug... Thanks, Dan On Fri, Dec 17, 2010 at 7:17 AM, Dan Checkoway dchecko...@gmail.comwrote: Check this out...today sometime between 4:01:43 AM PST and 4:01:53 AM PST (sorry for the ambiguity, those are our every 10 sec logging timestamps for other stuff), we saw the unsigned equivalent of -2 (4294967294) being sent by twitter in user.listed_count... Exception in thread Twitter Stream Handling Thread[Receiving stream] java.lang .NumberFormatException: For input string: 4294967294 at java.lang.NumberFormatException.forInputString(NumberFormatException. java:48) at java.lang.Integer.parseInt(Integer.java:459) at java.lang.Integer.valueOf(Integer.java:553) at twitter4j.internal.util.ParseUtil.getInt(ParseUtil.java:120) at twitter4j.UserJSONImpl.init(UserJSONImpl.java:103) at twitter4j.UserJSONImpl.init(UserJSONImpl.java:86) at twitter4j.StatusJSONImpl.init(StatusJSONImpl.java:101) at twitter4j.StatusJSONImpl.init(StatusJSONImpl.java:84) at twitter4j.StatusJSONImpl.init(StatusJSONImpl.java:118) at twitter4j.StatusJSONImpl.init(StatusJSONImpl.java:84) at twitter4j.StatusStreamImpl.handleNextElement(StatusStreamImpl.java:116) at twitter4j.StatusStreamImpl.next(StatusStreamImpl.java:89) at twitter4j.TwitterStream$StreamHandlingThread.run(TwitterStream.java:529) Any idea what's going on and/or when it might be fixed? Thanks, Dan On Tue, Dec 14, 2010 at 8:10 PM, Taylor Singletary taylorsinglet...@twitter.com wrote: Thanks! This is being looked into. I'll update when I have news. Taylor On Tuesday, December 14, 2010, Dan Checkoway dchecko...@gmail.com wrote: Yeah, you bet. Twitter4j isn't logging a timestamp when it happens, but here are a handful of timestamps for unrelated stuff that got logged no more than 10 seconds *prior* to the 4294967295 error popping out...so they're fairly close: Dec 14, 2010 12:34:11 PM PST Dec 14, 2010 1:13:07 PM PST Dec 14, 2010 1:22:48 PM PST Dec 14, 2010 1:27:22 PM PST Dec 14, 2010 1:29:48 PM PST Dec 14, 2010 1:33:36 PM PST Based on the twitter4j stack trace, I can tell you that it was *always* user.listed_count that had the funky value: Exception in thread Twitter Stream Handling Thread[Receiving stream] java.lang .NumberFormatException: For input string: 4294967295 at java.lang.NumberFormatException.forInputString(NumberFormatException. java:48) at java.lang.Integer.parseInt(Integer.java:459) at java.lang.Integer.valueOf(Integer.java:553) at twitter4j.internal.util.ParseUtil.getInt(ParseUtil.java:120) at twitter4j.UserJSONImpl.init(UserJSONImpl.java:103) Thanks, Dan On Tue, Dec 14, 2010 at 6:42 PM, Taylor Singletary taylorsinglet...@twitter.com wrote: Understandable, Dan. Can you tell me the last time an event like this happened? Taylor On Tue, Dec 14, 2010 at 3:41 PM, Dan Checkoway dchecko...@gmail.com wrote: I know this is the weenie answer, but I haven't been able to track a specific offending JSON object down yet, since it only seems to happen on the firehose, and we're using twitter4j to process that. If we were able to connect to the firehose more than once at a time, I could easily write a tool to detect and highlight the issue. Short of that, I'll try watching the sample stream for a while to see
Re: [twitter-dev] 4294967295
I just wanted to follow up on this, because the issue continues to happen, and it gets more and more interesting. We've now been seeing user.listed_count coming back as 4294967293 on occasion. So just to recap, we have now seen these values in the user.listed_count field: 4294967295 (a.k.a. unsigned -1) 4294967294 (a.k.a. unsigned -2) 4294967293 (a.k.a. unsigned -3) twitter4j has worked around this issue no problem, but I'm more than just a bit curious what these values represent. Should -1 and -2 and -3 be treated to mean anything other than we don't know what the listed count is? What happens if/when -4 starts popping out? I realize this is pretty low priority, but it's still a bug... Thanks, Dan On Fri, Dec 17, 2010 at 7:17 AM, Dan Checkoway dchecko...@gmail.com wrote: Check this out...today sometime between 4:01:43 AM PST and 4:01:53 AM PST (sorry for the ambiguity, those are our every 10 sec logging timestamps for other stuff), we saw the unsigned equivalent of -2 (4294967294) being sent by twitter in user.listed_count... Exception in thread Twitter Stream Handling Thread[Receiving stream] java.lang .NumberFormatException: For input string: 4294967294 at java.lang.NumberFormatException.forInputString(NumberFormatException. java:48) at java.lang.Integer.parseInt(Integer.java:459) at java.lang.Integer.valueOf(Integer.java:553) at twitter4j.internal.util.ParseUtil.getInt(ParseUtil.java:120) at twitter4j.UserJSONImpl.init(UserJSONImpl.java:103) at twitter4j.UserJSONImpl.init(UserJSONImpl.java:86) at twitter4j.StatusJSONImpl.init(StatusJSONImpl.java:101) at twitter4j.StatusJSONImpl.init(StatusJSONImpl.java:84) at twitter4j.StatusJSONImpl.init(StatusJSONImpl.java:118) at twitter4j.StatusJSONImpl.init(StatusJSONImpl.java:84) at twitter4j.StatusStreamImpl.handleNextElement(StatusStreamImpl.java:116) at twitter4j.StatusStreamImpl.next(StatusStreamImpl.java:89) at twitter4j.TwitterStream$StreamHandlingThread.run(TwitterStream.java:529) Any idea what's going on and/or when it might be fixed? Thanks, Dan On Tue, Dec 14, 2010 at 8:10 PM, Taylor Singletary taylorsinglet...@twitter.com wrote: Thanks! This is being looked into. I'll update when I have news. Taylor On Tuesday, December 14, 2010, Dan Checkoway dchecko...@gmail.com wrote: Yeah, you bet. Twitter4j isn't logging a timestamp when it happens, but here are a handful of timestamps for unrelated stuff that got logged no more than 10 seconds *prior* to the 4294967295 error popping out...so they're fairly close: Dec 14, 2010 12:34:11 PM PST Dec 14, 2010 1:13:07 PM PST Dec 14, 2010 1:22:48 PM PST Dec 14, 2010 1:27:22 PM PST Dec 14, 2010 1:29:48 PM PST Dec 14, 2010 1:33:36 PM PST Based on the twitter4j stack trace, I can tell you that it was *always* user.listed_count that had the funky value: Exception in thread Twitter Stream Handling Thread[Receiving stream] java.lang .NumberFormatException: For input string: 4294967295 at java.lang.NumberFormatException.forInputString(NumberFormatException. java:48) at java.lang.Integer.parseInt(Integer.java:459) at java.lang.Integer.valueOf(Integer.java:553) at twitter4j.internal.util.ParseUtil.getInt(ParseUtil.java:120) at twitter4j.UserJSONImpl.init(UserJSONImpl.java:103) Thanks, Dan On Tue, Dec 14, 2010 at 6:42 PM, Taylor Singletary taylorsinglet...@twitter.com wrote: Understandable, Dan. Can you tell me the last time an event like this happened? Taylor On Tue, Dec 14, 2010 at 3:41 PM, Dan Checkoway dchecko...@gmail.com wrote: I know this is the weenie answer, but I haven't been able to track a specific offending JSON object down yet, since it only seems to happen on the firehose, and we're using twitter4j to process that. If we were able to connect to the firehose more than once at a time, I could easily write a tool to detect and highlight the issue. Short of that, I'll try watching the sample stream for a while to see if the same issue pops up there. Will report any findings... Thanks, Dan On Tue, Dec 14, 2010 at 6:19 PM, Taylor Singletary taylorsinglet...@twitter.com wrote: Hi Dan, Do you continue to see events like this happening? Can you provide a recent example in as-provided JSON or XML? Thanks, Taylor On Tue, Dec 14, 2010 at 2:13 PM, Dan Checkoway dchecko...@gmail.com wrote: Anybody else seeing user.listed_count occasionally coming back as 4294967295? That value just happens to equate to: 1 + (2 * Integer.MAX_VALUE) Sure looks like an unsigned version of -1 to me... Anyway, it's breaking twitter4j.TwitterStream stuff. I've mentioned that separately on the twitter4j list, but I wanted to raise the issue here since the root cause is twitter sending the weird
Re: [twitter-dev] firehose exception: the end of stream has been reached
Thanks for the responses. That's unfortunately not the case. We've gone
massively concurrent with this (we've wrapped our own concurrent queueing
around twitter4j, which it looks like Yusuke has just updated to add his own
concurrent approach to this...thx), and falling behind on the stream is
definitely not the problem.
Bandwidth is not the problem.
I still don't understand why, right after reconnecting to the twitter
firehose, the stream just plain ends. It's not happening after some time of
possibly falling behind...it happens right away.
To clarify, this is NOT an issue all the time. And since I posted this the
other day, the problem has gone away as mysteriously as it arrived. But it
does happen from time to time. When everything else on my app's end is
steady-state, the flying fickle finger of blame points to twitter... :-)
Thanks,
Dan
On Fri, Dec 17, 2010 at 1:57 PM, Matt Harris thematthar...@twitter.comwrote:
Hey Dan,
If you fall too far behind when receiving the stream we will disconnect
you. Check the timestamp of the Tweets being received to the time on your
computer. If the times are drifting further apart you are falling behind.
The most common reasons for falling behind are:
1. You are attempting to process the stream in the same code that consumes
them - instead of running a queuing system.
2. Your connection is being used by other processes reducing your available
bandwidth.
As Tom suggested, run through the Streaming Documentation linked to from
http://dev.twitter.com/pages/streaming_api and make sure you implement the
suggestions.
Best,
@themattharris
Developer Advocate, Twitter
http://twitter.com/themattharris
On Fri, Dec 17, 2010 at 8:52 AM, Dan Checkoway dchecko...@gmail.comwrote:
Our app is using twitter4j 2.1.9-SNAPSHOT against the twitter firehose,
and lately we are having problems with the stream just ending out of the
blue. It happens several times a day lately with no rhyme or reason.
Here's an example of what we see:
Stream closed.TwitterException{exceptionCode=[a3652dee-000a1c7a
a3652dee-000a1c35], statusCode=-1, retryAfter=0, rateLimitStatus=null,
version=2.1.9-SNAPSHOT}
at
twitter4j.AbstractStreamImplementation.handleNextElement(AbstractStreamImplementation.java:149)
at twitter4j.StatusStreamImpl.next(StatusStreamImpl.java:74)
at
twitter4j.TwitterStream$TwitterStreamConsumer.run(TwitterStream.java:687)
Caused by: java.io.IOException: the end of the stream has been reached
at
twitter4j.AbstractStreamImplementation.handleNextElement(AbstractStreamImplementation.java:80)
... 2 more
We call cleanUp and then reconnect to the firehose, and then the same
thing happens again. Eventually we get temporarily rate limited due to the
reconnect attempts.
I'm wondering if anybody else out there has seen this issue on the
firehose, or if anybody has a suggestion on how to avoid or work around it?
Is the likely cause on twitter's side, or could something on the client side
be causing this?
Thanks,
Dan
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker:
http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker:
http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] 4294967295
Check this out...today sometime between 4:01:43 AM PST and 4:01:53 AM PST (sorry for the ambiguity, those are our every 10 sec logging timestamps for other stuff), we saw the unsigned equivalent of -2 (4294967294) being sent by twitter in user.listed_count... Exception in thread Twitter Stream Handling Thread[Receiving stream] java.lang .NumberFormatException: For input string: 4294967294 at java.lang.NumberFormatException.forInputString(NumberFormatException. java:48) at java.lang.Integer.parseInt(Integer.java:459) at java.lang.Integer.valueOf(Integer.java:553) at twitter4j.internal.util.ParseUtil.getInt(ParseUtil.java:120) at twitter4j.UserJSONImpl.init(UserJSONImpl.java:103) at twitter4j.UserJSONImpl.init(UserJSONImpl.java:86) at twitter4j.StatusJSONImpl.init(StatusJSONImpl.java:101) at twitter4j.StatusJSONImpl.init(StatusJSONImpl.java:84) at twitter4j.StatusJSONImpl.init(StatusJSONImpl.java:118) at twitter4j.StatusJSONImpl.init(StatusJSONImpl.java:84) at twitter4j.StatusStreamImpl.handleNextElement(StatusStreamImpl.java:116) at twitter4j.StatusStreamImpl.next(StatusStreamImpl.java:89) at twitter4j.TwitterStream$StreamHandlingThread.run(TwitterStream.java:529) Any idea what's going on and/or when it might be fixed? Thanks, Dan On Tue, Dec 14, 2010 at 8:10 PM, Taylor Singletary taylorsinglet...@twitter.com wrote: Thanks! This is being looked into. I'll update when I have news. Taylor On Tuesday, December 14, 2010, Dan Checkoway dchecko...@gmail.com wrote: Yeah, you bet. Twitter4j isn't logging a timestamp when it happens, but here are a handful of timestamps for unrelated stuff that got logged no more than 10 seconds *prior* to the 4294967295 error popping out...so they're fairly close: Dec 14, 2010 12:34:11 PM PST Dec 14, 2010 1:13:07 PM PST Dec 14, 2010 1:22:48 PM PST Dec 14, 2010 1:27:22 PM PST Dec 14, 2010 1:29:48 PM PST Dec 14, 2010 1:33:36 PM PST Based on the twitter4j stack trace, I can tell you that it was *always* user.listed_count that had the funky value: Exception in thread Twitter Stream Handling Thread[Receiving stream] java.lang .NumberFormatException: For input string: 4294967295 at java.lang.NumberFormatException.forInputString(NumberFormatException. java:48) at java.lang.Integer.parseInt(Integer.java:459) at java.lang.Integer.valueOf(Integer.java:553) at twitter4j.internal.util.ParseUtil.getInt(ParseUtil.java:120) at twitter4j.UserJSONImpl.init(UserJSONImpl.java:103) Thanks, Dan On Tue, Dec 14, 2010 at 6:42 PM, Taylor Singletary taylorsinglet...@twitter.com wrote: Understandable, Dan. Can you tell me the last time an event like this happened? Taylor On Tue, Dec 14, 2010 at 3:41 PM, Dan Checkoway dchecko...@gmail.com wrote: I know this is the weenie answer, but I haven't been able to track a specific offending JSON object down yet, since it only seems to happen on the firehose, and we're using twitter4j to process that. If we were able to connect to the firehose more than once at a time, I could easily write a tool to detect and highlight the issue. Short of that, I'll try watching the sample stream for a while to see if the same issue pops up there. Will report any findings... Thanks, Dan On Tue, Dec 14, 2010 at 6:19 PM, Taylor Singletary taylorsinglet...@twitter.com wrote: Hi Dan, Do you continue to see events like this happening? Can you provide a recent example in as-provided JSON or XML? Thanks, Taylor On Tue, Dec 14, 2010 at 2:13 PM, Dan Checkoway dchecko...@gmail.com wrote: Anybody else seeing user.listed_count occasionally coming back as 4294967295? That value just happens to equate to: 1 + (2 * Integer.MAX_VALUE) Sure looks like an unsigned version of -1 to me... Anyway, it's breaking twitter4j.TwitterStream stuff. I've mentioned that separately on the twitter4j list, but I wanted to raise the issue here since the root cause is twitter sending the weird value. Thanks, Dan -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi
[twitter-dev] firehose exception: the end of stream has been reached
Our app is using twitter4j 2.1.9-SNAPSHOT against the twitter firehose, and
lately we are having problems with the stream just ending out of the
blue. It happens several times a day lately with no rhyme or reason.
Here's an example of what we see:
Stream closed.TwitterException{exceptionCode=[a3652dee-000a1c7a
a3652dee-000a1c35], statusCode=-1, retryAfter=0, rateLimitStatus=null,
version=2.1.9-SNAPSHOT}
at
twitter4j.AbstractStreamImplementation.handleNextElement(AbstractStreamImplementation.java:149)
at twitter4j.StatusStreamImpl.next(StatusStreamImpl.java:74)
at
twitter4j.TwitterStream$TwitterStreamConsumer.run(TwitterStream.java:687)
Caused by: java.io.IOException: the end of the stream has been reached
at
twitter4j.AbstractStreamImplementation.handleNextElement(AbstractStreamImplementation.java:80)
... 2 more
We call cleanUp and then reconnect to the firehose, and then the same thing
happens again. Eventually we get temporarily rate limited due to the
reconnect attempts.
I'm wondering if anybody else out there has seen this issue on the firehose,
or if anybody has a suggestion on how to avoid or work around it? Is the
likely cause on twitter's side, or could something on the client side be
causing this?
Thanks,
Dan
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
[twitter-dev] 4294967295
Anybody else seeing user.listed_count occasionally coming back as 4294967295? That value just happens to equate to: 1 + (2 * Integer.MAX_VALUE) Sure looks like an unsigned version of -1 to me... Anyway, it's breaking twitter4j.TwitterStream stuff. I've mentioned that separately on the twitter4j list, but I wanted to raise the issue here since the root cause is twitter sending the weird value. Thanks, Dan -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] 4294967295
Yeah, you bet. Twitter4j isn't logging a timestamp when it happens, but here are a handful of timestamps for unrelated stuff that got logged no more than 10 seconds *prior* to the 4294967295 error popping out...so they're fairly close: Dec 14, 2010 12:34:11 PM PST Dec 14, 2010 1:13:07 PM PST Dec 14, 2010 1:22:48 PM PST Dec 14, 2010 1:27:22 PM PST Dec 14, 2010 1:29:48 PM PST Dec 14, 2010 1:33:36 PM PST Based on the twitter4j stack trace, I can tell you that it was *always* user.listed_count that had the funky value: Exception in thread Twitter Stream Handling Thread[Receiving stream] java.lang .NumberFormatException: For input string: 4294967295 at java.lang.NumberFormatException.forInputString(NumberFormatException. java:48) at java.lang.Integer.parseInt(Integer.java:459) at java.lang.Integer.valueOf(Integer.java:553) at twitter4j.internal.util.ParseUtil.getInt(ParseUtil.java:120) at twitter4j.UserJSONImpl.init(UserJSONImpl.java:103) Thanks, Dan On Tue, Dec 14, 2010 at 6:42 PM, Taylor Singletary taylorsinglet...@twitter.com wrote: Understandable, Dan. Can you tell me the last time an event like this happened? Taylor On Tue, Dec 14, 2010 at 3:41 PM, Dan Checkoway dchecko...@gmail.com wrote: I know this is the weenie answer, but I haven't been able to track a specific offending JSON object down yet, since it only seems to happen on the firehose, and we're using twitter4j to process that. If we were able to connect to the firehose more than once at a time, I could easily write a tool to detect and highlight the issue. Short of that, I'll try watching the sample stream for a while to see if the same issue pops up there. Will report any findings... Thanks, Dan On Tue, Dec 14, 2010 at 6:19 PM, Taylor Singletary taylorsinglet...@twitter.com wrote: Hi Dan, Do you continue to see events like this happening? Can you provide a recent example in as-provided JSON or XML? Thanks, Taylor On Tue, Dec 14, 2010 at 2:13 PM, Dan Checkoway dchecko...@gmail.com wrote: Anybody else seeing user.listed_count occasionally coming back as 4294967295? That value just happens to equate to: 1 + (2 * Integer.MAX_VALUE) Sure looks like an unsigned version of -1 to me... Anyway, it's breaking twitter4j.TwitterStream stuff. I've mentioned that separately on the twitter4j list, but I wanted to raise the issue here since the root cause is twitter sending the weird value. Thanks, Dan -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
Re: [twitter-dev] Re: Snowflake: An update and some very important information
I'm also patiently awaiting a response from twitter about this. Are the ids
sane for 64-bit *signed* long?
Dan
On Mon, Oct 18, 2010 at 9:08 PM, jon jonhoff...@gmail.com wrote:
Hi,
You wrote that the IDs are unsigned 64 bit ints, but the IdWorker is
pumping out java Longs which are signed. I'm assuming that was a
typo, but please clarify.
http://github.com/twitter/snowflake/blob/master/src/main/scala/com/twitter/service/snowflake/IdWorker.scala
Thanks,
- Jon
On Oct 18, 8:19 pm, Matt Harris thematthar...@twitter.com wrote:
Last week you may remember Twitter planned to enable the new Status ID
generator - 'Snowflake' but didn't. The purpose of this email is to
explain
the reason why this didn't happen, what we are doing about it, and what
the
new release plan is.
So what is Snowflake?
--
Snowflake is a service we will be using to generate unique Tweet IDs.
These
Tweet IDs are unique 64bit unsigned integers, which, instead of being
sequential like the current IDs, are based on time. The full ID is
composed
of a timestamp, a worker number, and a sequence number.
The problem
-
Before launch it came to our attention that some programming languages
such
as Javascript cannot support numbers with 53bits. This can be easily
examined by running a command similar to: (90071992547409921).toString()
in
your browsers console or by running the following JSON snippet through
your
JSON parser.
{id: 10765432100123456789, id_str: 10765432100123456789}
In affected JSON parsers the ID will not be converted successfully and
will
lose accuracy. In some parsers there may even be an exception.
The solution
To allow javascript and JSON parsers to read the IDs we need to include a
string version of any ID when responding in the JSON format. What this
means
is Status, User, Direct Message and Saved Search IDs in the Twitter API
will
now be returned as an integer and a string in JSON responses. This will
apply to the main Twitter API, the Streaming API and the Search API.
For example, a status object will now contain an id and an id_str. The
following JSON representation of a status object shows the two versions
of
the ID fields for each data point.
[
{
coordinates: null,
truncated: false,
created_at: Thu Oct 14 22:20:15 + 2010,
favorited: false,
entities: {
urls: [
],
hashtags: [
],
user_mentions: [
{
name: Matt Harris,
id: 777925,
id_str: 777925,
indices: [
0,
14
],
screen_name: themattharris
}
]
},
text: @themattharris hey how are things?,
annotations: null,
contributors: [
{
id: 819797,
id_str: 819797,
screen_name: episod
}
],
id: 12738165059,
id_str: 12738165059,
retweet_count: 0,
geo: null,
retweeted: false,
in_reply_to_user_id: 777925,
in_reply_to_user_id_str: 777925,
in_reply_to_screen_name: themattharris,
user: {
id: 6253282
id_str: 6253282
},
source: web,
place: null,
in_reply_to_status_id: 12738040524
in_reply_to_status_id_str: 12738040524
}
]
What should you do - RIGHT NOW
--
The first thing you should do is attempt to decode the JSON snippet above
using your production code parser. Observe the output to confirm the ID
has
not lost accuracy.
What you do next depends on what happens:
* If your code converts the ID successfully without losing accuracy you
are
OK but should consider converting to the _str versions of IDs as soon as
possible.
* If your code has lost accuracy, convert your code to using the _str
version immediately. If you do not do this your code will be unable to
interact with the Twitter API reliably.
* In some language parsers, the JSON may throw an exception when reading
the
ID value. If this happens in your parser you will need to ‘pre-parse’ the
data, removing or replacing ID parameters with their _str versions.
Summary
-
1) If you develop in Javascript, know that you will have to update your
code
to read the string version instead of the integer version.
2) If you use a JSON decoder, validate that the example JSON, above,
decodes
without throwing exceptions. If exceptions are thrown, you will need to
pre-parse the data. Please let us know the name, version, and language of
the parser which throws the exception so we can investigate.
Timeline
---
by 22nd October 2010 (Friday): String versions of ID numbers will start
appearing in the API responses
4th November 2010 (Thursday) : Snowflake will be turned on but at ~41bit
[twitter-dev] GET list memberships paging is broken?
When using the GET list memberships API ( http://api.twitter.com/1/twitterapi/lists/memberships.*), it looks like paging is broken. If the user is a member of 20 lists, you can never see anything beyond the first 20. I'm passing cursor=-1 (well, twitter4j is) on the first request, and I get back the first page of 20 lists, which is fine...but no matter what, I always get back: next_cursor:0, previous_cursor:0, next_cursor_str:0, previous_cursor_str:0 ...which prevents any paging beyond that first page of 20. This is the case no matter which user I've tried. What seems coincidental is that even on the twitter web site proper, only the first page of 20 is presented as well, with no way to page beyond that...for example: http://twitter.com/GamePro/lists/memberships I'm not sure if that's a related issue, or an intentional thing that has also affected the API, or what. Anyway, can twitter please fix paging on the GET list memberships API? Thanks, Dan Checkoway -- Subscription settings: http://groups.google.com/group/twitter-development-talk/subscribe?hl=en
Re: [twitter-dev] Re: Mad about lists and cursors... please help
+1 on needing this fix. Sorry for the duplicate report of this issue I slapped in another thread this morning. Thanks, Dan On Sat, Apr 17, 2010 at 12:04 PM, Mark McBride mmcbr...@twitter.com wrote: Yes. A fix has been identified, and should be deployed in a few days Sent from mobile device On Apr 17, 2010, at 7:08 AM, Zach zcox...@gmail.com wrote: It's 10 days later and next_cursor on http://apiwiki.twitter.com/Twitter-REST-API-Method%3A-GET-list-memberships is still always 0, even when the user is being followed by far more than 20 lists. This is completely broken and prevents 3rd party apps from discovering all lists that follow a given user. Has anyone at Twitter even looked into this? On Apr 7, 3:43 pm, Mark McBride mmcbr...@twitter.com wrote: Eugene, we're aware of the issue and will take a look at it today. ---Mark http://twitter.com/mccv On Wed, Apr 7, 2010 at 11:09 AM, eugene.man...@gmail.com eugene.man...@gmail.com wrote: I posted this issue to @twitterapi twice, but they ignored it. Dear API group, please address this question. Thank you! On Apr 6, 9:45 am, Spraycode joey.fernan...@gmail.com wrote: Has anyone been able to solve this issue? This is still crippling us. Thanks! On Apr 2, 5:25 am, luisfigo rsoeg...@gmail.com wrote: Having the same problem... Triedhttp://api.twitter.com/1/avinashkaushik/lists/memberships.xml and get 0 for cursor. This guy is followed by ton of lists in fact Below is the snapshot of the end result I got... This is screwing up our app right now... . profile_background_image_url http://a1.twimg.com/profile_background_images/79104366/twitter_backgr. .. /profile_background_image_url profile_background_tilefalse/profile_background_tile notificationsfalse/notifications geo_enabledfalse/geo_enabled verifiedfalse/verified followingfalse/following statuses_count3208/statuses_count langen/lang contributors_enabledfalse/contributors_enabled /user /list /lists next_cursor0/next_cursor previous_cursor0/previous_cursor /lists_list On Apr 1, 6:00 pm, Diego Rin Martin diego@gmail.com wrote: I think it's a API bug, even in the twitter page the paginator doesn't work as expected, sometimes appears, sometines not, and when appears it makes in a random manner. i'm getting cursor 0 from API, using int or string representation, the bug is in the API that sends the cursor 0 randomly. regards, diego. On Thu, Apr 1, 2010 at 2:38 AM, jmathai jmat...@gmail.com wrote: Are you sure you're using the string representation of the cursor instead of the int? The API's cursor exceeds PHP's max integer value (generally). jmathai ~ $ php -r '$x = json_decode(1); echo $x; echo \n; var_dump($x===1); var_dump($x===1.111E+52);' 1.111E+52 bool(false) bool(true) jmathai ~ $ php -r '$x = 1; echo $x; echo \n; var_dump($x===1); var_dump($x===1.111E+52);' 1.111E+52 bool(true) bool(false) On Mar 31, 2:03 am, Diego Rin Martín diego@gmail.com wrote: Hi there, this is my first post to this group, i'm a spanish developer dealing with twitter api surprises, excuse my poor english, i'will do my best to comunicate nicest. So, to the problem, I'm trying to retrieve the lists for a user, via list/membershipsget method, and passing cursor as parameter, I'm having got random results, I explain myself, sometimes I made a request (for user edans, that have a huge amount of pages to paginate) and I get one page, I pass cursor -1 and I get cursor 0, sometimes I get one page, I pass cursor -1 i get cursor 1331431515904087602, then I pass it and I get 0, sometimes I get a random number of pages, but never, never, be able to retrieve the total amount of pages. I use php twitter-async classes to comunicate with API, I thought that it could be the cause of the problem, but using direct curl (via php5- curl extension) calls I'm having the same issues. Same using json or xml. I'm always getting 200 responses, so the call finish in a correct way. any clue? I'm turning mad. Thanks in advance. diego. -- To unsubscribe, reply using remove me as the subject.
Re: [twitter-dev] Re: Basic Auth Deprecation
Dean, Basic auth sends the password essentially in the clear (encoded, but trivially so). It's really in everybody's best interest NOT to use basic auth, but to switch to something less sniffable/repeatable. For example, here's a sample credential you're passing to twitter (HTTP header) when you use basic auth: *Authorization: Basic bm9ib2R5OndoYXRldmVy * Go to http://ostermiller.org/calc/encode.html, paste bm9ib2R5OndoYXRldmVy into the box and click Decode next to Base 64. Now you have my password. Sniff some HTTP requests and you've got a whole lot of passwords. Completely non-secure. I'm not even sure why basic auth ever gained any sort of acceptance. Switching off basic auth and onto something like OAuth, any of your users who value their privacy will thank you! 1 for deprecating basic auth. Dan On Wed, Apr 14, 2010 at 9:28 AM, Dean Collins d...@cognation.net wrote: So basically you are saying Twitter wants a chokehold to block apps they don’t like which you don’t currently have with basic auth. Considering your recent purchase of a twitter client is that really a message you want to be spreading at the moment? How about leaving it up to end users to make the decision about which clients they do and don’t use to access twitter. Restricting all clients to oauth only is hardly going to give developers warm and fuzzy feelings that with a single keystroke a client can be banned instantly across the entire ecosystem. Or am I missing something? Cheers, Dean -- *From:* twitter-development-talk@googlegroups.com [mailto: twitter-development-t...@googlegroups.com] *On Behalf Of *Raffi Krikorian *Sent:* Wednesday, April 14, 2010 8:59 AM *To:* twitter-development-talk@googlegroups.com *Subject:* Re: [twitter-dev] Re: Basic Auth Deprecation in my ideal world, nobody would have access to a user's password except twitter.com -- oauth provides a framework so end applications are not storing the actual password. people are notoriously bad with using the same password on lots of different sites. additionally, oauth provides twitter better visibility into the traffic coming into our system, so we can better shape traffic needs, we can provide auditing back to users on which applications are doing what actions on their behalf, etc. On Wed, Apr 14, 2010 at 5:39 AM, Dean #39;at#39; Cognation dot Net d...@cognation.net wrote: But why is oauth better than basic for a desktop client? i understand it for the webapps but on a desktop client whats the point? Basically you are saying the desktop end user cant be trusted? Sorry but that doesn't make any sense. Please explain. Cheers, Dean On Apr 14, 1:15 am, Taylor Singletary taylorsinglet...@twitter.com wrote: Basic auto being turned off means just that.. Desktop clients can implement xAuth as an alternative, where you do a one-time exchange of login and password for an OAuth access token and continue from there signing your requests and doing things in the OAuth way. You'd no longer, as a best practice and one that I would stress in the upmost even on a desktop client, store the login and password beyond the xAuth access token negotiation step. If the token were revoked you would then query for the login and password again and so on and so on and also and also. Obtaining permission to use xAuth for desktop clients is as easy as sending a well-identified and verbose note to a...@twitter.com. Basic auth had a good run. It's nearly time to say goodnight. Taylor On Tuesday, April 13, 2010, Dean Collins d...@cognation.net wrote: Just so I understand this, applications running on the desktop will still work correct? Basic functionality is only being turned off for web apps correct? It's not like desktop apps will have to start using oauth. Cheers, Dean -Original Message- From: twitter-development-talk@googlegroups.com [mailto: twitter-development-t...@googlegroups.com] On Behalf Of Dewald Pretorius Sent: Tuesday, April 13, 2010 7:31 PM To: Twitter Development Talk Subject: [twitter-dev] Re: Basic Auth Deprecation Could you please announce the hard turn off date somewhere on one of your Twitter blogs about a month ahead of time, so that we all have an official source to point our users to when we explain to them why we're converting everything over to OAuth? On Apr 13, 8:19 pm, Raffi Krikorian ra...@twitter.com wrote: we have announced deprecation, and will hard turn off basic authentication in june. the exact date has not been set, but i presume it will be later in the month. Is Basic Auth going to be deprecated (as in hard switched-off) in June, or are you in June going to announce depracation, with the hard switch-off then coming a few months later? -- Raffi Krikorian Twitter Platform Teamhttp://twitter.com/raffi
