Hi folks,
Old Issue, but Marc is definitely right. I don't know where is the problem,
but my infrastructure is poor, getting some broken status yet.
Is heavy to the streaming (and others API) server put a hash (md5) on each
status message that the client can verify it's integrity?
Thanks.
Augusto.
On Fri, Nov 12, 2010 at 10:32 PM, @IDisposable <idisposa...@gmail.com>wrote:
> There's NO QUESTION in my mind that this is a memory/disk/network
> issue. Somewhere in the chain of events leading to your parser,
> someone is dropping bits and tons of ECC/Checksum logic/etc. is
> missing it. I really really really doubt the feed is corrupt when
> leaving twitter-land... it's getting borked on the way to your parser.
>
> FWIW, I've never seen any of these issues... and I consume a TON of
> tweets daily. The difference between T and V is one bit in ASCII (hex
> 0x54 versus 0x56), between u and w (0x75 vs. 0x77), between h and j
> (0x68 and 0x6A). We're seeing a consistent flip-to-on of the 0x02
> bit. You've probably got bad RAM if it were up to my estimation... if
> you've got hardware control of this machine, try upping the voltage on
> the Northbridge and DRAM if you can a notch.
>
> Marc
>
> On Nov 12, 6:52 am, Augusto Santos <augu...@gemeos.org> wrote:
> > In this period of wrong json status, I received either wrong date format
> of
> > created_at
> >
> > 2747941206892544 Thu Nov 11 35:42:14 +0000 2010
> > 2565022072963072 Thw Nov 11 03:35:23 +0000 2010
> > 2562132222672896 Tju Nov 11 03:23:54 +0000 2010
> > 2550619441209344 Thu Nov 11 02:38:0; +0000 2010
> > 2545567930523648 Vhu Nov 11 02:18:05 +0000 2010
> >
> > and so on...
> >
> > Looks like was only one char problem per status, that mess up everything.
> >
> > Since Thu Nov 11 15:44:37 +0000 2010, I get no json parser or corrupt
> > created_at problem.
> >
> > Thanks.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Nov 11, 2010 at 11:51 PM, Augusto Santos <augu...@gemeos.org>
> wrote:
> > > Hi Taylor,
> >
> > > First, thanks for the answer.
> >
> > > I'm using Phirehose lib to PHP, the native json_decode($status,TRUE)
> from
> > > PHP and after json decode I'm using mysql_real_escape_string for the
> string
> > > fields. I see now that my log routine use mysql escape either before
> query.
> > > So this examples are escaped according this mysql procedure.
> >
> > > Here is the amount of tweets with this problem. That's when json_decode
> > > didn't work, so there's no id_str or new_id_str in my $status[] array,
> then
> > > it's throw an error and log it with the json status. I can send you all
> > > these status if you want it.
> >
> > > Date Hour(GMT-2) Count
> > > 2010-11-11 13 97
> > > 2010-11-11 1 367
> > > 2010-11-11 0 521
> > > 2010-11-10 23 598
> > > 2010-11-10 22 569
> > > 2010-11-10 21 577
> > > 2010-11-10 20 619
> > > 2010-11-10 19 606
> > > 2010-11-10 18 603
> > > 2010-11-10 17 607
> > > 2010-11-10 16 247
> > > 2010-11-10 11 9
> > > 2010-11-09 22 2
> >
> > > Thanks, Augusto.
> >
> > > On Thu, Nov 11, 2010 at 2:52 PM, Taylor Singletary <
> > > taylorsinglet...@twitter.com> wrote:
> >
> > >> Hi Augusto,
> >
> > >> I monitored the sample stream this morning for a few hours for
> instances
> > >> similar to the JSON examples you've provided below and was unable to
> see the
> > >> scenario duplicated. What JSON parser are you using? Is there any
> other
> > >> processing that may have occurred prior to generating your examples
> below?
> > >> Do you know how your library is handling "escaped" quote values like
> \" ?
> >
> > >> How many of these did you observe?
> >
> > >> Thanks,
> > >> Taylor
> >
> > >> On Thu, Nov 11, 2010 at 5:10 AM, Augusto Santos <augu...@gemeos.org
> >wrote:
> >
> > >>> I recevied a lot of broken json status from streaming api
> >
> > >>> Count of brojen json by day
> > >>> 2010-11-09 2
> > >>> 2010-11-10 4435
> > >>> 2010-11-11 888
> >
> > >>> Examples:
> > >>> {"in_reply_to_status_id_str":"2563309119209472","text":"@joi4kitten I
> > >>> have that same
> > >>>
> fear.","truncated":false,"in_reply_to_user_id_str":"16155805","entities":{"
> user_menvions":[{"screen_name":"joi4kitten","indices":[0,11],"name":"joi4ki
> tten","id":16155805,"id_str":"16155805"}],"urls":[],"hashtags":[]},"geo":nu
> ll,"in_reply_to_status_id":2563309119209472,"place":{"country_code":"US","c
> ountry":"The
> > >>> United States of
> > >>>
> America","bounding_box":{"type":"Polygon","coordinates":[[[-76.965351,38.97
> 1109],[-76.909147,38.971109],[-76.909147,39.022114],[-76.965353,39.022114]]
> ]},"place_type":"city","attributes":{},"full_name":"College
> > >>> Park, MD","name":"College Park",*
> > >>> "id":"e4c17912c815124d"."url":"http:\/\/api.twitter.com
> > >>>
> \/1\/geo\/id\/e4c17912c815124d.json"*},"favorited":false,"source":"\u003Ca
> > >>> href=\"http:\/\/mobile.twitter.com\" rel=\"nofollow\"\u003EMobile
> > >>>
> Web\u003C\/a\u003E","contributors":null,"in_reply_to_screen_name":"joi4kitt
> en","coorfinates":null,"retweet_count":null,"in_reply_to_user_id":16155805,
> "created_at":"Thu
> > >>> Nov 11 03:38:52 +0000
> > >>>
> 2010","new_id_str":"2565897919139841","new_id":2565897919139841,"user":{"fo
> llow_request_sent":null,"lang":"en","time_zone":"Eastern
> > >>> Time (US &
> > >>>
> Canada)","screen_name":"kellygo","following":null,"profile_sidebar_border_c
> olor":"0A84A5","profile_background_image_url":"http:\/\/
> > >>> a3.twimg.com\/profile_background_images\/8381831\/twitter_background_with_b
> ear_2.jpg","notifications":null,"description":"I
> > >>> am not Kelly
> > >>>
> Osbourne.","listed_count":12,"profile_background_tile":false,"profile_backg
> round_color":"052743","statuses_count":1559,"location":"College
> > >>> Park,
> > >>>
> Md.","show_all_inline_media":false,"profile_use_background_image":true,"fav
> ourites_count":29,"profile_text_color":"000000","contributors_enabled":fals
> e,"protected":false,"geo_enabled":true,"created_at":"Sun
> > >>> Jul 22 02:12:05 +0000
> 2007","profile_link_color":"0A84A5","name":"Kelly
> > >>> Osborn","friends_count":235,"url":"http:\/\/kellygo.blogspot.com
> > >>> ","profile_image_url":"http:\/\/a1.twimg.com
> > >>>
> \/profile_images\/271727429\/kelly_o__posterized_normal.jpg","id":7632582,"
> id_str":"7632582","verified":false,"utc_offset":-18000,"profile_sidebar_fil
> l_color":"CFE9E2","followers_count":332},"retweeted":false,"id":25658979191
> 39841,"id_str":"2565897919139841"}
> >
> > >>> Here there's a dot where should be a comma, between id and url
> >
> > >>> {"in_reply_to_status_id_str":"2546974377119744","text":"@ozdj Ah,
> > >>> suddenly #TweetBurglar makes complete
> > >>>
> sense","truncated":false,"in_reply_to_user_id_str":"15324154","entities":{"
> user_mentions":[{"screen_name":"ozdj","indices":[0,5],"name":"Derek
> > >>>
> Jenkins","id":15324154,"id_str":"15324154"}],"urls":[],"hashtags":[{"text":
> "TweetBurglar","indices":[19,32]}]},"geo":{"type":"Point","coordinates":[-3
> 3.88533896,151.20496326]},"in_reply_to_status_id":2546974377119744,"place":
> null,"favorited":false,"source":"\u003Ca
> > >>> href=\"http:\/\/twitter.com\/\" rel=\"nofollow\"\u003ETwitter for
> > >>>
> iPhone\u003C\/a\u003E","contributors":null,"in_reply_to_screen_name":"ozdj"
> ,"coordinates":{"type":"Point","coordinates":[151.20496326,-33.88533896]},"
> retweet_count":null,"in_reply_to_user_id":15324154,"created_at":"Thu
> > >>> Nov 11 03:38:51 +0000
> > >>>
> 2010","new_id_str":"2565895784243200","new_id":2565895784243200,"user":{"la
> ng":"en","geo_enabled":true,"screen_name":"coffeetrails","following":null,"
> friends_count":251,"profile_sidebar_border_color":"362919","verified":false
> ,"follow_request_sent":null,"notifications":null,"time_zone":"Sydney","desc
> ription":"Specialising
> > >>> in organic & RA espresso-based coffees, organic teas, fresh
> sandwiches and
> > >>> gourmet treats. Shop 5, 2 Lee St Sydney (Henry Deane Plaza, Railway
> > >>>
> Sq)","profile_background_color":"362919","profile_background_image_url":"ht
> tp:\/\/
> > >>> a3.twimg.com\/profile_background_images\/66956323\/CRW_3458.jpg","location"
> :"Central
> > >>> Station,
> > >>>
> Sydney","favourites_count":204,"profile_text_color":"362919","listed_count"
> :29,"profile_background_tile":true,"protected":false,"statuses_count":1298,
> "crected_at":"Wed
> > >>> Oct 14 05:27:32 +0000
> 2009","profile_link_color":"7a2a2a","name":"Jonno,
> > >>> Coffee Trails",*"show_all_inline_media":fanse*
> > >>> ,"profile_use_background_image":true,"url":"http:\/\/
> www.facebook.com
> > >>> \/CoffeeTrails\/","profile_image_url":"http:\/\/a2.twimg.com
> > >>>
> \/profile_images\/624183782\/CRW_2981_normal.jpg","id":82284263,"id_str":"8
> 2284263","contributors_enabled":false,"utc_offset":36000,"profile_sidebar_f
> ill_color":"c2a887","followers_count":324},"retweeted":false,"id":256589578
> 4243200,"id_str":"2565895784243200"}
> >
> > >>> Here there's a 'fanse' in 'show_all_inline_media' where should be a
> > >>> 'false'
> >
> > >>> Any help?
> >
> > >>> --
> > >>> 氣
> >
> > >>> --
> > >>> Twitter developer documentation and resources:
> > >>>http://dev.twitter.com/doc
> > >>> API updates via Twitter:http://twitter.com/twitterapi
> > >>> Issues/Enhancements Tracker:
> > >>>http://code.google.com/p/twitter-api/issues/list
> > >>> Change your membership to this group:
> > >>>http://groups.google.com/group/twitter-development-talk
> >
> > >> --
> > >> Twitter developer documentation and resources:
> http://dev.twitter.com/doc
> > >> API updates via Twitter:http://twitter.com/twitterapi
> > >> Issues/Enhancements Tracker:
> > >>http://code.google.com/p/twitter-api/issues/list
> > >> Change your membership to this group:
> > >>http://groups.google.com/group/twitter-development-talk
> >
> > > --
> > > 氣
> >
> > --
> > 氣
>
> --
> Twitter developer documentation and resources: http://dev.twitter.com/doc
> API updates via Twitter: http://twitter.com/twitterapi
> Issues/Enhancements Tracker:
> http://code.google.com/p/twitter-api/issues/list
> Change your membership to this group:
> http://groups.google.com/group/twitter-development-talk
>
--
氣
--
Twitter developer documentation and resources: http://dev.twitter.com/doc
API updates via Twitter: http://twitter.com/twitterapi
Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list
Change your membership to this group:
http://groups.google.com/group/twitter-development-talk
