I asked because the IBM guys have some code to be released soon that will address some of these issues.
In particular, Kevin Breyer was at the HUG and we talked a bit about their format. Some important characteristics are: - it is pretty much a binary JSON implementation - value parsing doesn't happen because it is a binary format. That means that integers, doubles and dates can be read much more quickly. - names are repeated in the data just as in JSON, but are easily compressed away with any compression technique you care to use. - he says it is fast, but I haven't tried it yet. So this will address the first point very directly, will indirectly address the second point, and the third point may be FUD. Relative to thrift, JSON has the advantage of not requiring a schema as well as the disadvantage of not having a schema. The advantage is that the data is more fluid and I don't have to generate code to handle the records. The disadvantage is that I lose some data completeness and typing guarantees. On balance, I would like to use JSON-like data quite a bit in ad hoc data streams and in logs where the producer and consumer of the data are not visible to parts of the data processing chain. I am very excited to see the IBM code. On 5/23/08 6:37 AM, "Stuart Sierra" <[EMAIL PROTECTED]> wrote: > On 5/22/08 1:54 PM, "Stuart Sierra" <[EMAIL PROTECTED]> wrote: >> I've tried using JSON to store structured data in TextOutputFormat, >> which works but is not very efficient. Any better suggestions? > > On Thu, May 22, 2008 at 5:21 PM, Ted Dunning <[EMAIL PROTECTED]> wrote: >> What is it that makes you not think JSON has to be inefficient? >> ? Repeated value parsing ? >> ? Redundant redundant data labels ? >> ? Generic "parsing must be slow" prejudice ? > > All three :) but it's not a strong prejudice. I was just wondering if > there's a better way. Is it worthwhile, for example, to write my own > record class that implements Writable? > > -Stuart
