Re: [DISCUSS] Json libraries for Kafka

2015-07-15 Thread Ismael Juma
On Mon, Jul 13, 2015 at 10:05 PM, Ismael Juma ism...@juma.me.uk wrote: * It's not thread-safe (so external locks are needed to use it in a concurrent environment) I realise now that I didn't make the impact of this very clear. The current code is acquiring a JVM-wide lock for the duration of

Re: [DISCUSS] Json libraries for Kafka

2015-07-14 Thread Joe Stein
Fasterxml/Jackson +1 to that. The scala databinds to case classes are gr8. ~ Joestein On Jul 14, 2015 5:42 PM, Ewen Cheslack-Postava e...@confluent.io wrote: Currently the clients/server mismatch wouldn't be an issue since there are no client-side uses of JSON, right? That said, if Copycat

Re: [DISCUSS] Json libraries for Kafka

2015-07-14 Thread Joe Stein
Maybe after the existing scala clients are deprecated. ~ Joestein On Jul 14, 2015 6:04 PM, Jay Kreps j...@confluent.io wrote: Is this going to become a dependency for core and then transitively for the old clients? The current json library is definitely not great, but it does parse json and

Re: [DISCUSS] Json libraries for Kafka

2015-07-14 Thread Ewen Cheslack-Postava
Currently the clients/server mismatch wouldn't be an issue since there are no client-side uses of JSON, right? That said, if Copycat ends up included in Kafka we'll need to provide at least one serializer which would be written in Java and I suspect some people would like JSON to be a candidate

Re: [DISCUSS] Json libraries for Kafka

2015-07-14 Thread Jay Kreps
Is this going to become a dependency for core and then transitively for the old clients? The current json library is definitely not great, but it does parse json and it's not used in any context where performance is a concern. Because the older clients aren't well modularized, adding core

Re: [DISCUSS] Json libraries for Kafka

2015-07-14 Thread Geoffrey Anderson
Hi all, my pull request here: https://github.com/apache/kafka/pull/70/files#diff-59f3fe36571d1eee9f923df927a643eb would introduce a client-side dependency on the json-simple package. It is only used in a tool (VerifiableProducer.java), but it sounds like I should probably use Jackson instead?

Re: [DISCUSS] Json libraries for Kafka

2015-07-14 Thread Ismael Juma
On Tue, Jul 14, 2015 at 10:46 PM, Joe Stein joe.st...@stealth.ly wrote: Fasterxml/Jackson +1 to that. The scala databinds to case classes are gr8. To be clear, case classes support would require the Scala module for Jackson and the Scala versions headache that goes with it (2.9 support is

Re: [DISCUSS] Json libraries for Kafka

2015-07-14 Thread Ismael Juma
Hi Jay, Comments inline. On Tue, Jul 14, 2015 at 11:04 PM, Jay Kreps j...@confluent.io wrote: Is this going to become a dependency for core and then transitively for the old clients? That's right. The current json library is definitely not great, but it does parse json and it's not used

Re: [DISCUSS] Json libraries for Kafka

2015-07-14 Thread Jay Kreps
Yeah we really can't add any dependencies for that client jar. Maybe we could move the perf test to the tools jar though? -Jay On Tue, Jul 14, 2015 at 6:01 PM, Geoffrey Anderson ge...@confluent.io wrote: Hi all, my pull request here:

Re: [DISCUSS] Json libraries for Kafka

2015-07-14 Thread Ismael Juma
Ewen, On Tue, Jul 14, 2015 at 10:41 PM, Ewen Cheslack-Postava e...@confluent.io wrote: Currently the clients/server mismatch wouldn't be an issue since there are no client-side uses of JSON, right? That said, if Copycat ends up included in Kafka we'll need to provide at least one serializer

Re: [DISCUSS] Json libraries for Kafka

2015-07-14 Thread Jay Kreps
Ah, makes sense. Yes that addresses my concerns. -Jay On Tue, Jul 14, 2015 at 5:19 PM, Ismael Juma ism...@juma.me.uk wrote: Hi Jay, Comments inline. On Tue, Jul 14, 2015 at 11:04 PM, Jay Kreps j...@confluent.io wrote: Is this going to become a dependency for core and then transitively

[DISCUSS] Json libraries for Kafka

2015-07-13 Thread Ismael Juma
Hi all, Kafka currently use scala.util.parsing.json.JSON as its json parser and it has a number of issues: * It encourages unsafe casts (returns `Option[Any]`) * It's slow (it relies on parser combinators under the hood) * It's not thread-safe (so external locks are needed to use it in a