Is the new producer API going to maintain protocol compatibility with old
version if the API under the hood?
> On Jan 29, 2014, at 10:15, Jay Kreps <jay.kr...@gmail.com> wrote:
>
> The challenge of directly exposing ProduceRequestResult is that the offset
> provided is just the base offset and there is no way to know for a
> particular message where it was in relation to that base offset because the
> batching is transparent and non-deterministic. So I think we do need some
> kind of per-message result.
>
> I started with Future<RequestResult>, I think for the same reason you
> prefer it but then when I actually looked at some code samples it wasn't
> too great--checked exceptions, methods that we can't easily implement, etc.
> I moved away from that for two reasons:
> 1. When I actually wrote out some code samples of usage they were a little
> ugly for the reasons I described--checked exceptions, methods we can't
> implement, no helper methods, etc.
> 2. I originally intended to make the result send work like a
> ListenableFuture so that you would register the callback on the result
> rather than as part of the call. I moved away from this primarily because
> the implementation complexity was a little higher.
>
> Whether or not the code prettiness on its own outweighs the familiarity of
> a normal Future I don't know, but that was the evolution of my thinking.
>
> -Jay
>
>
>> On Wed, Jan 29, 2014 at 10:06 AM, Jay Kreps <jay.kr...@gmail.com> wrote:
>>
>> Hey Neha,
>>
>> Error handling in RecordSend works as in Future you will get the exception
>> if there is one from any of the accessor methods or await().
>>
>> The purpose of hasError was that you can write things slightly more simply
>> (which some people expressed preference for):
>> if(send.hasError())
>> // do something
>> long offset = send.offset();
>>
>> Instead of the more the slightly longer:
>> try {
>> long offset = send.offset();
>> } catch (KafkaException e) {
>> // do something
>> }
>>
>>
>> On Wed, Jan 29, 2014 at 10:01 AM, Neha Narkhede
>> <neha.narkh...@gmail.com>wrote:
>>
>>> Regarding the use of Futures -
>>>
>>> Agree that there are some downsides to using Futures but both approaches
>>> have some tradeoffs.
>>>
>>> - Standardization and usability
>>> Future is a widely used and understood Java API and given that the
>>> functionality that RecordSend hopes to provide is essentially that of
>>> Future, I think it makes sense to expose a widely understood public API
>>> for
>>> our clients. RecordSend, on the other hand, seems to provide some APIs
>>> that
>>> are very similar to that of Future, in addition to exposing a bunch of
>>> APIs
>>> that belong to ProduceRequestResult. As a user, I would've really
>>> preferred
>>> to deal with ProduceRequestResult directly -
>>> Future<ProduceRequestResult> send(...)
>>>
>>> - Error handling
>>> RecordSend's error handling is quite unintuitive where the user has to
>>> remember to invoke hasError and error, instead of just throwing the
>>> exception. Now there are
>>> some downsides regarding error handling with the Future as well, where the
>>> user has to catch InterruptedException when we would never run into it.
>>> However, it seems like a price worth paying for supporting a standard API
>>> and error handling
>>>
>>> - Unused APIs
>>> This is a downside of using Future, where the cancel() operation would
>>> always return false and mean nothing. But we can mention that caveat in
>>> our
>>> Java docs.
>>>
>>> To summarize, I would prefer to expose a well understood and widely
>>> adopted
>>> Java API and put up with the overhead of catching one unnecessary checked
>>> exception, rather than wrap the useful ProduceRequestResult in a custom
>>> async object (RecordSend) and explain that to our many users.
>>>
>>> Thanks,
>>> Neha
>>>
>>>
>>>
>>>
>>>> On Tue, Jan 28, 2014 at 8:10 PM, Jay Kreps <jay.kr...@gmail.com> wrote:
>>>>
>>>> Hey Neha,
>>>>
>>>> Can you elaborate on why you prefer using Java's Future? The downside
>>> in my
>>>> mind is the use of the checked InterruptedException and
>>> ExecutionException.
>>>> ExecutionException is arguable, but forcing you to catch
>>>> InterruptedException, often in code that can't be interrupted, seems
>>>> perverse. It also leaves us with the cancel() method which I don't
>>> think we
>>>> really can implement.
>>>>
>>>> Option 1A, to recap/elaborate, was the following. There is no
>>> Serializer or
>>>> Partitioner api. We take a byte[] key and value and an optional integer
>>>> partition. If you specify the integer partition it will be used. If you
>>> do
>>>> not specify a key or a partition the partition will be chosen in a round
>>>> robin fashion. If you specify a key but no partition we will chose a
>>>> partition based on a hash of the key. In order to let the user find the
>>>> partition we will need to given them access to the Cluster instance
>>>> directly from the producer.
>>>>
>>>> -Jay
>>>>
>>>>
>>>> On Tue, Jan 28, 2014 at 6:25 PM, Neha Narkhede <neha.narkh...@gmail.com
>>>>> wrote:
>>>>
>>>>> Here are more thoughts on the public APIs -
>>>>>
>>>>> - I suggest we use java's Future instead of custom Future especially
>>>> since
>>>>> it is part of the public API
>>>>>
>>>>> - Serialization: I like the simplicity of the producer APIs with the
>>>>> absence of serialization where we just deal with byte arrays for keys
>>> and
>>>>> values. What I don't like about this is the performance overhead on
>>> the
>>>>> Partitioner for any kind of custom partitioning based on the
>>>> partitionKey.
>>>>> Since the only purpose of partitionKey is to do custom partitioning,
>>> why
>>>>> can't we take it in directly as an integer and let the user figure out
>>>> the
>>>>> mapping from partition_key -> partition_id using the getCluster() API?
>>>> If I
>>>>> understand correctly, this is similar to what you suggested as part of
>>>>> option 1A. I like this approach since it maintains the simplicity of
>>> APIs
>>>>> by allowing us to deal with bytes and does not compromise performance
>>> in
>>>>> the custom partitioning case.
>>>>>
>>>>> Thanks,
>>>>> Neha
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Jan 28, 2014 at 5:42 PM, Jay Kreps <jay.kr...@gmail.com>
>>> wrote:
>>>>>
>>>>>> Hey Tom,
>>>>>>
>>>>>> That sounds cool. How did you end up handling parallel I/O if you
>>> wrap
>>>>> the
>>>>>> individual connections? Don't you need some selector that selects
>>> over
>>>>> all
>>>>>> the connections?
>>>>>>
>>>>>> -Jay
>>>>>>
>>>>>>
>>>>>> On Tue, Jan 28, 2014 at 2:31 PM, Tom Brown <tombrow...@gmail.com>
>>>> wrote:
>>>>>>
>>>>>>> I implemented a 0.7 client in pure java, and its API very closely
>>>>>> resembled
>>>>>>> this. (When multiple people independently engineer the same
>>> solution,
>>>>>> it's
>>>>>>> probably good... right?). However, there were a few architectural
>>>>>>> differences with my client:
>>>>>>>
>>>>>>> 1. The basic client itself was just an asynchronous layer around
>>> the
>>>>>>> different server functions. In and of itself it had no knowledge
>>> of
>>>>>>> partitions, only servers (and maintained TCP connections to them).
>>>>>>>
>>>>>>> 2. The main producer was an additional layer that provided a
>>>> high-level
>>>>>>> interface that could batch individual messages based on partition.
>>>>>>>
>>>>>>> 3. Knowledge of partitioning was done via an interface so that
>>>>> different
>>>>>>> strategies could be used.
>>>>>>>
>>>>>>> 4. Partitioning was done by the user, with knowledge of the
>>> available
>>>>>>> partitions provided by #3.
>>>>>>>
>>>>>>> 5. Serialization was done by the user to simplify the API.
>>>>>>>
>>>>>>> 6. Futures were used to make asynchronous emulate synchronous
>>> calls.
>>>>>>>
>>>>>>>
>>>>>>> The main benefit of this approach is flexibility. For example,
>>> since
>>>>> the
>>>>>>> base client was just a managed connection (and not inherently a
>>>>>> producer),
>>>>>>> it was easy to composite a produce request and an offsets request
>>>>>> together
>>>>>>> into a confirmed produce request (officially not available in
>>> 0.7).
>>>>>>>
>>>>>>> Decoupling the basic client from partition management allowed the
>>> me
>>>> to
>>>>>>> implement zk discovery as a separate project so that the main
>>> project
>>>>> had
>>>>>>> no complex dependencies. The same was true of decoupling
>>>> serialization.
>>>>>>> It's trivial to build an optional layer that adds those features
>>> in,
>>>>>> while
>>>>>>> allowing access to the base APIs for those that need it.
>>>>>>>
>>>>>>> Using standard Future objects was also beneficial, since I could
>>>>> combine
>>>>>>> them with existing tools (such as guava).
>>>>>>>
>>>>>>> It may be too late to be of use, but I have been working with my
>>>>>> company's
>>>>>>> legal department to release the implementation I described above.
>>> If
>>>>>> you're
>>>>>>> interested in it, let me know.
>>>>>>>
>>>>>>>
>>>>>>> To sum up my thoughts regarding the new API, I think it's a great
>>>>> start.
>>>>>> I
>>>>>>> would like to see a more layered approach so I can use the parts I
>>>>> want,
>>>>>>> and adapt the other parts as needed. I would also like to see
>>>> standard
>>>>>>> interfaces (especially Future) used where they makes sense.
>>>>>>>
>>>>>>> --Tom
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jan 28, 2014 at 1:33 PM, Roger Hoover <
>>>> roger.hoo...@gmail.com
>>>>>>>> wrote:
>>>>>>>
>>>>>>>> +1 ListenableFuture: If this works similar to Deferreds in
>>> Twisted
>>>>>> Python
>>>>>>>> or Promised IO in Javascript, I think this is a great pattern
>>> for
>>>>>>>> decoupling your callback logic from the place where the Future
>>> is
>>>>>>>> generated. You can register as many callbacks as you like,
>>> each in
>>>>> the
>>>>>>>> appropriate layer of the code and have each observer get
>>> notified
>>>>> when
>>>>>>> the
>>>>>>>> promised i/o is complete without any of them knowing about each
>>>>> other.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Jan 28, 2014 at 11:32 AM, Jay Kreps <
>>> jay.kr...@gmail.com>
>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hey Ross,
>>>>>>>>>
>>>>>>>>> - ListenableFuture: Interesting. That would be an alternative
>>> to
>>>>> the
>>>>>>>> direct
>>>>>>>>> callback support we provide. There could be pros to this, let
>>> me
>>>>>> think
>>>>>>>>> about it.
>>>>>>>>> - We could provide layering, but I feel that the
>>> serialization is
>>>>>> such
>>>>>>> a
>>>>>>>>> small thing we should just make a decision and chose one, it
>>>>> doesn't
>>>>>>> seem
>>>>>>>>> to me to justify a whole public facing layer.
>>>>>>>>> - Yes, this is fairly esoteric, essentially I think it is
>>> fairly
>>>>>>> similar
>>>>>>>> to
>>>>>>>>> databases like DynamoDB that allow you to specify two
>>> partition
>>>>> keys
>>>>>> (I
>>>>>>>>> think DynamoDB does this...). The reasoning is that in fact
>>> there
>>>>> are
>>>>>>>>> several things you can use the key field for: (1) to compute
>>> the
>>>>>>>> partition
>>>>>>>>> to store the data in, (2) as a unique identifier to
>>> deduplicate
>>>>> that
>>>>>>>>> partition's records within a log. These two things are almost
>>>>> always
>>>>>>> the
>>>>>>>>> same, but occationally may differ when you want to group data
>>> in
>>>> a
>>>>>> more
>>>>>>>>> sophisticated way then just a hash of the primary key but
>>> still
>>>>>> retain
>>>>>>>> the
>>>>>>>>> proper primary key for delivery to the consumer and log
>>>> compaction.
>>>>>>>>>
>>>>>>>>> -Jay
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Jan 28, 2014 at 3:24 AM, Ross Black <
>>>>> ross.w.bl...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Jay,
>>>>>>>>>>
>>>>>>>>>> - Just to add some more info/confusion about possibly using
>>>>> Future
>>>>>>> ...
>>>>>>>>>> If Kafka uses a JDK future, it plays nicely with other
>>>>> frameworks
>>>>>>> as
>>>>>>>>>> well.
>>>>>>>>>> Google Guava has a ListenableFuture that allows callback
>>>>> handling
>>>>>>> to
>>>>>>>> be
>>>>>>>>>> added via the returned future, and allows the callbacks to
>>> be
>>>>>> passed
>>>>>>>> off
>>>>>>>>> to
>>>>>>>>>> a specified executor.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>> http://docs.guava-libraries.googlecode.com/git-history/release/javadoc/com/google/common/util/concurrent/ListenableFuture.html
>>>>>>>>>> The JDK future can easily be converted to a listenable
>>>> future.
>>>>>>>>>>
>>>>>>>>>> - On the question of byte[] vs Object, could this be solved
>>> by
>>>>>>> layering
>>>>>>>>> the
>>>>>>>>>> API? eg. a raw producer (use byte[] and specify the
>>> partition
>>>>>>> number)
>>>>>>>>> and
>>>>>>>>>> a normal producer (use generic object and specify a
>>>> Partitioner)?
>>>>>>>>>>
>>>>>>>>>> - I am confused by the keys in ProducerRecord and
>>> Partitioner.
>>>>>> What
>>>>>>> is
>>>>>>>>> the
>>>>>>>>>> usage for both a key and a partition key? (I am not yet
>>> using
>>>>> 0.8)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Ross
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 28 January 2014 05:00, Xavier Stevens <xav...@gaikai.com
>>>>
>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> AutoCloseable would be nice for us as most of our code is
>>>> using
>>>>>>> Java
>>>>>>>> 7
>>>>>>>>> at
>>>>>>>>>>> this point.
>>>>>>>>>>>
>>>>>>>>>>> I like Dropwizard's configuration mapping to POJOs via
>>>> Jackson,
>>>>>> but
>>>>>>>> if
>>>>>>>>>> you
>>>>>>>>>>> wanted to stick with property maps I don't care enough to
>>>>> object.
>>>>>>>>>>>
>>>>>>>>>>> If the producer only dealt with bytes, is there a way we
>>>> could
>>>>>>> still
>>>>>>>>> due
>>>>>>>>>>> partition plugins without specifying the number
>>> explicitly? I
>>>>>> would
>>>>>>>>>> prefer
>>>>>>>>>>> to be able to pass in field(s) that would be used by the
>>>>>>> partitioner.
>>>>>>>>>>> Obviously if this wasn't possible you could always
>>>> deserialize
>>>>>> the
>>>>>>>>> object
>>>>>>>>>>> in the partitioner and grab the fields you want, but that
>>>> seems
>>>>>>>> really
>>>>>>>>>>> expensive to do on every message.
>>>>>>>>>>>
>>>>>>>>>>> It would also be nice to have a Java API Encoder
>>> constructor
>>>>>> taking
>>>>>>>> in
>>>>>>>>>>> VerifiableProperties. Scala understands how to handle
>>> "props:
>>>>>>>>>>> VerifiableProperties = null", but Java doesn't. So you
>>> don't
>>>>> run
>>>>>>> into
>>>>>>>>>> this
>>>>>>>>>>> problem until runtime.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -Xavier
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jan 27, 2014 at 9:37 AM, Clark Breyman <
>>>>>> cl...@breyman.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Jay -
>>>>>>>>>>>>
>>>>>>>>>>>> Config - your explanation makes sense. I'm just so
>>>> accustomed
>>>>>> to
>>>>>>>>> having
>>>>>>>>>>>> Jackson automatically map my configuration objects to
>>> POJOs
>>>>>> that
>>>>>>>> I've
>>>>>>>>>>>> stopped using property files. They are lingua franca.
>>> The
>>>>> only
>>>>>>>>> thought
>>>>>>>>>>>> might be to separate the config interface from the
>>>>>> implementation
>>>>>>>> to
>>>>>>>>>>> allow
>>>>>>>>>>>> for alternatives, but that might undermine your point of
>>>> "do
>>>>> it
>>>>>>>> this
>>>>>>>>>> way
>>>>>>>>>>> so
>>>>>>>>>>>> that everyone can find it where they expect it".
>>>>>>>>>>>>
>>>>>>>>>>>> Serialization: Of the options, I like 1A the best,
>>> though
>>>>>>> possibly
>>>>>>>>> with
>>>>>>>>>>>> either an option to specify a partition key rather than
>>> ID
>>>>> or a
>>>>>>>>> helper
>>>>>>>>>> to
>>>>>>>>>>>> translate an arbitrary byte[] or long into a partition
>>>>> number.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Clark
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Jan 26, 2014 at 9:13 PM, Jay Kreps <
>>>>>> jay.kr...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the detailed thoughts. Let me elaborate on
>>> the
>>>>>>> config
>>>>>>>>>> thing.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I agree that at first glance key-value strings don't
>>> seem
>>>>>> like
>>>>>>> a
>>>>>>>>> very
>>>>>>>>>>>> good
>>>>>>>>>>>>> configuration api for a client. Surely a well-typed
>>>> config
>>>>>>> class
>>>>>>>>>> would
>>>>>>>>>>> be
>>>>>>>>>>>>> better! I actually disagree and let me see if I can
>>>>> convince
>>>>>>> you.
>>>>>>>>>>>>>
>>>>>>>>>>>>> My reasoning has nothing to do with the api and
>>>> everything
>>>>> to
>>>>>>> do
>>>>>>>>> with
>>>>>>>>>>>>> operations.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Clients are embedded in applications which are
>>> themselves
>>>>>>>>> configured.
>>>>>>>>>>> In
>>>>>>>>>>>>> any place that takes operations seriously the
>>>> configuration
>>>>>> for
>>>>>>>>> these
>>>>>>>>>>>>> applications will be version controlled and maintained
>>>>>> through
>>>>>>>> some
>>>>>>>>>>> kind
>>>>>>>>>>>> of
>>>>>>>>>>>>> config management system. If we give a config class
>>> with
>>>>>>> getters
>>>>>>>>> and
>>>>>>>>>>>>> setters the application has to expose those
>>> properties to
>>>>> its
>>>>>>>>>>>>> configuration. What invariably happens is that the
>>>>>> application
>>>>>>>>>> exposes
>>>>>>>>>>>> only
>>>>>>>>>>>>> a choice few properties that they thought they would
>>>>> change.
>>>>>>>>>>> Furthermore
>>>>>>>>>>>>> the application will make up a name for these configs
>>>> that
>>>>>>> seems
>>>>>>>>>>>> intuitive
>>>>>>>>>>>>> at the time in the 2 seconds the engineer spends
>>> thinking
>>>>>> about
>>>>>>>> it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Now consider the result of this in the large. You end
>>> up
>>>>> with
>>>>>>>>> dozens
>>>>>>>>>> or
>>>>>>>>>>>>> hundreds of applications that have the client
>>> embedded.
>>>>> Each
>>>>>>>>> exposes
>>>>>>>>>> a
>>>>>>>>>>>>> different, inadequate subset of the possible configs,
>>>> each
>>>>>> with
>>>>>>>>>>> different
>>>>>>>>>>>>> names. It is a nightmare.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If you use a string-string map the config system can
>>>>> directly
>>>>>>>> get a
>>>>>>>>>>>> bundle
>>>>>>>>>>>>> of config key-value pairs and put them into the
>>> client.
>>>>> This
>>>>>>>> means
>>>>>>>>>> that
>>>>>>>>>>>> all
>>>>>>>>>>>>> configuration is automatically available with the name
>>>>>>> documented
>>>>>>>>> on
>>>>>>>>>>> the
>>>>>>>>>>>>> website in every application that does this. If you
>>>> upgrade
>>>>>> to
>>>>>>> a
>>>>>>>>> new
>>>>>>>>>>>> kafka
>>>>>>>>>>>>> version with more configs those will be exposed too.
>>> If
>>>> you
>>>>>>>> realize
>>>>>>>>>>> that
>>>>>>>>>>>>> you need to change a default you can just go through
>>> your
>>>>>>> configs
>>>>>>>>> and
>>>>>>>>>>>>> change it everywhere as it will have the same name
>>>>>> everywhere.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Jay
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Jan 26, 2014 at 4:47 PM, Clark Breyman <
>>>>>>>> cl...@breyman.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks Jay. I'll see if I can put together a more
>>>>> complete
>>>>>>>>>> response,
>>>>>>>>>>>>>> perhaps as separate threads so that topics don't get
>>>>>>> entangled.
>>>>>>>>> In
>>>>>>>>>>> the
>>>>>>>>>>>>> mean
>>>>>>>>>>>>>> time, here's a couple responses:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Serialization: you've broken out a sub-thread so
>>> i'll
>>>>> reply
>>>>>>>>> there.
>>>>>>>>>> My
>>>>>>>>>>>>> bias
>>>>>>>>>>>>>> is that I like generics (except for type-erasure)
>>> and
>>>> in
>>>>>>>>> particular
>>>>>>>>>>>> they
>>>>>>>>>>>>>> make it easy to compose serializers for compound
>>>> payloads
>>>>>>> (e.g.
>>>>>>>>>> when
>>>>>>>>>>> a
>>>>>>>>>>>>>> common header wraps a payload of parameterized
>>> type).
>>>>> I'll
>>>>>>>>> respond
>>>>>>>>>> to
>>>>>>>>>>>>> your
>>>>>>>>>>>>>> 4-options message with an example.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Build: I've seen a lot of "maven-compatible" build
>>>>> systems
>>>>>>>>> produce
>>>>>>>>>>>>>> "artifacts" that aren't really artifacts - no
>>> embedded
>>>>> POM
>>>>>>> or,
>>>>>>>>>> worst,
>>>>>>>>>>>>>> malformed POM. I know the sbt-generated artifacts
>>> were
>>>>> this
>>>>>>>> way -
>>>>>>>>>>> onus
>>>>>>>>>>>> is
>>>>>>>>>>>>>> on me to see what gradle is spitting out and what a
>>>> maven
>>>>>>> build
>>>>>>>>>> might
>>>>>>>>>>>>> look
>>>>>>>>>>>>>> like. Maven may be old and boring, but it gets out
>>> of
>>>> the
>>>>>> way
>>>>>>>> and
>>>>>>>>>>>>>> integrates really seamlessly with a lot of IDEs.
>>> When
>>>>> some
>>>>>>>> scala
>>>>>>>>>>>>> projects I
>>>>>>>>>>>>>> was working on in the fall of 2011 switched from
>>> sbt to
>>>>>>> maven,
>>>>>>>>>> build
>>>>>>>>>>>>> became
>>>>>>>>>>>>>> a non-issue.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Config: Not a big deal and no, I don't think a
>>>>> dropwizard
>>>>>>>>>> dependency
>>>>>>>>>>>> is
>>>>>>>>>>>>>> appropriate. I do like using simple entity beans
>>>> (POJO's
>>>>>> not
>>>>>>>>> j2EE)
>>>>>>>>>>> for
>>>>>>>>>>>>>> configuration, especially if they can be marshalled
>>>>> without
>>>>>>>>>>> annotation
>>>>>>>>>>>> by
>>>>>>>>>>>>>> Jackson. I only mentioned the dropwizard-extras
>>>> because
>>>>> it
>>>>>>> has
>>>>>>>>>> some
>>>>>>>>>>>>> entity
>>>>>>>>>>>>>> bean versions of the ZK and Kafka configs.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Domain-packaging: Also not a big deal - it's what's
>>>>>> expected
>>>>>>>> and
>>>>>>>>>> it's
>>>>>>>>>>>>>> pretty free in most IDE's. The advantages I see is
>>> that
>>>>> it
>>>>>> is
>>>>>>>>> clear
>>>>>>>>>>>>> whether
>>>>>>>>>>>>>> something is from the Apache Kafka project and
>>> whether
>>>>>>>> something
>>>>>>>>> is
>>>>>>>>>>>> from
>>>>>>>>>>>>>> another org and related to Kafka. That said, nothing
>>>>> really
>>>>>>>>>> enforces
>>>>>>>>>>>> it.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Futures: I'll see if I can create some examples to
>>>>>>> demonstrate
>>>>>>>>>> Future
>>>>>>>>>>>>>> making interop easier.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> C
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Jan 24, 2014 at 4:36 PM, Jay Kreps <
>>>>>>>> jay.kr...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hey Clark,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Serialization: Yes I agree with these though I
>>>> don't
>>>>>>>> consider
>>>>>>>>>> the
>>>>>>>>>>>>> loss
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> generics a big issue. I'll try to summarize what I
>>>>> would
>>>>>>>>> consider
>>>>>>>>>>> the
>>>>>>>>>>>>>> best
>>>>>>>>>>>>>>> alternative api with raw byte[].
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Maven: We had this debate a few months back and
>>> the
>>>>>>>> consensus
>>>>>>>>>> was
>>>>>>>>>>>>>> gradle.
>>>>>>>>>>>>>>> Is there a specific issue with the poms gradle
>>>> makes? I
>>>>>> am
>>>>>>>>>>> extremely
>>>>>>>>>>>>>> loath
>>>>>>>>>>>>>>> to revisit the issue as build issues are a
>>> recurring
>>>>>> thing
>>>>>>>> and
>>>>>>>>> no
>>>>>>>>>>> one
>>>>>>>>>>>>>> ever
>>>>>>>>>>>>>>> agrees and ultimately our build needs are very
>>>> simple.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Config: I'm not sure if I follow the point. Are
>>> you
>>>>>>> saying
>>>>>>>> we
>>>>>>>>>>>> should
>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>> something in dropwizard for config? One principle
>>>> here
>>>>> is
>>>>>>> to
>>>>>>>>> try
>>>>>>>>>> to
>>>>>>>>>>>>>> remove
>>>>>>>>>>>>>>> as many client dependencies as possible as we
>>>>> inevitably
>>>>>>> run
>>>>>>>>> into
>>>>>>>>>>>>>> terrible
>>>>>>>>>>>>>>> compatibility issues with users who use the same
>>>>> library
>>>>>> or
>>>>>>>> its
>>>>>>>>>>>>>>> dependencies at different versions. Or are you
>>>> talking
>>>>>>> about
>>>>>>>>>>>>> maintaining
>>>>>>>>>>>>>>> compatibility with existing config parameters? I
>>>> think
>>>>> as
>>>>>>>> much
>>>>>>>>>> as a
>>>>>>>>>>>>>> config
>>>>>>>>>>>>>>> in the existing client makes sense it should have
>>> the
>>>>>> same
>>>>>>>> name
>>>>>>>>>> (I
>>>>>>>>>>>> was
>>>>>>>>>>>>> a
>>>>>>>>>>>>>>> bit sloppy about that so I'll fix any errors
>>> there).
>>>>>> There
>>>>>>>> are
>>>>>>>>> a
>>>>>>>>>>> few
>>>>>>>>>>>>> new
>>>>>>>>>>>>>>> things and we should give those reasonable
>>> defaults.
>>>> I
>>>>>>> think
>>>>>>>>>> config
>>>>>>>>>>>> is
>>>>>>>>>>>>>>> important so I'll start a thread on the config
>>>> package
>>>>> in
>>>>>>>>> there.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - org.apache.kafka: We could do this. I always
>>>>> considered
>>>>>>> it
>>>>>>>>> kind
>>>>>>>>>>> of
>>>>>>>>>>>> an
>>>>>>>>>>>>>> odd
>>>>>>>>>>>>>>> thing Java programmers do that has no real
>>> motivation
>>>>>> (but
>>>>>>> I
>>>>>>>>>> could
>>>>>>>>>>> be
>>>>>>>>>>>>>>> re-educated!). I don't think it ends up reducing
>>>> naming
>>>>>>>>> conflicts
>>>>>>>>>>> in
>>>>>>>>>>>>>>> practice and it adds a lot of noise and nested
>>>>>> directories.
>>>>>>>> Is
>>>>>>>>>>> there
>>>>>>>>>>>> a
>>>>>>>>>>>>>>> reason you prefer this or just to be more
>>> standard?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Future: Basically I didn't see any particular
>>>>>> advantage.
>>>>>>>> The
>>>>>>>>>>>> cancel()
>>>>>>>>>>>>>>> method doesn't really make sense so probably
>>> wouldn't
>>>>>> work.
>>>>>>>>>>> Likewise
>>>>>>>>>>>> I
>>>>>>>>>>>>>>> dislike the checked exceptions it requires.
>>>> Basically I
>>>>>>> just
>>>>>>>>>> wrote
>>>>>>>>>>>> out
>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>> code examples and it seemed cleaner with a special
>>>>>> purpose
>>>>>>>>>> object.
>>>>>>>>>>> I
>>>>>>>>>>>>>> wasn't
>>>>>>>>>>>>>>> actually aware of plans for improved futures in
>>> java
>>>> 8
>>>>> or
>>>>>>> the
>>>>>>>>>> other
>>>>>>>>>>>>>>> integrations. Maybe you could elaborate on this a
>>> bit
>>>>> and
>>>>>>>> show
>>>>>>>>>> how
>>>>>>>>>>> it
>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>> be used? Sounds promising, I just don't know a lot
>>>>> about
>>>>>>> it.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Jay
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Jan 24, 2014 at 3:30 PM, Clark Breyman <
>>>>>>>>>> cl...@breyman.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Jay - Thanks for the call for comments. Here's
>>> some
>>>>>>> initial
>>>>>>>>>>> input:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Make message serialization a client
>>>> responsibility
>>>>>>>> (making
>>>>>>>>>> all
>>>>>>>>>>>>>> messages
>>>>>>>>>>>>>>>> byte[]). Reflection-based loading makes it
>>> harder
>>>> to
>>>>>> use
>>>>>>>>>> generic
>>>>>>>>>>>>> codecs
>>>>>>>>>>>>>>>> (e.g. Envelope<PREFIX, DATA, SUFFIX>) or build
>>> up
>>>>>> codec
>>>>>>>>>>>>>>> programmatically.
>>>>>>>>>>>>>>>> Non-default partitioning should require an
>>> explicit
>>>>>>>> partition
>>>>>>>>>>> key.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - I really like the fact that it will be native
>>>> Java.
>>>>>>>> Please
>>>>>>>>>>>> consider
>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>> native maven and not sbt, gradle, ivy, etc as
>>> they
>>>>>> don't
>>>>>>>>>> reliably
>>>>>>>>>>>>> play
>>>>>>>>>>>>>>> nice
>>>>>>>>>>>>>>>> in the maven ecosystem. A jar without a
>>> well-formed
>>>>> pom
>>>>>>>>> doesn't
>>>>>>>>>>>> feel
>>>>>>>>>>>>>>> like a
>>>>>>>>>>>>>>>> real artifact. The pom's generated by sbt et al.
>>>> are
>>>>>> not
>>>>>>>> well
>>>>>>>>>>>> formed.
>>>>>>>>>>>>>>> Using
>>>>>>>>>>>>>>>> maven will make builds and IDE integration much
>>>>>> smoother.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Look at Nick Telford's dropwizard-extras
>>> package
>>>> in
>>>>>>> which
>>>>>>>>> he
>>>>>>>>>>>>> defines
>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>> Jackson-compatible POJO's for loading
>>>> configuration.
>>>>>>> Seems
>>>>>>>>> like
>>>>>>>>>>>> your
>>>>>>>>>>>>>>> client
>>>>>>>>>>>>>>>> migration is similar. The config objects should
>>>> have
>>>>>>>>>> constructors
>>>>>>>>>>>> or
>>>>>>>>>>>>>>>> factories that accept Map<String, String> and
>>>>>> Properties
>>>>>>>> for
>>>>>>>>>> ease
>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> migration.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Would you consider using the org.apache.kafka
>>>>> package
>>>>>>> for
>>>>>>>>> the
>>>>>>>>>>> new
>>>>>>>>>>>>> API
>>>>>>>>>>>>>>>> (quibble)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Why create your own futures rather than use
>>>>>>>>>>>>>>>> java.util.concurrent.Future<Long> or similar?
>>>>> Standard
>>>>>>>>> futures
>>>>>>>>>>> will
>>>>>>>>>>>>>> play
>>>>>>>>>>>>>>>> nice with other reactive libs and things like
>>> J8's
>>>>>>>>>>>> ComposableFuture.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks again,
>>>>>>>>>>>>>>>> C
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Jan 24, 2014 at 2:46 PM, Roger Hoover <
>>>>>>>>>>>>> roger.hoo...@gmail.com
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> A couple comments:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 1) Why does the config use a broker list
>>> instead
>>>> of
>>>>>>>>>> discovering
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> brokers
>>>>>>>>>>>>>>>>> in ZooKeeper? It doesn't match the
>>>>> HighLevelConsumer
>>>>>>>> API.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2) It looks like broker connections are
>>> created
>>>> on
>>>>>>>> demand.
>>>>>>>>>> I'm
>>>>>>>>>>>>>>> wondering
>>>>>>>>>>>>>>>>> if sometimes you might want to flush out
>>> config
>>>> or
>>>>>>>> network
>>>>>>>>>>>>>> connectivity
>>>>>>>>>>>>>>>>> issues before pushing the first message
>>> through.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Should there also be a
>>> KafkaProducer.connect() or
>>>>>>> .open()
>>>>>>>>>>> method
>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>> connectAll()? I guess it would try to
>>> connect to
>>>>> all
>>>>>>>>> brokers
>>>>>>>>>>> in
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> BROKER_LIST_CONFIG
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> HTH,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Roger
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Jan 24, 2014 at 11:54 AM, Jay Kreps <
>>>>>>>>>>> jay.kr...@gmail.com
>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> As mentioned in a previous email we are
>>> working
>>>>> on
>>>>>> a
>>>>>>>>>>>>>>> re-implementation
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> the producer. I would like to use this email
>>>>> thread
>>>>>>> to
>>>>>>>>>>> discuss
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> details
>>>>>>>>>>>>>>>>>> of the public API and the configuration. I
>>>> would
>>>>>> love
>>>>>>>> for
>>>>>>>>>> us
>>>>>>>>>>> to
>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> incredibly picky about this public api now
>>> so
>>>> it
>>>>> is
>>>>>>> as
>>>>>>>>> good
>>>>>>>>>>> as
>>>>>>>>>>>>>>> possible
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> we don't need to break it in the future.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The best way to get a feel for the API is
>>>>> actually
>>>>>> to
>>>>>>>>> take
>>>>>>>>>> a
>>>>>>>>>>>> look
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> javadoc, my hope is to get the api docs good
>>>>> enough
>>>>>>> so
>>>>>>>>> that
>>>>>>>>>>> it
>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> self-explanatory:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>> http://empathybox.com/kafka-javadoc/index.html?kafka/clients/producer/KafkaProducer.html
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Please take a look at this API and give me
>>> any
>>>>>>> thoughts
>>>>>>>>> you
>>>>>>>>>>> may
>>>>>>>>>>>>>> have!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> It may also be reasonable to take a look at
>>> the
>>>>>>>> configs:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>> http://empathybox.com/kafka-javadoc/kafka/clients/producer/ProducerConfig.html
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The actual code is posted here:
>>>>>>>>>>>>>>>>>>
>>>> https://issues.apache.org/jira/browse/KAFKA-1227
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> A few questions or comments to kick things
>>> off:
>>>>>>>>>>>>>>>>>> 1. We need to make a decision on whether
>>>>>>> serialization
>>>>>>>> of
>>>>>>>>>> the
>>>>>>>>>>>>>> user's
>>>>>>>>>>>>>>>> key
>>>>>>>>>>>>>>>>>> and value should be done by the user (with
>>> our
>>>>> api
>>>>>>> just
>>>>>>>>>>> taking
>>>>>>>>>>>>>>> byte[])
>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>> if we should take an object and allow the
>>> user
>>>> to
>>>>>>>>>> configure a
>>>>>>>>>>>>>>>> Serializer
>>>>>>>>>>>>>>>>>> class which we instantiate via reflection.
>>> We
>>>>> take
>>>>>>> the
>>>>>>>>>> later
>>>>>>>>>>>>>> approach
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> the current producer, and I have carried
>>> this
>>>>>> through
>>>>>>>> to
>>>>>>>>>> this
>>>>>>>>>>>>>>>> prototype.
>>>>>>>>>>>>>>>>>> The tradeoff I see is this: taking byte[] is
>>>>>> actually
>>>>>>>>>>> simpler,
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>> directly do whatever serialization they
>>> like.
>>>> The
>>>>>>>>>>> complication
>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> actually
>>>>>>>>>>>>>>>>>> partitioning. Currently partitioning is done
>>>> by a
>>>>>>>> similar
>>>>>>>>>>>> plug-in
>>>>>>>>>>>>>> api
>>>>>>>>>>>>>>>>>> (Partitioner) which the user can implement
>>> and
>>>>>>>> configure
>>>>>>>>> to
>>>>>>>>>>>>>> override
>>>>>>>>>>>>>>>> how
>>>>>>>>>>>>>>>>>> partitions are assigned. If we take byte[]
>>> as
>>>>> input
>>>>>>>> then
>>>>>>>>> we
>>>>>>>>>>>> have
>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>>> access
>>>>>>>>>>>>>>>>>> to the original object and partitioning
>>> MUST be
>>>>>> done
>>>>>>> on
>>>>>>>>> the
>>>>>>>>>>>>> byte[].
>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> fine for hash partitioning. However for
>>> various
>>>>>> types
>>>>>>>> of
>>>>>>>>>>>> semantic
>>>>>>>>>>>>>>>>>> partitioning (range partitioning, or
>>> whatever)
>>>>> you
>>>>>>>> would
>>>>>>>>>> want
>>>>>>>>>>>>>> access
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> original object. In the current approach a
>>>>> producer
>>>>>>> who
>>>>>>>>>>> wishes
>>>>>>>>>>>> to
>>>>>>>>>>>>>>> send
>>>>>>>>>>>>>>>>>> byte[] they have serialized in their own
>>> code
>>>> can
>>>>>>>>> configure
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> BytesSerialization we supply which is just a
>>>> "no
>>>>>> op"
>>>>>>>>>>>>> serialization.
>>>>>>>>>>>>>>>>>> 2. We should obsess over naming and make
>>> sure
>>>>> each
>>>>>> of
>>>>>>>> the
>>>>>>>>>>> class
>>>>>>>>>>>>>> names
>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>> good.
>>>>>>>>>>>>>>>>>> 3. Jun has already pointed out that we need
>>> to
>>>>>>> include
>>>>>>>>> the
>>>>>>>>>>>> topic
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> partition in the response, which is
>>> absolutely
>>>>>>> right. I
>>>>>>>>>>> haven't
>>>>>>>>>>>>>> done
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> yet but that definitely needs to be there.
>>>>>>>>>>>>>>>>>> 4. Currently RecordSend.await will throw an
>>>>>> exception
>>>>>>>> if
>>>>>>>>>> the
>>>>>>>>>>>>>> request
>>>>>>>>>>>>>>>>>> failed. The intention here is that
>>>>>>>>>>>> producer.send(message).await()
>>>>>>>>>>>>>>>> exactly
>>>>>>>>>>>>>>>>>> simulates a synchronous call. Guozhang has
>>>> noted
>>>>>> that
>>>>>>>>> this
>>>>>>>>>>> is a
>>>>>>>>>>>>>>> little
>>>>>>>>>>>>>>>>>> annoying since the user must then catch
>>>>> exceptions.
>>>>>>>>> However
>>>>>>>>>>> if
>>>>>>>>>>>> we
>>>>>>>>>>>>>>>> remove
>>>>>>>>>>>>>>>>>> this then if the user doesn't check for
>>> errors
>>>>> they
>>>>>>>> won't
>>>>>>>>>>> know
>>>>>>>>>>>>> one
>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>> occurred, which I predict will be a common
>>>>> mistake.
>>>>>>>>>>>>>>>>>> 5. Perhaps there is more we could do to make
>>>> the
>>>>>>> async
>>>>>>>>>>>> callbacks
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> future
>>>>>>>>>>>>>>>>>> we give back intuitive and easy to program
>>>>> against?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Some background info on implementation:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> At a high level the primary difference in
>>> this
>>>>>>> producer
>>>>>>>>> is
>>>>>>>>>>> that
>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> removes
>>>>>>>>>>>>>>>>>> the distinction between the "sync" and
>>> "async"
>>>>>>>> producer.
>>>>>>>>>>>>>> Effectively
>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>> requests are sent asynchronously but always
>>>>> return
>>>>>> a
>>>>>>>>> future
>>>>>>>>>>>>>> response
>>>>>>>>>>>>>>>>> object
>>>>>>>>>>>>>>>>>> that gives the offset as well as any error
>>> that
>>>>> may
>>>>>>>> have
>>>>>>>>>>>> occurred
>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> request is complete. The batching that is
>>> done
>>>> in
>>>>>> the
>>>>>>>>> async
>>>>>>>>>>>>>> producer
>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>> today is done whenever possible now. This
>>> means
>>>>>> that
>>>>>>>> the
>>>>>>>>>> sync
>>>>>>>>>>>>>>> producer,
>>>>>>>>>>>>>>>>>> under load, can get performance as good as
>>> the
>>>>>> async
>>>>>>>>>> producer
>>>>>>>>>>>>>>>>> (preliminary
>>>>>>>>>>>>>>>>>> results show the producer getting 1m
>>>>> messages/sec).
>>>>>>>> This
>>>>>>>>>>> works
>>>>>>>>>>>>>>> similar
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> group commit in databases but with respect
>>> to
>>>> the
>>>>>>>> actual
>>>>>>>>>>>> network
>>>>>>>>>>>>>>>>>> transmission--any messages that arrive
>>> while a
>>>>> send
>>>>>>> is
>>>>>>>> in
>>>>>>>>>>>>> progress
>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>> batched together. It is also possible to
>>>>> encourage
>>>>>>>>> batching
>>>>>>>>>>>> even
>>>>>>>>>>>>>>> under
>>>>>>>>>>>>>>>>> low
>>>>>>>>>>>>>>>>>> load to save server resources by
>>> introducing a
>>>>>> delay
>>>>>>> on
>>>>>>>>> the
>>>>>>>>>>>> send
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> allow
>>>>>>>>>>>>>>>>>> more messages to accumulate; this is done
>>> using
>>>>> the
>>>>>>>>>>>>> linger.msconfig
>>>>>>>>>>>>>>>>> (this
>>>>>>>>>>>>>>>>>> is similar to Nagle's algorithm in TCP).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This producer does all network communication
>>>>>>>>> asynchronously
>>>>>>>>>>> and
>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> parallel
>>>>>>>>>>>>>>>>>> to all servers so the performance penalty
>>> for
>>>>>> acks=-1
>>>>>>>> and
>>>>>>>>>>>> waiting
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>> replication should be much reduced. I
>>> haven't
>>>>> done
>>>>>>> much
>>>>>>>>>>>>>> benchmarking
>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>> this yet, though.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The high level design is described a little
>>>> here,
>>>>>>>> though
>>>>>>>>>> this
>>>>>>>>>>>> is
>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>> little out of date:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -Jay
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
