Jason and Ismael,
It seems like the only thing we need to regard if we define a new error
code (i.e., UNSUPPORTED_COMPRESSION_TYPE) would be the implementation of
the other language clients, right? At least, this strategy causes any
problem for Java client. Do I understand correctly?
Thanks,
Dongjin
On Wed, Aug 22, 2018 at 5:43 PM Dongjin Lee <dong...@apache.org> wrote:
> Jason,
>
> > I think we would only use this error code when we /know/ that zstd was
> in use and the client doesn't support it? This is true if either 1) the
> message needs down-conversion and we encounter a zstd compressed message,
> or 2) if the topic is explicitly configured to use zstd.
>
> Yes, it is right. And you know, the case 1 includes 1.a) old clients'
> request v0, v1 records or 1.b) implicit zstd, the compression type of
> "producer" with Zstd compressed data.
>
> > However, if the compression type is set to "producer," then the fetched
> data may or may not be compressed with zstd. In this case, we return the
> data to the client and expect it to fail parsing. Is that correct?
>
> Exactly.
>
> Following your message, I reviewed the implementation of
> `KafkaApis#handleFetchRequest,` which handles the fetch request. And found
> that the information we can use is like the following:
>
> 1. Client's fetch request version. (`versionId` variable)
> 2. Log's compression type. (`logConfig` variable)
>
> We can't detect the actual compression type of the data, unless we inspect
> the `RecordBatch` included in the `Records` instance (i.e.,
> `unconvertedRecords` variable.) Since it requires some performance issue,
> it is not our option - in short, we can't be sure if given chunks of data
> are compressed with zstd or not.
>
> So, conclusion: we can return an error in the case of 1.a and 2 easily,
> with the information above. In the case 1.b (implicit zstd), we can just
> return the data by do nothing special and expect it to fail parsing.
>
> Thanks,
> Dongjin
>
> On Wed, Aug 22, 2018 at 12:02 PM Ismael Juma <isma...@gmail.com> wrote:
>
>> Jason, that's an interesting point regarding the Java client. Do we know
>> what clients in other languages do in these cases?
>>
>> Ismael
>>
>> On Tue, 21 Aug 2018, 17:30 Jason Gustafson, <ja...@confluent.io> wrote:
>>
>> > Hi Dongjin,
>> >
>> > One of the complications is that old versions of the API will not
>> expect a
>> > new error code. However, since we expect this to be a fatal error anyway
>> > for old clients, it may still be more useful to return the correct error
>> > code. For example, the Kafka clients use the following code to convert
>> the
>> > error code:
>> >
>> > public static Errors forCode(short code) {
>> > Errors error = codeToError.get(code);
>> > if (error != null) {
>> > return error;
>> > } else {
>> > log.warn("Unexpected error code: {}.", code);
>> > return UNKNOWN_SERVER_ERROR;
>> > }
>> > }
>> >
>> > If we return an unsupported error code, it will be converted to an
>> UNKNOWN
>> > error, but at least we will get the message in the log with the correct
>> > code. That seems preferable to returning a misleading error code. So I
>> > wonder if we can use the new UNSUPPORTED_COMPRESSION_TYPE error even for
>> > older versions.
>> >
>> > Also, one question just to check my understanding. I think we would only
>> > use this error code when we /know/ that zstd was in use and the client
>> > doesn't support it? This is true if either 1) the message needs
>> > down-conversion and we encounter a zstd compressed message, or 2) if the
>> > topic is explicitly configured to use zstd. However, if the compression
>> > type is set to "producer," then the fetched data may or may not be
>> > compressed with zstd. In this case, we return the data to the client and
>> > expect it to fail parsing. Is that correct?
>> >
>> > Thanks,
>> > Jason
>> >
>> >
>> >
>> > On Tue, Aug 21, 2018 at 9:08 AM, Dongjin Lee <dong...@apache.org>
>> wrote:
>> >
>> > > Ismael, Jason and all,
>> > >
>> > > I rewrote the backward compatibility strategy & its alternatives like
>> > > following, based on Ismael & Jason's comments. Since it is not
>> updated to
>> > > the wiki yet, don't hesitate to give me a message if you have any
>> opinion
>> > > on it.
>> > >
>> > > ```
>> > > *Backward Compatibility*
>> > >
>> > > We need to establish some backward-compatibility strategy for the
>> case an
>> > > old client subscribes a topic using ZStandard implicitly (i.e.,
>> > > 'compression.type' configuration of given topic is 'producer' and the
>> > > producer compressed the records with ZStandard). We have the following
>> > > options for this situation:
>> > >
>> > > *A. Support ZStandard to the old clients which can understand v0, v1
>> > > messages only.*
>> > >
>> > > This strategy necessarily requires the down-conversion of v2 message
>> > > compressed with Zstandard into v0 or v1 messages, which means a
>> > > considerable performance degradation. So we rejected this strategy.
>> > >
>> > > *B. Bump the API version and support only v2-available clients*
>> > >
>> > > With this approach, we can message the old clients that they are old
>> and
>> > > should be upgraded. However, there are still several options for the
>> > Error
>> > > code.
>> > >
>> > > *B.1. INVALID_REQUEST (42)*
>> > >
>> > > This option gives the client so little information; the user can be
>> > > confused about why the client worked correctly in the past suddenly
>> > > encounters a problem. So we rejected this strategy.
>> > >
>> > > *B.2. CORRUPT_MESSAGE (2)*
>> > >
>> > > This option gives inaccurate information; the user can be surprised
>> and
>> > > misunderstand that the log files are broken in some way. So we
>> rejected
>> > > this strategy.
>> > >
>> > > *B.3 UNSUPPORTED_FOR_MESSAGE_FORMAT (43)*
>> > >
>> > > The advantage of this approach is we don't need to define a new error
>> > code;
>> > > we can reuse it and that's all.
>> > >
>> > > The disadvantage of this approach is that it is also a little bit
>> vague;
>> > > This error code is defined as a work for KIP-98[^1] and now returned
>> in
>> > the
>> > > transaction error.
>> > >
>> > > *B.4. UNSUPPORTED_COMPRESSION_TYPE (new)*
>> > >
>> > > The advantage of this approach is that it is clear and provides an
>> exact
>> > > description. The disadvantage is we need to add a new error code.
>> > > ```
>> > >
>> > > *It seems like what we need to choose is now so clear:
>> > > UNSUPPORTED_FOR_MESSAGE_FORMAT (B.3) or UNSUPPORTED_COMPRESSION_TYPE
>> > > (B.4).*
>> > > The first one doesn't need a new error message but the latter is more
>> > > explicit. Which one do you prefer? Since all of you have much more
>> > > experience and knowledge than me, I will follow your decision. The
>> wiki
>> > > page will be updated following the decision also.
>> > >
>> > > Best,
>> > > Dongjin
>> > >
>> > > [^1]: https://issues.apache.org/jira/browse/KAFKA-4990
>> > >
>> > > On Sun, Aug 19, 2018 at 4:58 AM Ismael Juma <isma...@gmail.com>
>> wrote:
>> > >
>> > > > Sounds reasonable to me.
>> > > >
>> > > > Ismael
>> > > >
>> > > > On Sat, 18 Aug 2018, 12:20 Jason Gustafson, <ja...@confluent.io>
>> > wrote:
>> > > >
>> > > > > Hey Ismael,
>> > > > >
>> > > > > Your summary looks good to me. I think it might also be a good
>> idea
>> > to
>> > > > add
>> > > > > a new UNSUPPORTED_COMPRESSION_TYPE error code to go along with the
>> > > > version
>> > > > > bumps. We won't be able to use it for old api versions since the
>> > > clients
>> > > > > will not understand it, but we can use it going forward so that
>> we're
>> > > not
>> > > > > stuck in a similar situation with a new message format and a new
>> > codec
>> > > to
>> > > > > support. Another option is to use UNSUPPORTED_FOR_MESSAGE_FORMAT,
>> but
>> > > it
>> > > > is
>> > > > > not as explicit.
>> > > > >
>> > > > > -Jason
>> > > > >
>> > > > > On Fri, Aug 17, 2018 at 5:19 PM, Ismael Juma <ism...@juma.me.uk>
>> > > wrote:
>> > > > >
>> > > > > > Hi Dongjin and Jason,
>> > > > > >
>> > > > > > I would agree. My summary:
>> > > > > >
>> > > > > > 1. Support zstd with message format 2 only.
>> > > > > > 2. Bump produce and fetch request versions.
>> > > > > > 3. Provide broker errors whenever possible based on the request
>> > > version
>> > > > > and
>> > > > > > rely on clients for the cases where the broker can't validate
>> > > > efficiently
>> > > > > > (example message format 2 consumer that supports the latest
>> fetch
>> > > > version
>> > > > > > but doesn't support zstd).
>> > > > > >
>> > > > > > If there's general agreement on this, I suggest we update the
>> KIP
>> > to
>> > > > > state
>> > > > > > the proposal and to move the rejected options to its own
>> section.
>> > And
>> > > > > then
>> > > > > > start a vote!
>> > > > > >
>> > > > > > Ismael
>> > > > > >
>> > > > > > On Fri, Aug 17, 2018 at 4:00 PM Jason Gustafson <
>> > ja...@confluent.io>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi Dongjin,
>> > > > > > >
>> > > > > > > Yes, that's a good summary. For clients which support v2, the
>> > > client
>> > > > > can
>> > > > > > > parse the message format and hopefully raise a useful error
>> > message
>> > > > > > > indicating the unsupported compression type. For older
>> clients,
>> > our
>> > > > > > options
>> > > > > > > are probably (1) to down-convert to the old format using no
>> > > > compression
>> > > > > > > type, or (2) to return an error code. I'm leaning toward the
>> > latter
>> > > > as
>> > > > > > the
>> > > > > > > simpler solution, but the challenge is finding a good error
>> code.
>> > > Two
>> > > > > > > possibilities might be INVALID_REQUEST or CORRUPT_MESSAGE. The
>> > > > downside
>> > > > > > is
>> > > > > > > that old clients probably won't get a helpful message.
>> However,
>> > at
>> > > > > least
>> > > > > > > the behavior will be consistent in the sense that all clients
>> > will
>> > > > fail
>> > > > > > if
>> > > > > > > they do not support zstandard.
>> > > > > > >
>> > > > > > > What do you think?
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Jason
>> > > > > > >
>> > > > > > > On Fri, Aug 17, 2018 at 8:08 AM, Dongjin Lee <
>> dong...@apache.org
>> > >
>> > > > > wrote:
>> > > > > > >
>> > > > > > > > Thanks Jason, I reviewed the down-converting logic following
>> > your
>> > > > > > > > explanation.[^1] You mean the following routines, right?
>> > > > > > > >
>> > > > > > > > -
>> > > > > > > > https://github.com/apache/kafka/blob/trunk/core/src/
>> > > > > > > > main/scala/kafka/server/KafkaApis.scala#L534
>> > > > > > > > -
>> > > > > > > > https://github.com/apache/kafka/blob/trunk/clients/src/
>> > > > > > > > main/java/org/apache/kafka/common/record/
>> > > LazyDownConversionRecords.
>> > > > > > > > java#L165
>> > > > > > > > -
>> > > > > > > > https://github.com/apache/kafka/blob/trunk/clients/src/
>> > > > > > > >
>> main/java/org/apache/kafka/common/record/RecordsUtil.java#L40
>> > > > > > > >
>> > > > > > > > It seems like your stance is like following:
>> > > > > > > >
>> > > > > > > > 1. In principle, Kafka does not change the compression codec
>> > when
>> > > > > > > > down-converting, since it requires inspecting the fetched
>> data,
>> > > > which
>> > > > > > is
>> > > > > > > > expensive.
>> > > > > > > > 2. However, there are some cases the fetched data is
>> inspected
>> > > > > anyway.
>> > > > > > In
>> > > > > > > > this case, we can provide compression conversion from
>> Zstandard
>> > > to
>> > > > > > > > classical ones[^2].
>> > > > > > > >
>> > > > > > > > And from what I understand, the cases where the client
>> without
>> > > > > > ZStandard
>> > > > > > > > support receives ZStandard compressed records can be
>> organized
>> > > into
>> > > > > two
>> > > > > > > > cases:
>> > > > > > > >
>> > > > > > > > a. The 'compression.type' configuration of given topic is
>> > > > 'producer'
>> > > > > > and
>> > > > > > > > the producer compressed the records with ZStandard. (that
>> is,
>> > > using
>> > > > > > > > ZStandard implicitly.)
>> > > > > > > > b. The 'compression.type' configuration of given topic is
>> > > 'zstd';
>> > > > > that
>> > > > > > > is,
>> > > > > > > > using ZStandard explicitly.
>> > > > > > > >
>> > > > > > > > As you stated, we don't have to handle the case b specially.
>> > So,
>> > > It
>> > > > > > seems
>> > > > > > > > like we can narrow the focus of the problem by joining case
>> 1
>> > and
>> > > > > case
>> > > > > > b
>> > > > > > > > like the following:
>> > > > > > > >
>> > > > > > > > > Given the topic with 'producer' as its 'compression.type'
>> > > > > > > configuration,
>> > > > > > > > ZStandard compressed records and old client without
>> ZStandard,
>> > is
>> > > > > there
>> > > > > > > any
>> > > > > > > > case we need to inspect the records and can change the
>> > > compression
>> > > > > > type?
>> > > > > > > If
>> > > > > > > > so, can we provide compression type converting?
>> > > > > > > >
>> > > > > > > > Do I understand correctly?
>> > > > > > > >
>> > > > > > > > Best,
>> > > > > > > > Dongjin
>> > > > > > > >
>> > > > > > > > [^1]: I'm sorry, I found that I was a little bit
>> > misunderstanding
>> > > > how
>> > > > > > API
>> > > > > > > > version works, after reviewing the downconvert logic & the
>> > > protocol
>> > > > > > > > documentation <https://kafka.apache.org/protocol>.
>> > > > > > > > [^2]: None, Gzip, Snappy, Lz4.
>> > > > > > > >
>> > > > > > > > On Tue, Aug 14, 2018 at 2:16 AM Jason Gustafson <
>> > > > ja...@confluent.io>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > But in my opinion, since the client will fail with the
>> API
>> > > > > version,
>> > > > > > > so
>> > > > > > > > we
>> > > > > > > > > > don't need to down-convert the messages anyway. Isn't
>> it?
>> > > So, I
>> > > > > > think
>> > > > > > > > we
>> > > > > > > > > > don't care about this case. (I'm sorry, I am not
>> familiar
>> > > with
>> > > > > > > > > down-convert
>> > > > > > > > > > logic.)
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > Currently the broker down-converts automatically when it
>> > > receives
>> > > > > an
>> > > > > > > old
>> > > > > > > > > version of the fetch request (a version which is known to
>> > > predate
>> > > > > the
>> > > > > > > > > message format in use). Typically when down-converting the
>> > > > message
>> > > > > > > > format,
>> > > > > > > > > we use the same compression type, but there is not much
>> point
>> > > in
>> > > > > > doing
>> > > > > > > so
>> > > > > > > > > when we know the client doesn't support it. So if
>> zstandard
>> > is
>> > > in
>> > > > > > use,
>> > > > > > > > and
>> > > > > > > > > we have to down-convert anyway, then we can choose to use
>> a
>> > > > > different
>> > > > > > > > > compression type or no compression type.
>> > > > > > > > >
>> > > > > > > > > From my perspective, there is no significant downside to
>> > > bumping
>> > > > > the
>> > > > > > > > > protocol version and it has several potential benefits.
>> > Version
>> > > > > bumps
>> > > > > > > are
>> > > > > > > > > cheap. The main question mark in my mind is about
>> > > > down-conversion.
>> > > > > > > > Figuring
>> > > > > > > > > out whether down-conversion is needed is hard generally
>> > without
>> > > > > > > > inspecting
>> > > > > > > > > the fetched data, which is expensive. I think we agree in
>> > > > principle
>> > > > > > > that
>> > > > > > > > we
>> > > > > > > > > do not want to have to pay this cost generally and prefer
>> the
>> > > > > clients
>> > > > > > > to
>> > > > > > > > > fail when they see an unhandled compression type. The
>> point I
>> > > was
>> > > > > > > making
>> > > > > > > > is
>> > > > > > > > > that there are some cases where we are either inspecting
>> the
>> > > data
>> > > > > > > anyway
>> > > > > > > > > (because we have to down-convert the message format), or
>> we
>> > > have
>> > > > an
>> > > > > > > easy
>> > > > > > > > > way to tell whether zstandard is in use (the topic has it
>> > > > > configured
>> > > > > > > > > explicitly). In the latter case, we don't have to handle
>> it
>> > > > > > specially.
>> > > > > > > > But
>> > > > > > > > > we do have to decide how we will handle down-conversion to
>> > > older
>> > > > > > > formats.
>> > > > > > > > >
>> > > > > > > > > -Jason
>> > > > > > > > >
>> > > > > > > > > On Sun, Aug 12, 2018 at 5:15 PM, Dongjin Lee <
>> > > dong...@apache.org
>> > > > >
>> > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Colin and Jason,
>> > > > > > > > > >
>> > > > > > > > > > Thanks for your opinions. In summarizing, the Pros and
>> Cons
>> > > of
>> > > > > > > bumping
>> > > > > > > > > > fetch API version are:
>> > > > > > > > > >
>> > > > > > > > > > Cons:
>> > > > > > > > > >
>> > > > > > > > > > - The Broker can't know whether a given message batch is
>> > > > > compressed
>> > > > > > > > with
>> > > > > > > > > > zstd or not.
>> > > > > > > > > > - Need some additional logic for the topic explicitly
>> > > > configured
>> > > > > to
>> > > > > > > use
>> > > > > > > > > > zstd.
>> > > > > > > > > >
>> > > > > > > > > > Pros:
>> > > > > > > > > >
>> > > > > > > > > > - The broker doesn't need to conduct expensive
>> > > down-conversion.
>> > > > > > > > > > - Can message the users to update their client.
>> > > > > > > > > >
>> > > > > > > > > > So, opinions for the backward-compatibility policy by
>> far:
>> > > > > > > > > >
>> > > > > > > > > > - A: bump the API version - +2 (Colin, Jason)
>> > > > > > > > > > - B: leave unchanged - +1 (Viktor)
>> > > > > > > > > >
>> > > > > > > > > > Here are my additional comments:
>> > > > > > > > > >
>> > > > > > > > > > @Colin
>> > > > > > > > > >
>> > > > > > > > > > I greatly appreciate your response. In the case of the
>> > > > dictionary
>> > > > > > > > > support,
>> > > > > > > > > > of course, this issue should be addressed later so we
>> don't
>> > > > need
>> > > > > it
>> > > > > > > in
>> > > > > > > > > the
>> > > > > > > > > > first version. You are right - it is not late to try it
>> > after
>> > > > > some
>> > > > > > > > > > benchmarks. What I mean is, we should keep in mind on
>> that
>> > > > > > potential
>> > > > > > > > > > feature.
>> > > > > > > > > >
>> > > > > > > > > > @Jason
>> > > > > > > > > >
>> > > > > > > > > > You wrote,
>> > > > > > > > > >
>> > > > > > > > > > > Similarly, if we have to down-convert anyway because
>> the
>> > > > client
>> > > > > > > does
>> > > > > > > > > not
>> > > > > > > > > > understand the message format, then we could also use a
>> > > > different
>> > > > > > > > > > compression type.
>> > > > > > > > > >
>> > > > > > > > > > But in my opinion, since the client will fail with the
>> API
>> > > > > version,
>> > > > > > > so
>> > > > > > > > we
>> > > > > > > > > > don't need to down-convert the messages anyway. Isn't
>> it?
>> > > So, I
>> > > > > > think
>> > > > > > > > we
>> > > > > > > > > > don't care about this case. (I'm sorry, I am not
>> familiar
>> > > with
>> > > > > > > > > down-convert
>> > > > > > > > > > logic.)
>> > > > > > > > > >
>> > > > > > > > > > Please give more opinions. Thanks!
>> > > > > > > > > >
>> > > > > > > > > > - Dongjin
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Wed, Aug 8, 2018 at 6:41 AM Jason Gustafson <
>> > > > > ja...@confluent.io
>> > > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hey Colin,
>> > > > > > > > > > >
>> > > > > > > > > > > The problem for the fetch API is that the broker does
>> not
>> > > > > > generally
>> > > > > > > > > know
>> > > > > > > > > > if
>> > > > > > > > > > > a batch was compressed with zstd unless it parses it.
>> I
>> > > think
>> > > > > the
>> > > > > > > > goal
>> > > > > > > > > > here
>> > > > > > > > > > > is to avoid the expensive down-conversion that is
>> needed
>> > to
>> > > > > > ensure
>> > > > > > > > > > > compatibility because it is only necessary if zstd is
>> > > > actually
>> > > > > in
>> > > > > > > > use.
>> > > > > > > > > > But
>> > > > > > > > > > > as long as old clients can parse the message format,
>> they
>> > > > > should
>> > > > > > > get
>> > > > > > > > a
>> > > > > > > > > > > reasonable error if they see an unsupported
>> compression
>> > > type
>> > > > in
>> > > > > > the
>> > > > > > > > > > > attributes. Basically the onus is on users to ensure
>> that
>> > > > their
>> > > > > > > > > consumers
>> > > > > > > > > > > have been updated prior to using zstd. It seems like a
>> > > > > reasonable
>> > > > > > > > > > tradeoff
>> > > > > > > > > > > to me. There are a couple cases that might be worth
>> > > thinking
>> > > > > > > through:
>> > > > > > > > > > >
>> > > > > > > > > > > 1. If a topic is explicitly configured to use zstd,
>> then
>> > we
>> > > > > don't
>> > > > > > > > need
>> > > > > > > > > to
>> > > > > > > > > > > check the fetched data for the compression type to
>> know
>> > if
>> > > we
>> > > > > > need
>> > > > > > > > > > > down-conversion. If we did bump the Fetch API version,
>> > then
>> > > > we
>> > > > > > > could
>> > > > > > > > > > handle
>> > > > > > > > > > > this case by either down-converting using a different
>> > > > > compression
>> > > > > > > > type
>> > > > > > > > > or
>> > > > > > > > > > > returning an error.
>> > > > > > > > > > > 2. Similarly, if we have to down-convert anyway
>> because
>> > the
>> > > > > > client
>> > > > > > > > does
>> > > > > > > > > > not
>> > > > > > > > > > > understand the message format, then we could also use
>> a
>> > > > > different
>> > > > > > > > > > > compression type.
>> > > > > > > > > > >
>> > > > > > > > > > > For the produce API, I think it's reasonable to bump
>> the
>> > > api
>> > > > > > > version.
>> > > > > > > > > > This
>> > > > > > > > > > > can be used by clients to check whether a broker
>> supports
>> > > > zstd.
>> > > > > > For
>> > > > > > > > > > > example, we might support a list of preferred
>> compression
>> > > > types
>> > > > > > in
>> > > > > > > > the
>> > > > > > > > > > > producer and we could use the broker to detect which
>> > > version
>> > > > to
>> > > > > > > use.
>> > > > > > > > > > >
>> > > > > > > > > > > -Jason
>> > > > > > > > > > >
>> > > > > > > > > > > On Tue, Aug 7, 2018 at 1:32 PM, Colin McCabe <
>> > > > > cmcc...@apache.org
>> > > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Thanks for bumping this, Dongjin. ZStd is a good
>> > > > compression
>> > > > > > > codec
>> > > > > > > > > > and I
>> > > > > > > > > > > > hope we can get this support in soon!
>> > > > > > > > > > > >
>> > > > > > > > > > > > I would say we can just bump the API version to
>> > indicate
>> > > > that
>> > > > > > > ZStd
>> > > > > > > > > > > support
>> > > > > > > > > > > > is expected in new clients. We probably need some
>> way
>> > of
>> > > > > > > > indicating
>> > > > > > > > > to
>> > > > > > > > > > > the
>> > > > > > > > > > > > older clients that they can't consume the
>> partitions,
>> > as
>> > > > > well.
>> > > > > > > > > Perhaps
>> > > > > > > > > > > we
>> > > > > > > > > > > > can use the UNSUPPORTED_FOR_MESSAGE_FORMAT error?
>> > > > > > > > > > > >
>> > > > > > > > > > > > The license thing seems straightforward -- it's
>> just a
>> > > > matter
>> > > > > > of
>> > > > > > > > > adding
>> > > > > > > > > > > > the text to the right files as per ASF guidelines.
>> > > > > > > > > > > >
>> > > > > > > > > > > > With regard to the dictionary support, do we really
>> > need
>> > > > that
>> > > > > > in
>> > > > > > > > the
>> > > > > > > > > > > first
>> > > > > > > > > > > > version? Hopefully message batches are big enough
>> that
>> > > > this
>> > > > > > > isn't
>> > > > > > > > > > > needed.
>> > > > > > > > > > > > Some benchmarks might help here.
>> > > > > > > > > > > >
>> > > > > > > > > > > > best,
>> > > > > > > > > > > > Colin
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Tue, Aug 7, 2018, at 08:02, Dongjin Lee wrote:
>> > > > > > > > > > > > > As Kafka 2.0.0 was released, let's reboot this
>> issue,
>> > > > > KIP-110
>> > > > > > > > > > > > > <https://cwiki.apache.org/
>> > > confluence/display/KAFKA/KIP-
>> > > > > > > > > > > > 110%3A+Add+Codec+for+ZStandard+Compression>
>> > > > > > > > > > > > > .
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > For newcomers, Here is some summary of the
>> history:
>> > > > KIP-110
>> > > > > > was
>> > > > > > > > > > > > originally
>> > > > > > > > > > > > > worked for the issue KAFKA-4514 but, it lacked
>> > > benchmark
>> > > > > > > results
>> > > > > > > > to
>> > > > > > > > > > get
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > agreement of the community. Later, Ivan Babrou and
>> > some
>> > > > > other
>> > > > > > > > users
>> > > > > > > > > > who
>> > > > > > > > > > > > > adopted the patch provided their excellent
>> > performance
>> > > > > report
>> > > > > > > > which
>> > > > > > > > > > is
>> > > > > > > > > > > > now
>> > > > > > > > > > > > > included in the KIP, but it postponed again
>> because
>> > of
>> > > > the
>> > > > > > > > > community
>> > > > > > > > > > > was
>> > > > > > > > > > > > > busy for 2.0.0 release. It is why I now reboot
>> this
>> > > > issue.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > The following is the current status of the
>> feature:
>> > You
>> > > > can
>> > > > > > > check
>> > > > > > > > > the
>> > > > > > > > > > > > > current draft implementation here
>> > > > > > > > > > > > > <https://github.com/apache/kafka/pull/2267>. It
>> is
>> > > based
>> > > > > on
>> > > > > > > zstd
>> > > > > > > > > > 1.3.5
>> > > > > > > > > > > > and
>> > > > > > > > > > > > > periodically rebased onto the latest trunk[^1].
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > The issues that should be addressed is like
>> > following:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > *1. Backward Compatibility*
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > To support old consumers, we need to take a
>> strategy
>> > to
>> > > > > > handle
>> > > > > > > > the
>> > > > > > > > > > old
>> > > > > > > > > > > > > consumers. Current candidates are:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > - Bump API version
>> > > > > > > > > > > > > - Leave unchanged: let the old clients fail.
>> > > > > > > > > > > > > - Improve the error messages:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > *2. Dictionary Support*
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > To support zstd's dictionary feature in the future
>> > (if
>> > > > > > needed),
>> > > > > > > > we
>> > > > > > > > > > need
>> > > > > > > > > > > > to
>> > > > > > > > > > > > > sketch how it should be and leave some room for
>> it.
>> > As
>> > > of
>> > > > > > now,
>> > > > > > > > > there
>> > > > > > > > > > > has
>> > > > > > > > > > > > > been no discussion on this topic yet.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > *3. License*
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > To use this feature, we need to add license of
>> zstd
>> > and
>> > > > > > > zstd-jni
>> > > > > > > > to
>> > > > > > > > > > the
>> > > > > > > > > > > > > project. (Thanks to Viktor Somogyi for raising
>> this
>> > > > issue!)
>> > > > > > It
>> > > > > > > > > seems
>> > > > > > > > > > > like
>> > > > > > > > > > > > > what Apache Spark did would be a good example but
>> > there
>> > > > has
>> > > > > > > been
>> > > > > > > > no
>> > > > > > > > > > > > > discussion yet.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > You can find the details of the above issues in
>> the
>> > KIP
>> > > > > > > document.
>> > > > > > > > > > > Please
>> > > > > > > > > > > > > have a look when you are free, and give me
>> feedback.
>> > > All
>> > > > > > kinds
>> > > > > > > of
>> > > > > > > > > > > > > participating are welcome.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > Dongjin
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > [^1]: At the time of writing, commit 6b4fb8152.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On Sat, Jul 14, 2018 at 10:45 PM Dongjin Lee <
>> > > > > > > dong...@apache.org
>> > > > > > > > >
>> > > > > > > > > > > wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > Sorry for the late reply.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > In short, I could not submit the updated KIP by
>> the
>> > > > > feature
>> > > > > > > > > freeze
>> > > > > > > > > > > > > > deadline of 2.0.0. For this reason, it will not
>> be
>> > > > > included
>> > > > > > > in
>> > > > > > > > > the
>> > > > > > > > > > > > 2.0.0
>> > > > > > > > > > > > > > release and all discussion for this issue were
>> > > > postponed
>> > > > > > > after
>> > > > > > > > > the
>> > > > > > > > > > > > release
>> > > > > > > > > > > > > > of 2.0.0.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > I have been updating the PR following recent
>> > updates.
>> > > > > Just
>> > > > > > > > now, I
>> > > > > > > > > > > > rebased
>> > > > > > > > > > > > > > it against the latest trunk and updated the zstd
>> > > > version
>> > > > > > into
>> > > > > > > > > > 1.3.5.
>> > > > > > > > > > > > If you
>> > > > > > > > > > > > > > need some request, don't hesitate to notify me.
>> > (But
>> > > > not
>> > > > > > this
>> > > > > > > > > > thread
>> > > > > > > > > > > -
>> > > > > > > > > > > > just
>> > > > > > > > > > > > > > send me the message directly.)
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > > Dongjin
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > On Tue, Jul 10, 2018 at 11:57 PM Bobby Evans <
>> > > > > > > bo...@apache.org
>> > > > > > > > >
>> > > > > > > > > > > wrote:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >> I there any update on this. The performance
>> > > > > improvements
>> > > > > > > are
>> > > > > > > > > > quite
>> > > > > > > > > > > > > >> impressive and I really would like to stop
>> forking
>> > > > kafka
>> > > > > > > just
>> > > > > > > > to
>> > > > > > > > > > get
>> > > > > > > > > > > > this
>> > > > > > > > > > > > > >> in.
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> Thanks,
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> Bobby
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> On Wed, Jun 13, 2018 at 8:56 PM Dongjin Lee <
>> > > > > > > > dong...@apache.org
>> > > > > > > > > >
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> > Ismael,
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> > Oh, I forgot all of you are on working frenzy
>> > for
>> > > > 2.0!
>> > > > > > No
>> > > > > > > > > > problem,
>> > > > > > > > > > > > take
>> > > > > > > > > > > > > >> > your time. I am also working at another issue
>> > now.
>> > > > > Thank
>> > > > > > > you
>> > > > > > > > > for
>> > > > > > > > > > > > > >> letting me
>> > > > > > > > > > > > > >> > know.
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> > Best,
>> > > > > > > > > > > > > >> > Dongjin
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> > On Wed, Jun 13, 2018, 11:44 PM Ismael Juma <
>> > > > > > > > isma...@gmail.com
>> > > > > > > > > >
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> > > Sorry for the delay Dongjin. Everyone is
>> busy
>> > > > > > finalising
>> > > > > > > > > > 2.0.0.
>> > > > > > > > > > > > This
>> > > > > > > > > > > > > >> KIP
>> > > > > > > > > > > > > >> > > seems like a great candidate for 2.1.0 and
>> > > > hopefully
>> > > > > > > there
>> > > > > > > > > > will
>> > > > > > > > > > > be
>> > > > > > > > > > > > > >> more
>> > > > > > > > > > > > > >> > of
>> > > > > > > > > > > > > >> > > a discussion next week. :)
>> > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > >> > > Ismael
>> > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > >> > > On Wed, 13 Jun 2018, 05:17 Dongjin Lee, <
>> > > > > > > > dong...@apache.org
>> > > > > > > > > >
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > >> > > > Hello. I just updated my draft
>> > implementation:
>> > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > >> > > > 1. Rebased to latest trunk (commit
>> 5145d6b)
>> > > > > > > > > > > > > >> > > > 2. Apply ZStd 1.3.4
>> > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > >> > > > You can check out the implementation from
>> > here
>> > > > > > > > > > > > > >> > > > <
>> https://github.com/apache/kafka/pull/2267
>> > >.
>> > > If
>> > > > > you
>> > > > > > > > > > > experience
>> > > > > > > > > > > > any
>> > > > > > > > > > > > > >> > > problem
>> > > > > > > > > > > > > >> > > > running it, don't hesitate to give me a
>> > > mention.
>> > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > >> > > > Best,
>> > > > > > > > > > > > > >> > > > Dongjin
>> > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > >> > > > On Tue, Jun 12, 2018 at 6:50 PM Dongjin
>> Lee
>> > <
>> > > > > > > > > > > dong...@apache.org
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > >> > wrote:
>> > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > >> > > > > Here is the short conclusion about the
>> > > license
>> > > > > > > > problem:
>> > > > > > > > > > *We
>> > > > > > > > > > > > can
>> > > > > > > > > > > > > >> use
>> > > > > > > > > > > > > >> > > zstd
>> > > > > > > > > > > > > >> > > > > and zstd-jni without any problem, but
>> we
>> > > need
>> > > > to
>> > > > > > > > include
>> > > > > > > > > > > their
>> > > > > > > > > > > > > >> > license,
>> > > > > > > > > > > > > >> > > > > e.g., BSD license.*
>> > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > >> > > > > Both of BSD 2 Clause License & 3 Clause
>> > > > License
>> > > > > > > > requires
>> > > > > > > > > > to
>> > > > > > > > > > > > > >> include
>> > > > > > > > > > > > > >> > the
>> > > > > > > > > > > > > >> > > > > license used, and BSD 3 Clause License
>> > > > requires
>> > > > > > that
>> > > > > > > > the
>> > > > > > > > > > > name
>> > > > > > > > > > > > of
>> > > > > > > > > > > > > >> the
>> > > > > > > > > > > > > >> > > > > contributor can't be used to endorse or
>> > > > promote
>> > > > > > the
>> > > > > > > > > > product.
>> > > > > > > > > > > > > >> That's
>> > > > > > > > > > > > > >> > it
>> > > > > > > > > > > > > >> > > > > <
>> > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >>
>> > http://www.mikestratton.net/2011/12/is-bsd-license-
>> > > > > > > > > > > > compatible-with-apache-2-0-license/
>> > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > >> > > > > - They are not listed in the list of
>> > > > prohibited
>> > > > > > > > licenses
>> > > > > > > > > > > > > >> > > > > <https://www.apache.org/legal/
>> > > > > > > > resolved.html#category-x>
>> > > > > > > > > > > also.
>> > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > >> > > > > Here is how Spark did for it
>> > > > > > > > > > > > > >> > > > > <https://issues.apache.org/
>> > > > > > jira/browse/SPARK-19112
>> > > > > > > >:
>> > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > >> > > > > - They made a directory dedicated to
>> the
>> > > > > > dependency
>> > > > > > > > > > license
>> > > > > > > > > > > > files
>> > > > > > > > > > > > > >> > > > > <
>> > > > > > > https://github.com/apache/spark/tree/master/licenses
>> > > > > > > > >
>> > > > > > > > > > and
>> > > > > > > > > > > > added
>> > > > > > > > > > > > > >> > > > licenses
>> > > > > > > > > > > > > >> > > > > for Zstd
>> > > > > > > > > > > > > >> > > > > <
>> > > > > > > > > > > > > >> >
>> > > > https://github.com/apache/spark/blob/master/licenses/
>> > > > > > > > > > > > LICENSE-zstd.txt
>> > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > >> > > > &
>> > > > > > > > > > > > > >> > > > > Zstd-jni
>> > > > > > > > > > > > > >> > > > > <
>> > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> https://github.com/apache/
>> > > spark/blob/master/licenses/
>> > > > > > > > > > > > LICENSE-zstd-jni.txt
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >> > > > > .
>> > > > > > > > > > > > > >> > > > > - Added a link to the original license
>> > files
>> > > > in
>> > > > > > > > LICENSE.
>> > > > > > > > > > > > > >> > > > > <
>> > > > > https://github.com/apache/spark/pull/18805/files
>> > > > > > >
>> > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > >> > > > > If needed, I can make a similar update.
>> > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > >> > > > > Thanks for pointing out this problem,
>> > > Viktor!
>> > > > > Nice
>> > > > > > > > > catch!
>> > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > >> > > > > Best,
>> > > > > > > > > > > > > >> > > > > Dongjin
>> > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > >> > > > > On Mon, Jun 11, 2018 at 11:50 PM
>> Dongjin
>> > > Lee <
>> > > > > > > > > > > > dong...@apache.org>
>> > > > > > > > > > > > > >> > > wrote:
>> > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > >> > > > >> I greatly appreciate your
>> comprehensive
>> > > > > > reasoning.
>> > > > > > > > so:
>> > > > > > > > > +1
>> > > > > > > > > > > > for b
>> > > > > > > > > > > > > >> > until
>> > > > > > > > > > > > > >> > > > now.
>> > > > > > > > > > > > > >> > > > >>
>> > > > > > > > > > > > > >> > > > >> For the license issues, I will have a
>> > check
>> > > > on
>> > > > > > how
>> > > > > > > > the
>> > > > > > > > > > over
>> > > > > > > > > > > > > >> projects
>> > > > > > > > > > > > > >> > > are
>> > > > > > > > > > > > > >> > > > >> doing and share the results.
>> > > > > > > > > > > > > >> > > > >>
>> > > > > > > > > > > > > >> > > > >> Best,
>> > > > > > > > > > > > > >> > > > >> Dongjin
>> > > > > > > > > > > > > >> > > > >>
>> > > > > > > > > > > > > >> > > > >> On Mon, Jun 11, 2018 at 10:08 PM
>> Viktor
>> > > > > Somogyi <
>> > > > > > > > > > > > > >> > > > viktorsomo...@gmail.com>
>> > > > > > > > > > > > > >> > > > >> wrote:
>> > > > > > > > > > > > > >> > > > >>
>> > > > > > > > > > > > > >> > > > >>> Hi Dongjin,
>> > > > > > > > > > > > > >> > > > >>>
>> > > > > > > > > > > > > >> > > > >>> A couple of comments:
>> > > > > > > > > > > > > >> > > > >>> I would vote for option b. in the
>> > > "backward
>> > > > > > > > > > compatibility"
>> > > > > > > > > > > > > >> section.
>> > > > > > > > > > > > > >> > > My
>> > > > > > > > > > > > > >> > > > >>> reasoning for this is that users
>> > upgrading
>> > > > to
>> > > > > a
>> > > > > > > zstd
>> > > > > > > > > > > > compatible
>> > > > > > > > > > > > > >> > > version
>> > > > > > > > > > > > > >> > > > >>> won't start to use it automatically,
>> so
>> > > > manual
>> > > > > > > > > > > > reconfiguration
>> > > > > > > > > > > > > >> is
>> > > > > > > > > > > > > >> > > > >>> required.
>> > > > > > > > > > > > > >> > > > >>> Therefore an upgrade won't mess up
>> the
>> > > > > cluster.
>> > > > > > If
>> > > > > > > > not
>> > > > > > > > > > all
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > >> > > clients
>> > > > > > > > > > > > > >> > > > >>> are
>> > > > > > > > > > > > > >> > > > >>> upgraded but just some of them and
>> > they'd
>> > > > > start
>> > > > > > to
>> > > > > > > > use
>> > > > > > > > > > > zstd
>> > > > > > > > > > > > > >> then it
>> > > > > > > > > > > > > >> > > > would
>> > > > > > > > > > > > > >> > > > >>> cause errors in the cluster. I'd
>> like to
>> > > > > presume
>> > > > > > > > > though
>> > > > > > > > > > > that
>> > > > > > > > > > > > > >> this
>> > > > > > > > > > > > > >> > is
>> > > > > > > > > > > > > >> > > a
>> > > > > > > > > > > > > >> > > > >>> very
>> > > > > > > > > > > > > >> > > > >>> obvious failure case and nobody
>> should
>> > be
>> > > > > > > surprised
>> > > > > > > > if
>> > > > > > > > > > it
>> > > > > > > > > > > > didn't
>> > > > > > > > > > > > > >> > > work.
>> > > > > > > > > > > > > >> > > > >>> I wouldn't choose a. as I think we
>> > should
>> > > > bump
>> > > > > > the
>> > > > > > > > > fetch
>> > > > > > > > > > > and
>> > > > > > > > > > > > > >> > produce
>> > > > > > > > > > > > > >> > > > >>> requests if it's a change in the
>> message
>> > > > > format.
>> > > > > > > > > > Moreover
>> > > > > > > > > > > if
>> > > > > > > > > > > > > >> some
>> > > > > > > > > > > > > >> > of
>> > > > > > > > > > > > > >> > > > the
>> > > > > > > > > > > > > >> > > > >>> producers and the brokers are
>> upgraded
>> > but
>> > > > > some
>> > > > > > of
>> > > > > > > > the
>> > > > > > > > > > > > consumers
>> > > > > > > > > > > > > >> > are
>> > > > > > > > > > > > > >> > > > not,
>> > > > > > > > > > > > > >> > > > >>> then we wouldn't prevent the error
>> when
>> > > the
>> > > > > old
>> > > > > > > > > consumer
>> > > > > > > > > > > > tries
>> > > > > > > > > > > > > >> to
>> > > > > > > > > > > > > >> > > > consume
>> > > > > > > > > > > > > >> > > > >>> the zstd compressed messages.
>> > > > > > > > > > > > > >> > > > >>> I wouldn't choose c. either as I
>> think
>> > > > binding
>> > > > > > the
>> > > > > > > > > > > > compression
>> > > > > > > > > > > > > >> type
>> > > > > > > > > > > > > >> > > to
>> > > > > > > > > > > > > >> > > > an
>> > > > > > > > > > > > > >> > > > >>> API is not so obvious from the
>> > developer's
>> > > > > > > > > perspective.
>> > > > > > > > > > > > > >> > > > >>>
>> > > > > > > > > > > > > >> > > > >>> I would also prefer to use the
>> existing
>> > > > > binding,
>> > > > > > > > > however
>> > > > > > > > > > > we
>> > > > > > > > > > > > must
>> > > > > > > > > > > > > >> > > > respect
>> > > > > > > > > > > > > >> > > > >>> the licenses:
>> > > > > > > > > > > > > >> > > > >>> "The code for these JNI bindings is
>> > > licenced
>> > > > > > under
>> > > > > > > > > > > 2-clause
>> > > > > > > > > > > > BSD
>> > > > > > > > > > > > > >> > > > license.
>> > > > > > > > > > > > > >> > > > >>> The native Zstd library is licensed
>> > under
>> > > > > > 3-clause
>> > > > > > > > BSD
>> > > > > > > > > > > > license
>> > > > > > > > > > > > > >> and
>> > > > > > > > > > > > > >> > > > GPL2"
>> > > > > > > > > > > > > >> > > > >>> Based on the FAQ page
>> > > > > > > > > > > > > >> > > > >>> https://www.apache.org/legal/
>> > > > > > > > resolved.html#category-a
>> > > > > > > > > > > > > >> > > > >>> we may use 2- and 3-clause BSD
>> licenses
>> > > but
>> > > > > the
>> > > > > > > > Apache
>> > > > > > > > > > > > license
>> > > > > > > > > > > > > >> is
>> > > > > > > > > > > > > >> > not
>> > > > > > > > > > > > > >> > > > >>> compatible with GPL2. I'm hoping that
>> > the
>> > > > > > > "3-clause
>> > > > > > > > > BSD
>> > > > > > > > > > > > license
>> > > > > > > > > > > > > >> and
>> > > > > > > > > > > > > >> > > > GPL2"
>> > > > > > > > > > > > > >> > > > >>> is really not an AND but an OR in
>> this
>> > > case,
>> > > > > but
>> > > > > > > I'm
>> > > > > > > > > no
>> > > > > > > > > > > > lawyer,
>> > > > > > > > > > > > > >> > just
>> > > > > > > > > > > > > >> > > > >>> wanted
>> > > > > > > > > > > > > >> > > > >>> to make the point that we should
>> watch
>> > out
>> > > > for
>> > > > > > > > > licenses.
>> > > > > > > > > > > :)
>> > > > > > > > > > > > > >> > > > >>>
>> > > > > > > > > > > > > >> > > > >>> Regards,
>> > > > > > > > > > > > > >> > > > >>> Viktor
>> > > > > > > > > > > > > >> > > > >>>
>> > > > > > > > > > > > > >> > > > >>>
>> > > > > > > > > > > > > >> > > > >>> On Sun, Jun 10, 2018 at 3:02 AM Ivan
>> > > Babrou
>> > > > <
>> > > > > > > > > > > > ibob...@gmail.com>
>> > > > > > > > > > > > > >> > > wrote:
>> > > > > > > > > > > > > >> > > > >>>
>> > > > > > > > > > > > > >> > > > >>> > Hello,
>> > > > > > > > > > > > > >> > > > >>> >
>> > > > > > > > > > > > > >> > > > >>> > This is Ivan and I still very much
>> > > support
>> > > > > the
>> > > > > > > > fact
>> > > > > > > > > > that
>> > > > > > > > > > > > zstd
>> > > > > > > > > > > > > >> > > > >>> compression
>> > > > > > > > > > > > > >> > > > >>> > should be included out of the box.
>> > > > > > > > > > > > > >> > > > >>> >
>> > > > > > > > > > > > > >> > > > >>> > Please think about the environment,
>> > you
>> > > > can
>> > > > > > save
>> > > > > > > > > > quite a
>> > > > > > > > > > > > lot
>> > > > > > > > > > > > > >> of
>> > > > > > > > > > > > > >> > > > >>> hardware
>> > > > > > > > > > > > > >> > > > >>> > with it.
>> > > > > > > > > > > > > >> > > > >>> >
>> > > > > > > > > > > > > >> > > > >>> > Thank you.
>> > > > > > > > > > > > > >> > > > >>> >
>> > > > > > > > > > > > > >> > > > >>> > On Sat, Jun 9, 2018 at 14:14
>> Dongjin
>> > > Lee <
>> > > > > > > > > > > > dong...@apache.org>
>> > > > > > > > > > > > > >> > > wrote:
>> > > > > > > > > > > > > >> > > > >>> >
>> > > > > > > > > > > > > >> > > > >>> > > Since there are no responses for
>> a
>> > > > week, I
>> > > > > > > > decided
>> > > > > > > > > > to
>> > > > > > > > > > > > > >> > reinitiate
>> > > > > > > > > > > > > >> > > > the
>> > > > > > > > > > > > > >> > > > >>> > > discussion thread.
>> > > > > > > > > > > > > >> > > > >>> > >
>> > > > > > > > > > > > > >> > > > >>> > >
>> > > > > > > > > > > > > >> > > > >>> > >
>> > > > > > > > > > > > > >> > > > >>> >
>> > > > > > > > > > > > > >> > > > >>>
>> > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >>
>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > > > > > > > > > > 110%3A+Add+Codec+for+ZStandard+Compression
>> > > > > > > > > > > > > >> > > > >>> > >
>> > > > > > > > > > > > > >> > > > >>> > > This KIP is about to introduce
>> > > ZStandard
>> > > > > > > > > Compression
>> > > > > > > > > > > > into
>> > > > > > > > > > > > > >> > Apache
>> > > > > > > > > > > > > >> > > > >>> Kafka.
>> > > > > > > > > > > > > >> > > > >>> > > The reason why it is posted again
>> > has
>> > > a
>> > > > > > story:
>> > > > > > > > It
>> > > > > > > > > > was
>> > > > > > > > > > > > > >> > originally
>> > > > > > > > > > > > > >> > > > >>> posted
>> > > > > > > > > > > > > >> > > > >>> > to
>> > > > > > > > > > > > > >> > > > >>> > > the dev mailing list more than
>> one
>> > > year
>> > > > > ago
>> > > > > > > but
>> > > > > > > > > > since
>> > > > > > > > > > > it
>> > > > > > > > > > > > > >> has no
>> > > > > > > > > > > > > >> > > > >>> > performance
>> > > > > > > > > > > > > >> > > > >>> > > report included, it was postponed
>> > > later.
>> > > > > But
>> > > > > > > > Some
>> > > > > > > > > > > people
>> > > > > > > > > > > > > >> > > (including
>> > > > > > > > > > > > > >> > > > >>> Ivan)
>> > > > > > > > > > > > > >> > > > >>> > > reported excellent performance
>> > report
>> > > > with
>> > > > > > the
>> > > > > > > > > draft
>> > > > > > > > > > > PR,
>> > > > > > > > > > > > > >> this
>> > > > > > > > > > > > > >> > > work
>> > > > > > > > > > > > > >> > > > >>> is now
>> > > > > > > > > > > > > >> > > > >>> > > reactivated.
>> > > > > > > > > > > > > >> > > > >>> > >
>> > > > > > > > > > > > > >> > > > >>> > > The updated KIP document includes
>> > some
>> > > > > > > expected
>> > > > > > > > > > > > problems and
>> > > > > > > > > > > > > >> > > their
>> > > > > > > > > > > > > >> > > > >>> > > candidate alternatives. Please
>> have
>> > a
>> > > > look
>> > > > > > > when
>> > > > > > > > > you
>> > > > > > > > > > > are
>> > > > > > > > > > > > > >> free,
>> > > > > > > > > > > > > >> > and
>> > > > > > > > > > > > > >> > > > >>> give
>> > > > > > > > > > > > > >> > > > >>> > me a
>> > > > > > > > > > > > > >> > > > >>> > > feedback. All kinds of
>> participating
>> > > are
>> > > > > > > > welcome.
>> > > > > > > > > > > > > >> > > > >>> > >
>> > > > > > > > > > > > > >> > > > >>> > > Best,
>> > > > > > > > > > > > > >> > > > >>> > > Dongjin
>> > > > > > > > > > > > > >> > > > >>> > >
>> > > > > > > > > > > > > >> > > > >>> > > --
>> > > > > > > > > > > > > >> > > > >>> > > *Dongjin Lee*
>> > > > > > > > > > > > > >> > > > >>> > >
>> > > > > > > > > > > > > >> > > > >>> > > *A hitchhiker in the mathematical
>> > > > world.*
>> > > > > > > > > > > > > >> > > > >>> > >
>> > > > > > > > > > > > > >> > > > >>> > > *github: <
>> http://goog_969573159/
>> > > > >github
>> > > > > > > > > > > > .com/dongjinleekr
>> > > > > > > > > > > > > >> > > > >>> > > <http://github.com/dongjinleekr
>> > > > >linkedin:
>> > > > > > > > > > > > > >> > > > >>> > kr.linkedin.com/in/dongjinleekr
>> > > > > > > > > > > > > >> > > > >>> > > <http://kr.linkedin.com/in/
>> > > dongjinleekr
>> > > > > > > > > >slideshare:
>> > > > > > > > > > > > > >> > > > >>> > www.slideshare.net/dongjinleekr
>> > > > > > > > > > > > > >> > > > >>> > > <http://www.slideshare.net/
>> > > dongjinleekr
>> > > > >*
>> > > > > > > > > > > > > >> > > > >>> > >
>> > > > > > > > > > > > > >> > > > >>> >
>> > > > > > > > > > > > > >> > > > >>>
>> > > > > > > > > > > > > >> > > > >> --
>> > > > > > > > > > > > > >> > > > >> *Dongjin Lee*
>> > > > > > > > > > > > > >> > > > >>
>> > > > > > > > > > > > > >> > > > >> *A hitchhiker in the mathematical
>> world.*
>> > > > > > > > > > > > > >> > > > >>
>> > > > > > > > > > > > > >> > > > >> *github: <http://goog_969573159/
>> >github
>> > > > > > > > > > .com/dongjinleekr
>> > > > > > > > > > > > > >> > > > >> <http://github.com/dongjinleekr
>> > >linkedin:
>> > > > > > > > > > > > > >> > > > kr.linkedin.com/in/dongjinleekr
>> > > > > > > > > > > > > >> > > > >> <
>> http://kr.linkedin.com/in/dongjinleekr
>> > > > > > > >slideshare:
>> > > > > > > > > > > > > >> > > > www.slideshare.net/dongjinleekr
>> > > > > > > > > > > > > >> > > > >> <
>> http://www.slideshare.net/dongjinleekr
>> > >*
>> > > > > > > > > > > > > >> > > > >>
>> > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > >> > > > > --
>> > > > > > > > > > > > > >> > > > > *Dongjin Lee*
>> > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > >> > > > > *A hitchhiker in the mathematical
>> world.*
>> > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > >> > > > > *github: <http://goog_969573159/>
>> > > > > > > > > github.com/dongjinleekr
>> > > > > > > > > > > > > >> > > > > <http://github.com/dongjinleekr
>> >linkedin:
>> > > > > > > > > > > > > >> > > > kr.linkedin.com/in/dongjinleekr
>> > > > > > > > > > > > > >> > > > > <
>> http://kr.linkedin.com/in/dongjinleekr
>> > > > > > >slideshare:
>> > > > > > > > > > > > > >> > > > www.slideshare.net/dongjinleekr
>> > > > > > > > > > > > > >> > > > > <
>> http://www.slideshare.net/dongjinleekr>*
>> > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > >> > > > --
>> > > > > > > > > > > > > >> > > > *Dongjin Lee*
>> > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > >> > > > *A hitchhiker in the mathematical world.*
>> > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > >> > > > *github: <http://goog_969573159/>github
>> > > > > > > > .com/dongjinleekr
>> > > > > > > > > > > > > >> > > > <http://github.com/dongjinleekr
>> >linkedin:
>> > > > > > > > > > > > > >> > > kr.linkedin.com/in/dongjinleekr
>> > > > > > > > > > > > > >> > > > <http://kr.linkedin.com/in/dongjinleekr
>> > > > > >slideshare:
>> > > > > > > > > > > > > >> > > > www.slideshare.net/dongjinleekr
>> > > > > > > > > > > > > >> > > > <http://www.slideshare.net/dongjinleekr
>> >*
>> > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> --
>> > > > > > > > > > > > > >> *Dongjin Lee*
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> *A hitchhiker in the mathematical world.*
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >> *github: <http://goog_969573159/>
>> > > > > github.com/dongjinleekr
>> > > > > > > > > > > > > >> <http://github.com/dongjinleekr>linkedin:
>> > > > > > > kr.linkedin.com/in/
>> > > > > > > > > > > > dongjinleekr
>> > > > > > > > > > > > > >> <http://kr.linkedin.com/in/dongjinleekr
>> > >slideshare:
>> > > > > > > > > > > > www.slideshare.net/dongjinleekr
>> > > > > > > > > > > > > >> <http://www.slideshare.net/dongjinleekr>*
>> > > > > > > > > > > > > >>
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > --
>> > > > > > > > > > > > > *Dongjin Lee*
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > *A hitchhiker in the mathematical world.*
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > *github: <http://goog_969573159/>
>> > > > github.com/dongjinleekr
>> > > > > > > > > > > > > <http://github.com/dongjinleekr>linkedin:
>> > > > > > kr.linkedin.com/in/
>> > > > > > > > > > > > dongjinleekr
>> > > > > > > > > > > > > <http://kr.linkedin.com/in/dongjinleekr
>> >slideshare:
>> > > > > > > > > > > > > www.slideshare.net/dongjinleekr
>> > > > > > > > > > > > > <http://www.slideshare.net/dongjinleekr>*
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > --
>> > > > > > > > > > > *Dongjin Lee*
>> > > > > > > > > > >
>> > > > > > > > > > > *A hitchhiker in the mathematical world.*
>> > > > > > > > > > >
>> > > > > > > > > > > *github: <http://goog_969573159/>
>> > github.com/dongjinleekr
>> > > > > > > > > > > <http://github.com/dongjinleekr>linkedin:
>> > > > kr.linkedin.com/in/
>> > > > > > > > > > dongjinleekr
>> > > > > > > > > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
>> > > > > > > > > www.slideshare.net/
>> > > > > > > > > > dongjinleekr
>> > > > > > > > > > > <http://www.slideshare.net/dongjinleekr>*
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > > *Dongjin Lee*
>> > > > > > > > >
>> > > > > > > > > *A hitchhiker in the mathematical world.*
>> > > > > > > > >
>> > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr
>> > > > > > > > > <http://github.com/dongjinleekr>linkedin:
>> > kr.linkedin.com/in/
>> > > > > > > > dongjinleekr
>> > > > > > > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
>> > > > > > > www.slideshare.net/
>> > > > > > > > dongjinleekr
>> > > > > > > > > <http://www.slideshare.net/dongjinleekr>*
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > > --
>> > > > *Dongjin Lee*
>> > > >
>> > > > *A hitchhiker in the mathematical world.*
>> > > >
>> > > > *github: <http://goog_969573159/>github.com/dongjinleekr
>> > > > <http://github.com/dongji <http://github.com/dongjinleekr>
>
>
--
*Dongjin Lee*
*A hitchhiker in the mathematical world.*
*github: <http://goog_969573159/>github.com/dongjinleekr
<http://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr
<http://kr.linkedin.com/in/dongjinleekr>slideshare:
www.slideshare.net/dongjinleekr
<http://www.slideshare.net/dongjinleekr>*
