Hi Jason, You are right. Explicit statements are always better. I updated the document following your suggestion.
@Magnus Thanks for the inspection. It seems like a new error code is not a problem. Thanks, Dongjin On Fri, Sep 7, 2018 at 3:23 AM Jason Gustafson <ja...@confluent.io> wrote: > Hi Dongjin, > > The KIP looks good to me. I'd suggest starting a vote. A couple minor > points that might be worth calling out explicitly in the compatibility > section: > > 1. Zstd will only be allowed for the bumped produce API. For older > versions, we return UNSUPPORTED_COMPRESSION_TYPE regardless of the message > format. > 2. Down-conversion of zstd-compressed records will not be supported. > Instead we will return UNSUPPORTED_COMPRESSION_TYPE. > > Does that sound right? > > Thanks, > Jason > > On Thu, Sep 6, 2018 at 1:45 AM, Magnus Edenhill <mag...@edenhill.se> > wrote: > > > > Ismael wrote: > > > Jason, that's an interesting point regarding the Java client. Do we > know > > > what clients in other languages do in these cases? > > > > librdkafka (and its bindings) passes unknown/future errors through to the > > application, the error code remains intact while > > the error string will be set to something like "Err-123?", which isn't > very > > helpful to the user but it at least > > preserves the original error code for further troubleshooting. > > For the producer any unknown error returned in the ProduceResponse will > be > > considered a permanent delivery failure (no retries), > > and for the consumer any unknown FetchResponse errors will propagate > > directly to the application, trigger a fetch backoff, and then > > continue fetching past that offset. > > > > So, from the client's perspective it is not really a problem if new error > > codes are added to older API versions. > > > > /Magnus > > > > > > Den tors 6 sep. 2018 kl 09:45 skrev Dongjin Lee <dong...@apache.org>: > > > > > I updated the KIP page > > > < > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 110%3A+Add+Codec+for+ZStandard+Compression > > > > > > > following the discussion here. Please take a look when you are free. > > > If you have any opinion, don't hesitate to give me a message. > > > > > > Best, > > > Dongjin > > > > > > On Fri, Aug 31, 2018 at 11:35 PM Dongjin Lee <dong...@apache.org> > wrote: > > > > > > > I just updated the draft implementation[^1], rebasing against the > > latest > > > > trunk and implementing error routine (i.e., Error code 74 for > > > > UnsupportedCompressionTypeException.) Since we decided to disallow > all > > > > fetch request below version 2.1.0 for the topics specifying > ZStandard, > > I > > > > added an error logic only. > > > > > > > > Please have a look when you are free. > > > > > > > > Thanks, > > > > Dongjin > > > > > > > > [^1]: Please check the last commit here: > > > > https://github.com/apache/kafka/pull/2267 > > > > > > > > On Thu, Aug 23, 2018, 8:55 AM Dongjin Lee <dong...@apache.org> > wrote: > > > > > > > >> Jason, > > > >> > > > >> Great. +1 for UNSUPPORTED_COMPRESSION_TYPE. > > > >> > > > >> Best, > > > >> Dongjin > > > >> > > > >> On Thu, Aug 23, 2018 at 8:19 AM Jason Gustafson <ja...@confluent.io > > > > > >> wrote: > > > >> > > > >>> Hey Dongjin, > > > >>> > > > >>> Yeah that's right. For what it's worth, librdkafka also appears to > > > handle > > > >>> unexpected error codes. I expect that most client implementations > > would > > > >>> either pass through the raw type or convert to an enum using > > something > > > >>> like > > > >>> what the java client does. Since we're expecting the client to fail > > > >>> anyway, > > > >>> I'm probably in favor of using the UNSUPPORTED_COMPRESSION_TYPE > error > > > >>> code. > > > >>> > > > >>> -Jason > > > >>> > > > >>> On Wed, Aug 22, 2018 at 1:46 AM, Dongjin Lee <dong...@apache.org> > > > wrote: > > > >>> > > > >>> > Jason and Ismael, > > > >>> > > > > >>> > It seems like the only thing we need to regard if we define a new > > > error > > > >>> > code (i.e., UNSUPPORTED_COMPRESSION_TYPE) would be the > > implementation > > > >>> of > > > >>> > the other language clients, right? At least, this strategy causes > > any > > > >>> > problem for Java client. Do I understand correctly? > > > >>> > > > > >>> > Thanks, > > > >>> > Dongjin > > > >>> > > > > >>> > On Wed, Aug 22, 2018 at 5:43 PM Dongjin Lee <dong...@apache.org> > > > >>> wrote: > > > >>> > > > > >>> > > Jason, > > > >>> > > > > > >>> > > > I think we would only use this error code when we /know/ that > > > zstd > > > >>> was > > > >>> > > in use and the client doesn't support it? This is true if > either > > 1) > > > >>> the > > > >>> > > message needs down-conversion and we encounter a zstd > compressed > > > >>> message, > > > >>> > > or 2) if the topic is explicitly configured to use zstd. > > > >>> > > > > > >>> > > Yes, it is right. And you know, the case 1 includes 1.a) old > > > clients' > > > >>> > > request v0, v1 records or 1.b) implicit zstd, the compression > > type > > > of > > > >>> > > "producer" with Zstd compressed data. > > > >>> > > > > > >>> > > > However, if the compression type is set to "producer," then > the > > > >>> fetched > > > >>> > > data may or may not be compressed with zstd. In this case, we > > > return > > > >>> the > > > >>> > > data to the client and expect it to fail parsing. Is that > > correct? > > > >>> > > > > > >>> > > Exactly. > > > >>> > > > > > >>> > > Following your message, I reviewed the implementation of > > > >>> > > `KafkaApis#handleFetchRequest,` which handles the fetch > request. > > > And > > > >>> > found > > > >>> > > that the information we can use is like the following: > > > >>> > > > > > >>> > > 1. Client's fetch request version. (`versionId` variable) > > > >>> > > 2. Log's compression type. (`logConfig` variable) > > > >>> > > > > > >>> > > We can't detect the actual compression type of the data, unless > > we > > > >>> > inspect > > > >>> > > the `RecordBatch` included in the `Records` instance (i.e., > > > >>> > > `unconvertedRecords` variable.) Since it requires some > > performance > > > >>> issue, > > > >>> > > it is not our option - in short, we can't be sure if given > chunks > > > of > > > >>> data > > > >>> > > are compressed with zstd or not. > > > >>> > > > > > >>> > > So, conclusion: we can return an error in the case of 1.a and 2 > > > >>> easily, > > > >>> > > with the information above. In the case 1.b (implicit zstd), we > > can > > > >>> just > > > >>> > > return the data by do nothing special and expect it to fail > > > parsing. > > > >>> > > > > > >>> > > Thanks, > > > >>> > > Dongjin > > > >>> > > > > > >>> > > On Wed, Aug 22, 2018 at 12:02 PM Ismael Juma < > isma...@gmail.com> > > > >>> wrote: > > > >>> > > > > > >>> > >> Jason, that's an interesting point regarding the Java client. > Do > > > we > > > >>> know > > > >>> > >> what clients in other languages do in these cases? > > > >>> > >> > > > >>> > >> Ismael > > > >>> > >> > > > >>> > >> On Tue, 21 Aug 2018, 17:30 Jason Gustafson, < > ja...@confluent.io > > > > > > >>> wrote: > > > >>> > >> > > > >>> > >> > Hi Dongjin, > > > >>> > >> > > > > >>> > >> > One of the complications is that old versions of the API > will > > > not > > > >>> > >> expect a > > > >>> > >> > new error code. However, since we expect this to be a fatal > > > error > > > >>> > anyway > > > >>> > >> > for old clients, it may still be more useful to return the > > > correct > > > >>> > error > > > >>> > >> > code. For example, the Kafka clients use the following code > to > > > >>> convert > > > >>> > >> the > > > >>> > >> > error code: > > > >>> > >> > > > > >>> > >> > public static Errors forCode(short code) { > > > >>> > >> > Errors error = codeToError.get(code); > > > >>> > >> > if (error != null) { > > > >>> > >> > return error; > > > >>> > >> > } else { > > > >>> > >> > log.warn("Unexpected error code: {}.", code); > > > >>> > >> > return UNKNOWN_SERVER_ERROR; > > > >>> > >> > } > > > >>> > >> > } > > > >>> > >> > > > > >>> > >> > If we return an unsupported error code, it will be converted > > to > > > an > > > >>> > >> UNKNOWN > > > >>> > >> > error, but at least we will get the message in the log with > > the > > > >>> > correct > > > >>> > >> > code. That seems preferable to returning a misleading error > > > code. > > > >>> So I > > > >>> > >> > wonder if we can use the new UNSUPPORTED_COMPRESSION_TYPE > > error > > > >>> even > > > >>> > for > > > >>> > >> > older versions. > > > >>> > >> > > > > >>> > >> > Also, one question just to check my understanding. I think > we > > > >>> would > > > >>> > only > > > >>> > >> > use this error code when we /know/ that zstd was in use and > > the > > > >>> client > > > >>> > >> > doesn't support it? This is true if either 1) the message > > needs > > > >>> > >> > down-conversion and we encounter a zstd compressed message, > or > > > 2) > > > >>> if > > > >>> > the > > > >>> > >> > topic is explicitly configured to use zstd. However, if the > > > >>> > compression > > > >>> > >> > type is set to "producer," then the fetched data may or may > > not > > > be > > > >>> > >> > compressed with zstd. In this case, we return the data to > the > > > >>> client > > > >>> > and > > > >>> > >> > expect it to fail parsing. Is that correct? > > > >>> > >> > > > > >>> > >> > Thanks, > > > >>> > >> > Jason > > > >>> > >> > > > > >>> > >> > > > > >>> > >> > > > > >>> > >> > On Tue, Aug 21, 2018 at 9:08 AM, Dongjin Lee < > > > dong...@apache.org> > > > >>> > >> wrote: > > > >>> > >> > > > > >>> > >> > > Ismael, Jason and all, > > > >>> > >> > > > > > >>> > >> > > I rewrote the backward compatibility strategy & its > > > alternatives > > > >>> > like > > > >>> > >> > > following, based on Ismael & Jason's comments. Since it is > > not > > > >>> > >> updated to > > > >>> > >> > > the wiki yet, don't hesitate to give me a message if you > > have > > > >>> any > > > >>> > >> opinion > > > >>> > >> > > on it. > > > >>> > >> > > > > > >>> > >> > > ``` > > > >>> > >> > > *Backward Compatibility* > > > >>> > >> > > > > > >>> > >> > > We need to establish some backward-compatibility strategy > > for > > > >>> the > > > >>> > >> case an > > > >>> > >> > > old client subscribes a topic using ZStandard implicitly > > > (i.e., > > > >>> > >> > > 'compression.type' configuration of given topic is > > 'producer' > > > >>> and > > > >>> > the > > > >>> > >> > > producer compressed the records with ZStandard). We have > the > > > >>> > following > > > >>> > >> > > options for this situation: > > > >>> > >> > > > > > >>> > >> > > *A. Support ZStandard to the old clients which can > > understand > > > >>> v0, v1 > > > >>> > >> > > messages only.* > > > >>> > >> > > > > > >>> > >> > > This strategy necessarily requires the down-conversion of > v2 > > > >>> message > > > >>> > >> > > compressed with Zstandard into v0 or v1 messages, which > > means > > > a > > > >>> > >> > > considerable performance degradation. So we rejected this > > > >>> strategy. > > > >>> > >> > > > > > >>> > >> > > *B. Bump the API version and support only v2-available > > > clients* > > > >>> > >> > > > > > >>> > >> > > With this approach, we can message the old clients that > they > > > >>> are old > > > >>> > >> and > > > >>> > >> > > should be upgraded. However, there are still several > options > > > >>> for the > > > >>> > >> > Error > > > >>> > >> > > code. > > > >>> > >> > > > > > >>> > >> > > *B.1. INVALID_REQUEST (42)* > > > >>> > >> > > > > > >>> > >> > > This option gives the client so little information; the > user > > > >>> can be > > > >>> > >> > > confused about why the client worked correctly in the past > > > >>> suddenly > > > >>> > >> > > encounters a problem. So we rejected this strategy. > > > >>> > >> > > > > > >>> > >> > > *B.2. CORRUPT_MESSAGE (2)* > > > >>> > >> > > > > > >>> > >> > > This option gives inaccurate information; the user can be > > > >>> surprised > > > >>> > >> and > > > >>> > >> > > misunderstand that the log files are broken in some way. > So > > we > > > >>> > >> rejected > > > >>> > >> > > this strategy. > > > >>> > >> > > > > > >>> > >> > > *B.3 UNSUPPORTED_FOR_MESSAGE_FORMAT (43)* > > > >>> > >> > > > > > >>> > >> > > The advantage of this approach is we don't need to define > a > > > new > > > >>> > error > > > >>> > >> > code; > > > >>> > >> > > we can reuse it and that's all. > > > >>> > >> > > > > > >>> > >> > > The disadvantage of this approach is that it is also a > > little > > > >>> bit > > > >>> > >> vague; > > > >>> > >> > > This error code is defined as a work for KIP-98[^1] and > now > > > >>> returned > > > >>> > >> in > > > >>> > >> > the > > > >>> > >> > > transaction error. > > > >>> > >> > > > > > >>> > >> > > *B.4. UNSUPPORTED_COMPRESSION_TYPE (new)* > > > >>> > >> > > > > > >>> > >> > > The advantage of this approach is that it is clear and > > > provides > > > >>> an > > > >>> > >> exact > > > >>> > >> > > description. The disadvantage is we need to add a new > error > > > >>> code. > > > >>> > >> > > ``` > > > >>> > >> > > > > > >>> > >> > > *It seems like what we need to choose is now so clear: > > > >>> > >> > > UNSUPPORTED_FOR_MESSAGE_FORMAT (B.3) or > > > >>> UNSUPPORTED_COMPRESSION_TYPE > > > >>> > >> > > (B.4).* > > > >>> > >> > > The first one doesn't need a new error message but the > > latter > > > is > > > >>> > more > > > >>> > >> > > explicit. Which one do you prefer? Since all of you have > > much > > > >>> more > > > >>> > >> > > experience and knowledge than me, I will follow your > > decision. > > > >>> The > > > >>> > >> wiki > > > >>> > >> > > page will be updated following the decision also. > > > >>> > >> > > > > > >>> > >> > > Best, > > > >>> > >> > > Dongjin > > > >>> > >> > > > > > >>> > >> > > [^1]: https://issues.apache.org/jira/browse/KAFKA-4990 > > > >>> > >> > > > > > >>> > >> > > On Sun, Aug 19, 2018 at 4:58 AM Ismael Juma < > > > isma...@gmail.com> > > > >>> > >> wrote: > > > >>> > >> > > > > > >>> > >> > > > Sounds reasonable to me. > > > >>> > >> > > > > > > >>> > >> > > > Ismael > > > >>> > >> > > > > > > >>> > >> > > > On Sat, 18 Aug 2018, 12:20 Jason Gustafson, < > > > >>> ja...@confluent.io> > > > >>> > >> > wrote: > > > >>> > >> > > > > > > >>> > >> > > > > Hey Ismael, > > > >>> > >> > > > > > > > >>> > >> > > > > Your summary looks good to me. I think it might also > be > > a > > > >>> good > > > >>> > >> idea > > > >>> > >> > to > > > >>> > >> > > > add > > > >>> > >> > > > > a new UNSUPPORTED_COMPRESSION_TYPE error code to go > > along > > > >>> with > > > >>> > the > > > >>> > >> > > > version > > > >>> > >> > > > > bumps. We won't be able to use it for old api versions > > > >>> since the > > > >>> > >> > > clients > > > >>> > >> > > > > will not understand it, but we can use it going > forward > > so > > > >>> that > > > >>> > >> we're > > > >>> > >> > > not > > > >>> > >> > > > > stuck in a similar situation with a new message format > > and > > > >>> a new > > > >>> > >> > codec > > > >>> > >> > > to > > > >>> > >> > > > > support. Another option is to use > > UNSUPPORTED_FOR_MESSAGE_ > > > >>> > FORMAT, > > > >>> > >> but > > > >>> > >> > > it > > > >>> > >> > > > is > > > >>> > >> > > > > not as explicit. > > > >>> > >> > > > > > > > >>> > >> > > > > -Jason > > > >>> > >> > > > > > > > >>> > >> > > > > On Fri, Aug 17, 2018 at 5:19 PM, Ismael Juma < > > > >>> ism...@juma.me.uk > > > >>> > > > > > >>> > >> > > wrote: > > > >>> > >> > > > > > > > >>> > >> > > > > > Hi Dongjin and Jason, > > > >>> > >> > > > > > > > > >>> > >> > > > > > I would agree. My summary: > > > >>> > >> > > > > > > > > >>> > >> > > > > > 1. Support zstd with message format 2 only. > > > >>> > >> > > > > > 2. Bump produce and fetch request versions. > > > >>> > >> > > > > > 3. Provide broker errors whenever possible based on > > the > > > >>> > request > > > >>> > >> > > version > > > >>> > >> > > > > and > > > >>> > >> > > > > > rely on clients for the cases where the broker can't > > > >>> validate > > > >>> > >> > > > efficiently > > > >>> > >> > > > > > (example message format 2 consumer that supports the > > > >>> latest > > > >>> > >> fetch > > > >>> > >> > > > version > > > >>> > >> > > > > > but doesn't support zstd). > > > >>> > >> > > > > > > > > >>> > >> > > > > > If there's general agreement on this, I suggest we > > > update > > > >>> the > > > >>> > >> KIP > > > >>> > >> > to > > > >>> > >> > > > > state > > > >>> > >> > > > > > the proposal and to move the rejected options to its > > own > > > >>> > >> section. > > > >>> > >> > And > > > >>> > >> > > > > then > > > >>> > >> > > > > > start a vote! > > > >>> > >> > > > > > > > > >>> > >> > > > > > Ismael > > > >>> > >> > > > > > > > > >>> > >> > > > > > On Fri, Aug 17, 2018 at 4:00 PM Jason Gustafson < > > > >>> > >> > ja...@confluent.io> > > > >>> > >> > > > > > wrote: > > > >>> > >> > > > > > > > > >>> > >> > > > > > > Hi Dongjin, > > > >>> > >> > > > > > > > > > >>> > >> > > > > > > Yes, that's a good summary. For clients which > > support > > > >>> v2, > > > >>> > the > > > >>> > >> > > client > > > >>> > >> > > > > can > > > >>> > >> > > > > > > parse the message format and hopefully raise a > > useful > > > >>> error > > > >>> > >> > message > > > >>> > >> > > > > > > indicating the unsupported compression type. For > > older > > > >>> > >> clients, > > > >>> > >> > our > > > >>> > >> > > > > > options > > > >>> > >> > > > > > > are probably (1) to down-convert to the old format > > > >>> using no > > > >>> > >> > > > compression > > > >>> > >> > > > > > > type, or (2) to return an error code. I'm leaning > > > >>> toward the > > > >>> > >> > latter > > > >>> > >> > > > as > > > >>> > >> > > > > > the > > > >>> > >> > > > > > > simpler solution, but the challenge is finding a > > good > > > >>> error > > > >>> > >> code. > > > >>> > >> > > Two > > > >>> > >> > > > > > > possibilities might be INVALID_REQUEST or > > > >>> CORRUPT_MESSAGE. > > > >>> > The > > > >>> > >> > > > downside > > > >>> > >> > > > > > is > > > >>> > >> > > > > > > that old clients probably won't get a helpful > > message. > > > >>> > >> However, > > > >>> > >> > at > > > >>> > >> > > > > least > > > >>> > >> > > > > > > the behavior will be consistent in the sense that > > all > > > >>> > clients > > > >>> > >> > will > > > >>> > >> > > > fail > > > >>> > >> > > > > > if > > > >>> > >> > > > > > > they do not support zstandard. > > > >>> > >> > > > > > > > > > >>> > >> > > > > > > What do you think? > > > >>> > >> > > > > > > > > > >>> > >> > > > > > > Thanks, > > > >>> > >> > > > > > > Jason > > > >>> > >> > > > > > > > > > >>> > >> > > > > > > On Fri, Aug 17, 2018 at 8:08 AM, Dongjin Lee < > > > >>> > >> dong...@apache.org > > > >>> > >> > > > > > >>> > >> > > > > wrote: > > > >>> > >> > > > > > > > > > >>> > >> > > > > > > > Thanks Jason, I reviewed the down-converting > logic > > > >>> > following > > > >>> > >> > your > > > >>> > >> > > > > > > > explanation.[^1] You mean the following > routines, > > > >>> right? > > > >>> > >> > > > > > > > > > > >>> > >> > > > > > > > - > > > >>> > >> > > > > > > > > > > https://github.com/apache/kafka/blob/trunk/core/src/ > > > >>> > >> > > > > > > > main/scala/kafka/server/KafkaApis.scala#L534 > > > >>> > >> > > > > > > > - > > > >>> > >> > > > > > > > > > > >>> https://github.com/apache/kafka/blob/trunk/clients/src/ > > > >>> > >> > > > > > > > main/java/org/apache/kafka/common/record/ > > > >>> > >> > > LazyDownConversionRecords. > > > >>> > >> > > > > > > > java#L165 > > > >>> > >> > > > > > > > - > > > >>> > >> > > > > > > > > > > >>> https://github.com/apache/kafka/blob/trunk/clients/src/ > > > >>> > >> > > > > > > > > > > >>> > >> main/java/org/apache/kafka/common/record/RecordsUtil.java#L40 > > > >>> > >> > > > > > > > > > > >>> > >> > > > > > > > It seems like your stance is like following: > > > >>> > >> > > > > > > > > > > >>> > >> > > > > > > > 1. In principle, Kafka does not change the > > > compression > > > >>> > codec > > > >>> > >> > when > > > >>> > >> > > > > > > > down-converting, since it requires inspecting > the > > > >>> fetched > > > >>> > >> data, > > > >>> > >> > > > which > > > >>> > >> > > > > > is > > > >>> > >> > > > > > > > expensive. > > > >>> > >> > > > > > > > 2. However, there are some cases the fetched > data > > is > > > >>> > >> inspected > > > >>> > >> > > > > anyway. > > > >>> > >> > > > > > In > > > >>> > >> > > > > > > > this case, we can provide compression conversion > > > from > > > >>> > >> Zstandard > > > >>> > >> > > to > > > >>> > >> > > > > > > > classical ones[^2]. > > > >>> > >> > > > > > > > > > > >>> > >> > > > > > > > And from what I understand, the cases where the > > > client > > > >>> > >> without > > > >>> > >> > > > > > ZStandard > > > >>> > >> > > > > > > > support receives ZStandard compressed records > can > > be > > > >>> > >> organized > > > >>> > >> > > into > > > >>> > >> > > > > two > > > >>> > >> > > > > > > > cases: > > > >>> > >> > > > > > > > > > > >>> > >> > > > > > > > a. The 'compression.type' configuration of given > > > >>> topic is > > > >>> > >> > > > 'producer' > > > >>> > >> > > > > > and > > > >>> > >> > > > > > > > the producer compressed the records with > > ZStandard. > > > >>> (that > > > >>> > >> is, > > > >>> > >> > > using > > > >>> > >> > > > > > > > ZStandard implicitly.) > > > >>> > >> > > > > > > > b. The 'compression.type' configuration of > given > > > >>> topic is > > > >>> > >> > > 'zstd'; > > > >>> > >> > > > > that > > > >>> > >> > > > > > > is, > > > >>> > >> > > > > > > > using ZStandard explicitly. > > > >>> > >> > > > > > > > > > > >>> > >> > > > > > > > As you stated, we don't have to handle the case > b > > > >>> > specially. > > > >>> > >> > So, > > > >>> > >> > > It > > > >>> > >> > > > > > seems > > > >>> > >> > > > > > > > like we can narrow the focus of the problem by > > > joining > > > >>> > case > > > >>> > >> 1 > > > >>> > >> > and > > > >>> > >> > > > > case > > > >>> > >> > > > > > b > > > >>> > >> > > > > > > > like the following: > > > >>> > >> > > > > > > > > > > >>> > >> > > > > > > > > Given the topic with 'producer' as its > > > >>> > 'compression.type' > > > >>> > >> > > > > > > configuration, > > > >>> > >> > > > > > > > ZStandard compressed records and old client > > without > > > >>> > >> ZStandard, > > > >>> > >> > is > > > >>> > >> > > > > there > > > >>> > >> > > > > > > any > > > >>> > >> > > > > > > > case we need to inspect the records and can > change > > > the > > > >>> > >> > > compression > > > >>> > >> > > > > > type? > > > >>> > >> > > > > > > If > > > >>> > >> > > > > > > > so, can we provide compression type converting? > > > >>> > >> > > > > > > > > > > >>> > >> > > > > > > > Do I understand correctly? > > > >>> > >> > > > > > > > > > > >>> > >> > > > > > > > Best, > > > >>> > >> > > > > > > > Dongjin > > > >>> > >> > > > > > > > > > > >>> > >> > > > > > > > [^1]: I'm sorry, I found that I was a little bit > > > >>> > >> > misunderstanding > > > >>> > >> > > > how > > > >>> > >> > > > > > API > > > >>> > >> > > > > > > > version works, after reviewing the downconvert > > logic > > > >>> & the > > > >>> > >> > > protocol > > > >>> > >> > > > > > > > documentation < > https://kafka.apache.org/protocol > > >. > > > >>> > >> > > > > > > > [^2]: None, Gzip, Snappy, Lz4. > > > >>> > >> > > > > > > > > > > >>> > >> > > > > > > > On Tue, Aug 14, 2018 at 2:16 AM Jason Gustafson > < > > > >>> > >> > > > ja...@confluent.io> > > > >>> > >> > > > > > > > wrote: > > > >>> > >> > > > > > > > > > > >>> > >> > > > > > > > > > > > > >>> > >> > > > > > > > > > But in my opinion, since the client will > fail > > > >>> with the > > > >>> > >> API > > > >>> > >> > > > > version, > > > >>> > >> > > > > > > so > > > >>> > >> > > > > > > > we > > > >>> > >> > > > > > > > > > don't need to down-convert the messages > > anyway. > > > >>> Isn't > > > >>> > >> it? > > > >>> > >> > > So, I > > > >>> > >> > > > > > think > > > >>> > >> > > > > > > > we > > > >>> > >> > > > > > > > > > don't care about this case. (I'm sorry, I am > > not > > > >>> > >> familiar > > > >>> > >> > > with > > > >>> > >> > > > > > > > > down-convert > > > >>> > >> > > > > > > > > > logic.) > > > >>> > >> > > > > > > > > > > > >>> > >> > > > > > > > > > > > >>> > >> > > > > > > > > Currently the broker down-converts > automatically > > > >>> when it > > > >>> > >> > > receives > > > >>> > >> > > > > an > > > >>> > >> > > > > > > old > > > >>> > >> > > > > > > > > version of the fetch request (a version which > is > > > >>> known > > > >>> > to > > > >>> > >> > > predate > > > >>> > >> > > > > the > > > >>> > >> > > > > > > > > message format in use). Typically when > > > >>> down-converting > > > >>> > the > > > >>> > >> > > > message > > > >>> > >> > > > > > > > format, > > > >>> > >> > > > > > > > > we use the same compression type, but there is > > not > > > >>> much > > > >>> > >> point > > > >>> > >> > > in > > > >>> > >> > > > > > doing > > > >>> > >> > > > > > > so > > > >>> > >> > > > > > > > > when we know the client doesn't support it. So > > if > > > >>> > >> zstandard > > > >>> > >> > is > > > >>> > >> > > in > > > >>> > >> > > > > > use, > > > >>> > >> > > > > > > > and > > > >>> > >> > > > > > > > > we have to down-convert anyway, then we can > > choose > > > >>> to > > > >>> > use > > > >>> > >> a > > > >>> > >> > > > > different > > > >>> > >> > > > > > > > > compression type or no compression type. > > > >>> > >> > > > > > > > > > > > >>> > >> > > > > > > > > From my perspective, there is no significant > > > >>> downside to > > > >>> > >> > > bumping > > > >>> > >> > > > > the > > > >>> > >> > > > > > > > > protocol version and it has several potential > > > >>> benefits. > > > >>> > >> > Version > > > >>> > >> > > > > bumps > > > >>> > >> > > > > > > are > > > >>> > >> > > > > > > > > cheap. The main question mark in my mind is > > about > > > >>> > >> > > > down-conversion. > > > >>> > >> > > > > > > > Figuring > > > >>> > >> > > > > > > > > out whether down-conversion is needed is hard > > > >>> generally > > > >>> > >> > without > > > >>> > >> > > > > > > > inspecting > > > >>> > >> > > > > > > > > the fetched data, which is expensive. I think > we > > > >>> agree > > > >>> > in > > > >>> > >> > > > principle > > > >>> > >> > > > > > > that > > > >>> > >> > > > > > > > we > > > >>> > >> > > > > > > > > do not want to have to pay this cost generally > > and > > > >>> > prefer > > > >>> > >> the > > > >>> > >> > > > > clients > > > >>> > >> > > > > > > to > > > >>> > >> > > > > > > > > fail when they see an unhandled compression > > type. > > > >>> The > > > >>> > >> point I > > > >>> > >> > > was > > > >>> > >> > > > > > > making > > > >>> > >> > > > > > > > is > > > >>> > >> > > > > > > > > that there are some cases where we are either > > > >>> inspecting > > > >>> > >> the > > > >>> > >> > > data > > > >>> > >> > > > > > > anyway > > > >>> > >> > > > > > > > > (because we have to down-convert the message > > > >>> format), or > > > >>> > >> we > > > >>> > >> > > have > > > >>> > >> > > > an > > > >>> > >> > > > > > > easy > > > >>> > >> > > > > > > > > way to tell whether zstandard is in use (the > > topic > > > >>> has > > > >>> > it > > > >>> > >> > > > > configured > > > >>> > >> > > > > > > > > explicitly). In the latter case, we don't have > > to > > > >>> handle > > > >>> > >> it > > > >>> > >> > > > > > specially. > > > >>> > >> > > > > > > > But > > > >>> > >> > > > > > > > > we do have to decide how we will handle > > > >>> down-conversion > > > >>> > to > > > >>> > >> > > older > > > >>> > >> > > > > > > formats. > > > >>> > >> > > > > > > > > > > > >>> > >> > > > > > > > > -Jason > > > >>> > >> > > > > > > > > > > > >>> > >> > > > > > > > > On Sun, Aug 12, 2018 at 5:15 PM, Dongjin Lee < > > > >>> > >> > > dong...@apache.org > > > >>> > >> > > > > > > > >>> > >> > > > > > > wrote: > > > >>> > >> > > > > > > > > > > > >>> > >> > > > > > > > > > Colin and Jason, > > > >>> > >> > > > > > > > > > > > > >>> > >> > > > > > > > > > Thanks for your opinions. In summarizing, > the > > > >>> Pros and > > > >>> > >> Cons > > > >>> > >> > > of > > > >>> > >> > > > > > > bumping > > > >>> > >> > > > > > > > > > fetch API version are: > > > >>> > >> > > > > > > > > > > > > >>> > >> > > > > > > > > > Cons: > > > >>> > >> > > > > > > > > > > > > >>> > >> > > > > > > > > > - The Broker can't know whether a given > > message > > > >>> batch > > > >>> > is > > > >>> > >> > > > > compressed > > > >>> > >> > > > > > > > with > > > >>> > >> > > > > > > > > > zstd or not. > > > >>> > >> > > > > > > > > > - Need some additional logic for the topic > > > >>> explicitly > > > >>> > >> > > > configured > > > >>> > >> > > > > to > > > >>> > >> > > > > > > use > > > >>> > >> > > > > > > > > > zstd. > > > >>> > >> > > > > > > > > > > > > >>> > >> > > > > > > > > > Pros: > > > >>> > >> > > > > > > > > > > > > >>> > >> > > > > > > > > > - The broker doesn't need to conduct > expensive > > > >>> > >> > > down-conversion. > > > >>> > >> > > > > > > > > > - Can message the users to update their > > client. > > > >>> > >> > > > > > > > > > > > > >>> > >> > > > > > > > > > So, opinions for the backward-compatibility > > > >>> policy by > > > >>> > >> far: > > > >>> > >> > > > > > > > > > > > > >>> > >> > > > > > > > > > - A: bump the API version - +2 (Colin, > Jason) > > > >>> > >> > > > > > > > > > - B: leave unchanged - +1 (Viktor) > > > >>> > >> > > > > > > > > > > > > >>> > >> > > > > > > > > > Here are my additional comments: > > > >>> > >> > > > > > > > > > > > > >>> > >> > > > > > > > > > @Colin > > > >>> > >> > > > > > > > > > > > > >>> > >> > > > > > > > > > I greatly appreciate your response. In the > > case > > > >>> of the > > > >>> > >> > > > dictionary > > > >>> > >> > > > > > > > > support, > > > >>> > >> > > > > > > > > > of course, this issue should be addressed > > later > > > >>> so we > > > >>> > >> don't > > > >>> > >> > > > need > > > >>> > >> > > > > it > > > >>> > >> > > > > > > in > > > >>> > >> > > > > > > > > the > > > >>> > >> > > > > > > > > > first version. You are right - it is not > late > > to > > > >>> try > > > >>> > it > > > >>> > >> > after > > > >>> > >> > > > > some > > > >>> > >> > > > > > > > > > benchmarks. What I mean is, we should keep > in > > > >>> mind on > > > >>> > >> that > > > >>> > >> > > > > > potential > > > >>> > >> > > > > > > > > > feature. > > > >>> > >> > > > > > > > > > > > > >>> > >> > > > > > > > > > @Jason > > > >>> > >> > > > > > > > > > > > > >>> > >> > > > > > > > > > You wrote, > > > >>> > >> > > > > > > > > > > > > >>> > >> > > > > > > > > > > Similarly, if we have to down-convert > anyway > > > >>> because > > > >>> > >> the > > > >>> > >> > > > client > > > >>> > >> > > > > > > does > > > >>> > >> > > > > > > > > not > > > >>> > >> > > > > > > > > > understand the message format, then we could > > > also > > > >>> use > > > >>> > a > > > >>> > >> > > > different > > > >>> > >> > > > > > > > > > compression type. > > > >>> > >> > > > > > > > > > > > > >>> > >> > > > > > > > > > But in my opinion, since the client will > fail > > > >>> with the > > > >>> > >> API > > > >>> > >> > > > > version, > > > >>> > >> > > > > > > so > > > >>> > >> > > > > > > > we > > > >>> > >> > > > > > > > > > don't need to down-convert the messages > > anyway. > > > >>> Isn't > > > >>> > >> it? > > > >>> > >> > > So, I > > > >>> > >> > > > > > think > > > >>> > >> > > > > > > > we > > > >>> > >> > > > > > > > > > don't care about this case. (I'm sorry, I am > > not > > > >>> > >> familiar > > > >>> > >> > > with > > > >>> > >> > > > > > > > > down-convert > > > >>> > >> > > > > > > > > > logic.) > > > >>> > >> > > > > > > > > > > > > >>> > >> > > > > > > > > > Please give more opinions. Thanks! > > > >>> > >> > > > > > > > > > > > > >>> > >> > > > > > > > > > - Dongjin > > > >>> > >> > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > >>> > >> > > > > > > > > > On Wed, Aug 8, 2018 at 6:41 AM Jason > > Gustafson < > > > >>> > >> > > > > ja...@confluent.io > > > >>> > >> > > > > > > > > > >>> > >> > > > > > > > > wrote: > > > >>> > >> > > > > > > > > > > > > >>> > >> > > > > > > > > > > Hey Colin, > > > >>> > >> > > > > > > > > > > > > > >>> > >> > > > > > > > > > > The problem for the fetch API is that the > > > broker > > > >>> > does > > > >>> > >> not > > > >>> > >> > > > > > generally > > > >>> > >> > > > > > > > > know > > > >>> > >> > > > > > > > > > if > > > >>> > >> > > > > > > > > > > a batch was compressed with zstd unless it > > > >>> parses > > > >>> > it. > > > >>> > >> I > > > >>> > >> > > think > > > >>> > >> > > > > the > > > >>> > >> > > > > > > > goal > > > >>> > >> > > > > > > > > > here > > > >>> > >> > > > > > > > > > > is to avoid the expensive down-conversion > > that > > > >>> is > > > >>> > >> needed > > > >>> > >> > to > > > >>> > >> > > > > > ensure > > > >>> > >> > > > > > > > > > > compatibility because it is only necessary > > if > > > >>> zstd > > > >>> > is > > > >>> > >> > > > actually > > > >>> > >> > > > > in > > > >>> > >> > > > > > > > use. > > > >>> > >> > > > > > > > > > But > > > >>> > >> > > > > > > > > > > as long as old clients can parse the > message > > > >>> format, > > > >>> > >> they > > > >>> > >> > > > > should > > > >>> > >> > > > > > > get > > > >>> > >> > > > > > > > a > > > >>> > >> > > > > > > > > > > reasonable error if they see an > unsupported > > > >>> > >> compression > > > >>> > >> > > type > > > >>> > >> > > > in > > > >>> > >> > > > > > the > > > >>> > >> > > > > > > > > > > attributes. Basically the onus is on users > > to > > > >>> ensure > > > >>> > >> that > > > >>> > >> > > > their > > > >>> > >> > > > > > > > > consumers > > > >>> > >> > > > > > > > > > > have been updated prior to using zstd. It > > > seems > > > >>> > like a > > > >>> > >> > > > > reasonable > > > >>> > >> > > > > > > > > > tradeoff > > > >>> > >> > > > > > > > > > > to me. There are a couple cases that might > > be > > > >>> worth > > > >>> > >> > > thinking > > > >>> > >> > > > > > > through: > > > >>> > >> > > > > > > > > > > > > > >>> > >> > > > > > > > > > > 1. If a topic is explicitly configured to > > use > > > >>> zstd, > > > >>> > >> then > > > >>> > >> > we > > > >>> > >> > > > > don't > > > >>> > >> > > > > > > > need > > > >>> > >> > > > > > > > > to > > > >>> > >> > > > > > > > > > > check the fetched data for the compression > > > type > > > >>> to > > > >>> > >> know > > > >>> > >> > if > > > >>> > >> > > we > > > >>> > >> > > > > > need > > > >>> > >> > > > > > > > > > > down-conversion. If we did bump the Fetch > > API > > > >>> > version, > > > >>> > >> > then > > > >>> > >> > > > we > > > >>> > >> > > > > > > could > > > >>> > >> > > > > > > > > > handle > > > >>> > >> > > > > > > > > > > this case by either down-converting using > a > > > >>> > different > > > >>> > >> > > > > compression > > > >>> > >> > > > > > > > type > > > >>> > >> > > > > > > > > or > > > >>> > >> > > > > > > > > > > returning an error. > > > >>> > >> > > > > > > > > > > 2. Similarly, if we have to down-convert > > > anyway > > > >>> > >> because > > > >>> > >> > the > > > >>> > >> > > > > > client > > > >>> > >> > > > > > > > does > > > >>> > >> > > > > > > > > > not > > > >>> > >> > > > > > > > > > > understand the message format, then we > could > > > >>> also > > > >>> > use > > > >>> > >> a > > > >>> > >> > > > > different > > > >>> > >> > > > > > > > > > > compression type. > > > >>> > >> > > > > > > > > > > > > > >>> > >> > > > > > > > > > > For the produce API, I think it's > reasonable > > > to > > > >>> bump > > > >>> > >> the > > > >>> > >> > > api > > > >>> > >> > > > > > > version. > > > >>> > >> > > > > > > > > > This > > > >>> > >> > > > > > > > > > > can be used by clients to check whether a > > > broker > > > >>> > >> supports > > > >>> > >> > > > zstd. > > > >>> > >> > > > > > For > > > >>> > >> > > > > > > > > > > example, we might support a list of > > preferred > > > >>> > >> compression > > > >>> > >> > > > types > > > >>> > >> > > > > > in > > > >>> > >> > > > > > > > the > > > >>> > >> > > > > > > > > > > producer and we could use the broker to > > detect > > > >>> which > > > >>> > >> > > version > > > >>> > >> > > > to > > > >>> > >> > > > > > > use. > > > >>> > >> > > > > > > > > > > > > > >>> > >> > > > > > > > > > > -Jason > > > >>> > >> > > > > > > > > > > > > > >>> > >> > > > > > > > > > > On Tue, Aug 7, 2018 at 1:32 PM, Colin > > McCabe < > > > >>> > >> > > > > cmcc...@apache.org > > > >>> > >> > > > > > > > > > >>> > >> > > > > > > > > wrote: > > > >>> > >> > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > Thanks for bumping this, Dongjin. ZStd > > is a > > > >>> good > > > >>> > >> > > > compression > > > >>> > >> > > > > > > codec > > > >>> > >> > > > > > > > > > and I > > > >>> > >> > > > > > > > > > > > hope we can get this support in soon! > > > >>> > >> > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > I would say we can just bump the API > > version > > > >>> to > > > >>> > >> > indicate > > > >>> > >> > > > that > > > >>> > >> > > > > > > ZStd > > > >>> > >> > > > > > > > > > > support > > > >>> > >> > > > > > > > > > > > is expected in new clients. We probably > > > need > > > >>> some > > > >>> > >> way > > > >>> > >> > of > > > >>> > >> > > > > > > > indicating > > > >>> > >> > > > > > > > > to > > > >>> > >> > > > > > > > > > > the > > > >>> > >> > > > > > > > > > > > older clients that they can't consume > the > > > >>> > >> partitions, > > > >>> > >> > as > > > >>> > >> > > > > well. > > > >>> > >> > > > > > > > > Perhaps > > > >>> > >> > > > > > > > > > > we > > > >>> > >> > > > > > > > > > > > can use the > UNSUPPORTED_FOR_MESSAGE_FORMAT > > > >>> error? > > > >>> > >> > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > The license thing seems straightforward > -- > > > >>> it's > > > >>> > >> just a > > > >>> > >> > > > matter > > > >>> > >> > > > > > of > > > >>> > >> > > > > > > > > adding > > > >>> > >> > > > > > > > > > > > the text to the right files as per ASF > > > >>> guidelines. > > > >>> > >> > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > With regard to the dictionary support, > do > > we > > > >>> > really > > > >>> > >> > need > > > >>> > >> > > > that > > > >>> > >> > > > > > in > > > >>> > >> > > > > > > > the > > > >>> > >> > > > > > > > > > > first > > > >>> > >> > > > > > > > > > > > version? Hopefully message batches are > > big > > > >>> enough > > > >>> > >> that > > > >>> > >> > > > this > > > >>> > >> > > > > > > isn't > > > >>> > >> > > > > > > > > > > needed. > > > >>> > >> > > > > > > > > > > > Some benchmarks might help here. > > > >>> > >> > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > best, > > > >>> > >> > > > > > > > > > > > Colin > > > >>> > >> > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > On Tue, Aug 7, 2018, at 08:02, Dongjin > Lee > > > >>> wrote: > > > >>> > >> > > > > > > > > > > > > As Kafka 2.0.0 was released, let's > > reboot > > > >>> this > > > >>> > >> issue, > > > >>> > >> > > > > KIP-110 > > > >>> > >> > > > > > > > > > > > > <https://cwiki.apache.org/ > > > >>> > >> > > confluence/display/KAFKA/KIP- > > > >>> > >> > > > > > > > > > > > 110%3A+Add+Codec+for+ > > ZStandard+Compression> > > > >>> > >> > > > > > > > > > > > > . > > > >>> > >> > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > For newcomers, Here is some summary of > > the > > > >>> > >> history: > > > >>> > >> > > > KIP-110 > > > >>> > >> > > > > > was > > > >>> > >> > > > > > > > > > > > originally > > > >>> > >> > > > > > > > > > > > > worked for the issue KAFKA-4514 but, > it > > > >>> lacked > > > >>> > >> > > benchmark > > > >>> > >> > > > > > > results > > > >>> > >> > > > > > > > to > > > >>> > >> > > > > > > > > > get > > > >>> > >> > > > > > > > > > > > the > > > >>> > >> > > > > > > > > > > > > agreement of the community. Later, > Ivan > > > >>> Babrou > > > >>> > and > > > >>> > >> > some > > > >>> > >> > > > > other > > > >>> > >> > > > > > > > users > > > >>> > >> > > > > > > > > > who > > > >>> > >> > > > > > > > > > > > > adopted the patch provided their > > excellent > > > >>> > >> > performance > > > >>> > >> > > > > report > > > >>> > >> > > > > > > > which > > > >>> > >> > > > > > > > > > is > > > >>> > >> > > > > > > > > > > > now > > > >>> > >> > > > > > > > > > > > > included in the KIP, but it postponed > > > again > > > >>> > >> because > > > >>> > >> > of > > > >>> > >> > > > the > > > >>> > >> > > > > > > > > community > > > >>> > >> > > > > > > > > > > was > > > >>> > >> > > > > > > > > > > > > busy for 2.0.0 release. It is why I > now > > > >>> reboot > > > >>> > >> this > > > >>> > >> > > > issue. > > > >>> > >> > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > The following is the current status of > > the > > > >>> > >> feature: > > > >>> > >> > You > > > >>> > >> > > > can > > > >>> > >> > > > > > > check > > > >>> > >> > > > > > > > > the > > > >>> > >> > > > > > > > > > > > > current draft implementation here > > > >>> > >> > > > > > > > > > > > > < > > > https://github.com/apache/kafka/pull/2267>. > > > >>> It > > > >>> > >> is > > > >>> > >> > > based > > > >>> > >> > > > > on > > > >>> > >> > > > > > > zstd > > > >>> > >> > > > > > > > > > 1.3.5 > > > >>> > >> > > > > > > > > > > > and > > > >>> > >> > > > > > > > > > > > > periodically rebased onto the latest > > > >>> trunk[^1]. > > > >>> > >> > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > The issues that should be addressed is > > > like > > > >>> > >> > following: > > > >>> > >> > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > *1. Backward Compatibility* > > > >>> > >> > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > To support old consumers, we need to > > take > > > a > > > >>> > >> strategy > > > >>> > >> > to > > > >>> > >> > > > > > handle > > > >>> > >> > > > > > > > the > > > >>> > >> > > > > > > > > > old > > > >>> > >> > > > > > > > > > > > > consumers. Current candidates are: > > > >>> > >> > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > - Bump API version > > > >>> > >> > > > > > > > > > > > > - Leave unchanged: let the old clients > > > fail. > > > >>> > >> > > > > > > > > > > > > - Improve the error messages: > > > >>> > >> > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > *2. Dictionary Support* > > > >>> > >> > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > To support zstd's dictionary feature > in > > > the > > > >>> > future > > > >>> > >> > (if > > > >>> > >> > > > > > needed), > > > >>> > >> > > > > > > > we > > > >>> > >> > > > > > > > > > need > > > >>> > >> > > > > > > > > > > > to > > > >>> > >> > > > > > > > > > > > > sketch how it should be and leave some > > > room > > > >>> for > > > >>> > >> it. > > > >>> > >> > As > > > >>> > >> > > of > > > >>> > >> > > > > > now, > > > >>> > >> > > > > > > > > there > > > >>> > >> > > > > > > > > > > has > > > >>> > >> > > > > > > > > > > > > been no discussion on this topic yet. > > > >>> > >> > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > *3. License* > > > >>> > >> > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > To use this feature, we need to add > > > license > > > >>> of > > > >>> > >> zstd > > > >>> > >> > and > > > >>> > >> > > > > > > zstd-jni > > > >>> > >> > > > > > > > to > > > >>> > >> > > > > > > > > > the > > > >>> > >> > > > > > > > > > > > > project. (Thanks to Viktor Somogyi for > > > >>> raising > > > >>> > >> this > > > >>> > >> > > > issue!) > > > >>> > >> > > > > > It > > > >>> > >> > > > > > > > > seems > > > >>> > >> > > > > > > > > > > like > > > >>> > >> > > > > > > > > > > > > what Apache Spark did would be a good > > > >>> example > > > >>> > but > > > >>> > >> > there > > > >>> > >> > > > has > > > >>> > >> > > > > > > been > > > >>> > >> > > > > > > > no > > > >>> > >> > > > > > > > > > > > > discussion yet. > > > >>> > >> > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > You can find the details of the above > > > >>> issues in > > > >>> > >> the > > > >>> > >> > KIP > > > >>> > >> > > > > > > document. > > > >>> > >> > > > > > > > > > > Please > > > >>> > >> > > > > > > > > > > > > have a look when you are free, and > give > > me > > > >>> > >> feedback. > > > >>> > >> > > All > > > >>> > >> > > > > > kinds > > > >>> > >> > > > > > > of > > > >>> > >> > > > > > > > > > > > > participating are welcome. > > > >>> > >> > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > Best, > > > >>> > >> > > > > > > > > > > > > Dongjin > > > >>> > >> > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > [^1]: At the time of writing, commit > > > >>> 6b4fb8152. > > > >>> > >> > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > On Sat, Jul 14, 2018 at 10:45 PM > Dongjin > > > >>> Lee < > > > >>> > >> > > > > > > dong...@apache.org > > > >>> > >> > > > > > > > > > > > >>> > >> > > > > > > > > > > wrote: > > > >>> > >> > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > > Sorry for the late reply. > > > >>> > >> > > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > > In short, I could not submit the > > updated > > > >>> KIP > > > >>> > by > > > >>> > >> the > > > >>> > >> > > > > feature > > > >>> > >> > > > > > > > > freeze > > > >>> > >> > > > > > > > > > > > > > deadline of 2.0.0. For this reason, > it > > > >>> will > > > >>> > not > > > >>> > >> be > > > >>> > >> > > > > included > > > >>> > >> > > > > > > in > > > >>> > >> > > > > > > > > the > > > >>> > >> > > > > > > > > > > > 2.0.0 > > > >>> > >> > > > > > > > > > > > > > release and all discussion for this > > > issue > > > >>> were > > > >>> > >> > > > postponed > > > >>> > >> > > > > > > after > > > >>> > >> > > > > > > > > the > > > >>> > >> > > > > > > > > > > > release > > > >>> > >> > > > > > > > > > > > > > of 2.0.0. > > > >>> > >> > > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > > I have been updating the PR > following > > > >>> recent > > > >>> > >> > updates. > > > >>> > >> > > > > Just > > > >>> > >> > > > > > > > now, I > > > >>> > >> > > > > > > > > > > > rebased > > > >>> > >> > > > > > > > > > > > > > it against the latest trunk and > > updated > > > >>> the > > > >>> > zstd > > > >>> > >> > > > version > > > >>> > >> > > > > > into > > > >>> > >> > > > > > > > > > 1.3.5. > > > >>> > >> > > > > > > > > > > > If you > > > >>> > >> > > > > > > > > > > > > > need some request, don't hesitate to > > > >>> notify > > > >>> > me. > > > >>> > >> > (But > > > >>> > >> > > > not > > > >>> > >> > > > > > this > > > >>> > >> > > > > > > > > > thread > > > >>> > >> > > > > > > > > > > - > > > >>> > >> > > > > > > > > > > > just > > > >>> > >> > > > > > > > > > > > > > send me the message directly.) > > > >>> > >> > > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > > Best, > > > >>> > >> > > > > > > > > > > > > > Dongjin > > > >>> > >> > > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > > On Tue, Jul 10, 2018 at 11:57 PM > Bobby > > > >>> Evans < > > > >>> > >> > > > > > > bo...@apache.org > > > >>> > >> > > > > > > > > > > > >>> > >> > > > > > > > > > > wrote: > > > >>> > >> > > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > >> I there any update on this. The > > > >>> performance > > > >>> > >> > > > > improvements > > > >>> > >> > > > > > > are > > > >>> > >> > > > > > > > > > quite > > > >>> > >> > > > > > > > > > > > > >> impressive and I really would like > to > > > >>> stop > > > >>> > >> forking > > > >>> > >> > > > kafka > > > >>> > >> > > > > > > just > > > >>> > >> > > > > > > > to > > > >>> > >> > > > > > > > > > get > > > >>> > >> > > > > > > > > > > > this > > > >>> > >> > > > > > > > > > > > > >> in. > > > >>> > >> > > > > > > > > > > > > >> > > > >>> > >> > > > > > > > > > > > > >> Thanks, > > > >>> > >> > > > > > > > > > > > > >> > > > >>> > >> > > > > > > > > > > > > >> Bobby > > > >>> > >> > > > > > > > > > > > > >> > > > >>> > >> > > > > > > > > > > > > >> On Wed, Jun 13, 2018 at 8:56 PM > > Dongjin > > > >>> Lee < > > > >>> > >> > > > > > > > dong...@apache.org > > > >>> > >> > > > > > > > > > > > > >>> > >> > > > > > > > > > > > wrote: > > > >>> > >> > > > > > > > > > > > > >> > > > >>> > >> > > > > > > > > > > > > >> > Ismael, > > > >>> > >> > > > > > > > > > > > > >> > > > > >>> > >> > > > > > > > > > > > > >> > Oh, I forgot all of you are on > > > working > > > >>> > frenzy > > > >>> > >> > for > > > >>> > >> > > > 2.0! > > > >>> > >> > > > > > No > > > >>> > >> > > > > > > > > > problem, > > > >>> > >> > > > > > > > > > > > take > > > >>> > >> > > > > > > > > > > > > >> > your time. I am also working at > > > another > > > >>> > issue > > > >>> > >> > now. > > > >>> > >> > > > > Thank > > > >>> > >> > > > > > > you > > > >>> > >> > > > > > > > > for > > > >>> > >> > > > > > > > > > > > > >> letting me > > > >>> > >> > > > > > > > > > > > > >> > know. > > > >>> > >> > > > > > > > > > > > > >> > > > > >>> > >> > > > > > > > > > > > > >> > Best, > > > >>> > >> > > > > > > > > > > > > >> > Dongjin > > > >>> > >> > > > > > > > > > > > > >> > > > > >>> > >> > > > > > > > > > > > > >> > On Wed, Jun 13, 2018, 11:44 PM > > Ismael > > > >>> Juma > > > >>> > < > > > >>> > >> > > > > > > > isma...@gmail.com > > > >>> > >> > > > > > > > > > > > > >>> > >> > > > > > > > > > > > wrote: > > > >>> > >> > > > > > > > > > > > > >> > > > > >>> > >> > > > > > > > > > > > > >> > > Sorry for the delay Dongjin. > > > >>> Everyone is > > > >>> > >> busy > > > >>> > >> > > > > > finalising > > > >>> > >> > > > > > > > > > 2.0.0. > > > >>> > >> > > > > > > > > > > > This > > > >>> > >> > > > > > > > > > > > > >> KIP > > > >>> > >> > > > > > > > > > > > > >> > > seems like a great candidate > for > > > >>> 2.1.0 > > > >>> > and > > > >>> > >> > > > hopefully > > > >>> > >> > > > > > > there > > > >>> > >> > > > > > > > > > will > > > >>> > >> > > > > > > > > > > be > > > >>> > >> > > > > > > > > > > > > >> more > > > >>> > >> > > > > > > > > > > > > >> > of > > > >>> > >> > > > > > > > > > > > > >> > > a discussion next week. :) > > > >>> > >> > > > > > > > > > > > > >> > > > > > >>> > >> > > > > > > > > > > > > >> > > Ismael > > > >>> > >> > > > > > > > > > > > > >> > > > > > >>> > >> > > > > > > > > > > > > >> > > On Wed, 13 Jun 2018, 05:17 > > Dongjin > > > >>> Lee, < > > > >>> > >> > > > > > > > dong...@apache.org > > > >>> > >> > > > > > > > > > > > > >>> > >> > > > > > > > > > > > wrote: > > > >>> > >> > > > > > > > > > > > > >> > > > > > >>> > >> > > > > > > > > > > > > >> > > > Hello. I just updated my > draft > -- *Dongjin Lee* *A hitchhiker in the mathematical world.* *github: <http://goog_969573159/>github.com/dongjinleekr <http://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr <http://kr.linkedin.com/in/dongjinleekr>slideshare: www.slideshare.net/dongjinleekr <http://www.slideshare.net/dongjinleekr>*
