I there any update on this. The performance improvements are quite impressive and I really would like to stop forking kafka just to get this in.
Thanks, Bobby On Wed, Jun 13, 2018 at 8:56 PM Dongjin Lee <dong...@apache.org> wrote: > Ismael, > > Oh, I forgot all of you are on working frenzy for 2.0! No problem, take > your time. I am also working at another issue now. Thank you for letting me > know. > > Best, > Dongjin > > On Wed, Jun 13, 2018, 11:44 PM Ismael Juma <isma...@gmail.com> wrote: > > > Sorry for the delay Dongjin. Everyone is busy finalising 2.0.0. This KIP > > seems like a great candidate for 2.1.0 and hopefully there will be more > of > > a discussion next week. :) > > > > Ismael > > > > On Wed, 13 Jun 2018, 05:17 Dongjin Lee, <dong...@apache.org> wrote: > > > > > Hello. I just updated my draft implementation: > > > > > > 1. Rebased to latest trunk (commit 5145d6b) > > > 2. Apply ZStd 1.3.4 > > > > > > You can check out the implementation from here > > > <https://github.com/apache/kafka/pull/2267>. If you experience any > > problem > > > running it, don't hesitate to give me a mention. > > > > > > Best, > > > Dongjin > > > > > > On Tue, Jun 12, 2018 at 6:50 PM Dongjin Lee <dong...@apache.org> > wrote: > > > > > > > Here is the short conclusion about the license problem: *We can use > > zstd > > > > and zstd-jni without any problem, but we need to include their > license, > > > > e.g., BSD license.* > > > > > > > > Both of BSD 2 Clause License & 3 Clause License requires to include > the > > > > license used, and BSD 3 Clause License requires that the name of the > > > > contributor can't be used to endorse or promote the product. That's > it > > > > < > > > > > > http://www.mikestratton.net/2011/12/is-bsd-license-compatible-with-apache-2-0-license/ > > > > > > > > - They are not listed in the list of prohibited licenses > > > > <https://www.apache.org/legal/resolved.html#category-x> also. > > > > > > > > Here is how Spark did for it > > > > <https://issues.apache.org/jira/browse/SPARK-19112>: > > > > > > > > - They made a directory dedicated to the dependency license files > > > > <https://github.com/apache/spark/tree/master/licenses> and added > > > licenses > > > > for Zstd > > > > < > https://github.com/apache/spark/blob/master/licenses/LICENSE-zstd.txt > > > > > > & > > > > Zstd-jni > > > > < > > > > > > https://github.com/apache/spark/blob/master/licenses/LICENSE-zstd-jni.txt> > > > > . > > > > - Added a link to the original license files in LICENSE. > > > > <https://github.com/apache/spark/pull/18805/files> > > > > > > > > If needed, I can make a similar update. > > > > > > > > Thanks for pointing out this problem, Viktor! Nice catch! > > > > > > > > Best, > > > > Dongjin > > > > > > > > > > > > > > > > On Mon, Jun 11, 2018 at 11:50 PM Dongjin Lee <dong...@apache.org> > > wrote: > > > > > > > >> I greatly appreciate your comprehensive reasoning. so: +1 for b > until > > > now. > > > >> > > > >> For the license issues, I will have a check on how the over projects > > are > > > >> doing and share the results. > > > >> > > > >> Best, > > > >> Dongjin > > > >> > > > >> On Mon, Jun 11, 2018 at 10:08 PM Viktor Somogyi < > > > viktorsomo...@gmail.com> > > > >> wrote: > > > >> > > > >>> Hi Dongjin, > > > >>> > > > >>> A couple of comments: > > > >>> I would vote for option b. in the "backward compatibility" section. > > My > > > >>> reasoning for this is that users upgrading to a zstd compatible > > version > > > >>> won't start to use it automatically, so manual reconfiguration is > > > >>> required. > > > >>> Therefore an upgrade won't mess up the cluster. If not all the > > clients > > > >>> are > > > >>> upgraded but just some of them and they'd start to use zstd then it > > > would > > > >>> cause errors in the cluster. I'd like to presume though that this > is > > a > > > >>> very > > > >>> obvious failure case and nobody should be surprised if it didn't > > work. > > > >>> I wouldn't choose a. as I think we should bump the fetch and > produce > > > >>> requests if it's a change in the message format. Moreover if some > of > > > the > > > >>> producers and the brokers are upgraded but some of the consumers > are > > > not, > > > >>> then we wouldn't prevent the error when the old consumer tries to > > > consume > > > >>> the zstd compressed messages. > > > >>> I wouldn't choose c. either as I think binding the compression type > > to > > > an > > > >>> API is not so obvious from the developer's perspective. > > > >>> > > > >>> I would also prefer to use the existing binding, however we must > > > respect > > > >>> the licenses: > > > >>> "The code for these JNI bindings is licenced under 2-clause BSD > > > license. > > > >>> The native Zstd library is licensed under 3-clause BSD license and > > > GPL2" > > > >>> Based on the FAQ page > > > >>> https://www.apache.org/legal/resolved.html#category-a > > > >>> we may use 2- and 3-clause BSD licenses but the Apache license is > not > > > >>> compatible with GPL2. I'm hoping that the "3-clause BSD license and > > > GPL2" > > > >>> is really not an AND but an OR in this case, but I'm no lawyer, > just > > > >>> wanted > > > >>> to make the point that we should watch out for licenses. :) > > > >>> > > > >>> Regards, > > > >>> Viktor > > > >>> > > > >>> > > > >>> On Sun, Jun 10, 2018 at 3:02 AM Ivan Babrou <ibob...@gmail.com> > > wrote: > > > >>> > > > >>> > Hello, > > > >>> > > > > >>> > This is Ivan and I still very much support the fact that zstd > > > >>> compression > > > >>> > should be included out of the box. > > > >>> > > > > >>> > Please think about the environment, you can save quite a lot of > > > >>> hardware > > > >>> > with it. > > > >>> > > > > >>> > Thank you. > > > >>> > > > > >>> > On Sat, Jun 9, 2018 at 14:14 Dongjin Lee <dong...@apache.org> > > wrote: > > > >>> > > > > >>> > > Since there are no responses for a week, I decided to > reinitiate > > > the > > > >>> > > discussion thread. > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > >>> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-110%3A+Add+Codec+for+ZStandard+Compression > > > >>> > > > > > >>> > > This KIP is about to introduce ZStandard Compression into > Apache > > > >>> Kafka. > > > >>> > > The reason why it is posted again has a story: It was > originally > > > >>> posted > > > >>> > to > > > >>> > > the dev mailing list more than one year ago but since it has no > > > >>> > performance > > > >>> > > report included, it was postponed later. But Some people > > (including > > > >>> Ivan) > > > >>> > > reported excellent performance report with the draft PR, this > > work > > > >>> is now > > > >>> > > reactivated. > > > >>> > > > > > >>> > > The updated KIP document includes some expected problems and > > their > > > >>> > > candidate alternatives. Please have a look when you are free, > and > > > >>> give > > > >>> > me a > > > >>> > > feedback. All kinds of participating are welcome. > > > >>> > > > > > >>> > > Best, > > > >>> > > Dongjin > > > >>> > > > > > >>> > > -- > > > >>> > > *Dongjin Lee* > > > >>> > > > > > >>> > > *A hitchhiker in the mathematical world.* > > > >>> > > > > > >>> > > *github: <http://goog_969573159/>github.com/dongjinleekr > > > >>> > > <http://github.com/dongjinleekr>linkedin: > > > >>> > kr.linkedin.com/in/dongjinleekr > > > >>> > > <http://kr.linkedin.com/in/dongjinleekr>slideshare: > > > >>> > www.slideshare.net/dongjinleekr > > > >>> > > <http://www.slideshare.net/dongjinleekr>* > > > >>> > > > > > >>> > > > > >>> > > > >> -- > > > >> *Dongjin Lee* > > > >> > > > >> *A hitchhiker in the mathematical world.* > > > >> > > > >> *github: <http://goog_969573159/>github.com/dongjinleekr > > > >> <http://github.com/dongjinleekr>linkedin: > > > kr.linkedin.com/in/dongjinleekr > > > >> <http://kr.linkedin.com/in/dongjinleekr>slideshare: > > > www.slideshare.net/dongjinleekr > > > >> <http://www.slideshare.net/dongjinleekr>* > > > >> > > > > > > > > > > > > -- > > > > *Dongjin Lee* > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > > > <http://github.com/dongjinleekr>linkedin: > > > kr.linkedin.com/in/dongjinleekr > > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare: > > > www.slideshare.net/dongjinleekr > > > > <http://www.slideshare.net/dongjinleekr>* > > > > > > > > > > > > > -- > > > *Dongjin Lee* > > > > > > *A hitchhiker in the mathematical world.* > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > > <http://github.com/dongjinleekr>linkedin: > > kr.linkedin.com/in/dongjinleekr > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare: > > > www.slideshare.net/dongjinleekr > > > <http://www.slideshare.net/dongjinleekr>* > > > > > >
