Hi Josep, A contributor just raised a regression [1] that I think should be addressed in 3.8.0 prior to the release.
Summary: This change [2] causes multiple ERROR logs to appear during worker startup when operators install other, unrelated plugins that package the Jackson library. Severity: Workers will continue to operate normally and the errors will be cosmetic. Operators and automatic systems that watch logs may roll-back upgrades due to the perception of a severe problem. Impact: I found 12 out of 250 third-party plugins that package the Jackson library and trigger this error on upgrade. This will almost certainly affect several users upon upgrades, and does not require obscure setups. Risk: The contributor has opened a simple fix PR [3], and I have verified that it addresses the problem, and can be merged tomorrow. As an alternative, we can revert the performance change completely [4] but I want to avoid this. With the above, what would you like to do for this release? Merge the fix, revert, or leave as-is? Thanks, Greg [1] https://issues.apache.org/jira/browse/KAFKA-17111 [2] https://issues.apache.org/jira/browse/KAFKA-15996 [3] https://github.com/apache/kafka/pull/16565 [4] https://github.com/apache/kafka/pull/16568 On Wed, Jul 10, 2024 at 7:37 AM Mickael Maison <mickael.mai...@gmail.com> wrote: > Hi Dongjin, > > It's great to see you back! > I hope what we did with KIP-390 matches your expectations. Looking > forward to seeing the reboot of KIP-780. > > Thanks, > Mickael > > On Wed, Jul 10, 2024 at 4:21 PM Dongjin Lee <dong...@apache.org> wrote: > > > > Hi Josep, > > > > OMG, what happened while I could not be involved with the Kafka > community? > > Thanks for digging down the whole situation. > > > > @Mickael I greatly appreciate your effort in finalizing the feature. It > > seems like I only have to re-boot the KIP-780. > > > > Thanks, > > Dongjin > > > > On Tue, Jul 9, 2024 at 12:52 AM Josep Prat <josep.p...@aiven.io.invalid> > > wrote: > > > > > Hi Dongjin, > > > > > > KIP-390 is part of the 3.8 release because the JIRA associated with it: > > > https://issues.apache.org/jira/browse/KAFKA-7632 is closed as > resolved, > > > hence the KIP is declared done and ready. I did some digging, and I saw > > > that Mickael was the one doing the PR that closed the JIRA ticket: > > > https://github.com/apache/kafka/pull/15516 > > > This means that the KIP work is merged and unfortunately it is now > quite > > > late to perform a rollback for this feature. > > > > > > @Mickael Maison <mickael.mai...@gmail.com> let me know if anything I > > > mentioned is not accurate (as you were the one bringing the KIP to > > > completion). > > > > > > Best, > > > > > > On Mon, Jul 8, 2024 at 5:38 PM Dongjin Lee <dong...@apache.org> wrote: > > > > > > > Hi Josep, > > > > > > > > Thanks for managing the 3.8 release. I have a request: could you > please > > > > move the KIP-390 into the 3.9 release? > > > > > > > > Here is the background: KIP-390 was adopted first but hasn't been > > > released > > > > for a long time. After some time, I proposed KIP-780 with further > > > > improvements and also corrected an obvious design error > > > > (`compression.level` → `compression.(gzip|lz4|zstd). level`), but it > > > hasn't > > > > been adopted due to the community's lack of response, my changing > job, > > > > focusing the in-house fork, etc. And last weekend, I found that > KIP-380 > > > has > > > > been included in the 3.8 release plan. > > > > > > > > - KIP-390: > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Support+Compression+Level > > > > - KIP-780: > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-780%3A+Support+fine-grained+compression+options > > > > > > > > However, shipping those two features at once has the following > benefits: > > > > > > > > 1. Full functionality without design error. > > > > > > > > We can provide full functionality, particularly useful with tiered > > > storage > > > > feature at once. I found that several users of tiered storage use > > > > server-side recompression and want to improve the compression > efficiency. > > > > Of course, it does not include any design errors :) > > > > > > > > 2. More chance of testing. > > > > > > > > Currently, I am managing an in-house fork of Apache Kafka and Cruise > > > > Control[^1], running on thousands of clusters on k8s. With our > ongoing > > > work > > > > on the tiered storage plugin, we can test both KIPs at once. Since > we are > > > > planning to move the terabytes of logs from thousands of > microservices > > > into > > > > the object storage, some of them can be ideal testbeds. > > > > > > > > If you are okay, I will re-initiate the discussion of KIP-780 and > rework > > > > KIP-380 on the latest trunk. > > > > > > > > Thanks, > > > > Dongjin > > > > > > > > [^1]: For example: > https://github.com/linkedin/cruise-control/pull/2145 > > > > > > > > On Mon, Feb 26, 2024 at 8:38 PM Josep Prat > <josep.p...@aiven.io.invalid> > > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > I'd like to volunteer as release manager for the Apache Kafka 3.8.0 > > > > > release. > > > > > If there are no objections, I'll start building a release plan (or > > > > adapting > > > > > the one Colin made some weeks ago) in the wiki in the next days. > > > > > > > > > > Thank you. > > > > > > > > > > -- > > > > > [image: Aiven] <https://www.aiven.io> > > > > > > > > > > *Josep Prat* > > > > > Open Source Engineering Director, *Aiven* > > > > > josep.p...@aiven.io | +491715557497 > > > > > aiven.io <https://www.aiven.io> | < > > > > https://www.facebook.com/aivencloud > > > > > > > > > > > <https://www.linkedin.com/company/aiven/> < > > > > > https://twitter.com/aiven_io> > > > > > *Aiven Deutschland GmbH* > > > > > Alexanderufer 3-7, 10117 Berlin > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > > > > > > > > > > -- > > > > *Dongjin Lee* > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > > > <https://github.com/dongjinleekr>keybase: > > > https://keybase.io/dongjinleekr > > > > <https://keybase.io/dongjinleekr>linkedin: > > > kr.linkedin.com/in/dongjinleekr > > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > > > > speakerdeck.com/dongjin > > > > <https://speakerdeck.com/dongjin>* > > > > > > > > > > > > > -- > > > [image: Aiven] <https://www.aiven.io> > > > > > > *Josep Prat* > > > Open Source Engineering Director, *Aiven* > > > josep.p...@aiven.io | +491715557497 > > > aiven.io <https://www.aiven.io> | < > https://www.facebook.com/aivencloud > > > > > > > <https://www.linkedin.com/company/aiven/> < > > > https://twitter.com/aiven_io> > > > *Aiven Deutschland GmbH* > > > Alexanderufer 3-7, 10117 Berlin > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > > -- > > *Dongjin Lee* > > > > *A hitchhiker in the mathematical world.* > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > <https://github.com/dongjinleekr>keybase: > https://keybase.io/dongjinleekr > > <https://keybase.io/dongjinleekr>linkedin: > kr.linkedin.com/in/dongjinleekr > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > speakerdeck.com/dongjin > > <https://speakerdeck.com/dongjin>* >
