Hey Josep, I'm glad you agree that a KIP is not needed here, and I agree with you that how to publish these artifacts should be discussed with the Kafka team. In fact, this is what I created this thread for đ This is my first time contributing to Kafka, so I'm going to have to ask what a DISCUSS thread is. Is it just a mailing list thread with a subject that starts with [DISCUSS], or is there more behind it?
Best regards, Matthias Am Fr., 9. Feb. 2024 um 18:31 Uhr schrieb Josep Prat <josep.p...@aiven.io.invalid>: > Hi Matthias, > It's not adding a new functionality but it's changing the way to generate > artifacts. In the end we are talking about generating a new binary. > > I could live with not having a KIP, but a DISCUSS thread I think it's > necessary. This signals the community members and maintainers that their > input is needed. > > I could help you with writing the KIP if you want. > > Best, > > Josep Prat > Open Source Engineering Director, aivenjosep.p...@aiven.io | > +491715557497 | aiven.io > Aiven Deutschland GmbH > Alexanderufer 3-7, 10117 Berlin > GeschäftsfĂźhrer: Oskari Saarenmaa & Hannu Valtonen > Amtsgericht Charlottenburg, HRB 209739 B > > On Fri, Feb 9, 2024, 18:02 Matthias Berndt <matthias.ber...@ttmzero.com> > wrote: > > > Hi Matthias, Hi Josep, > > > > I'm afraid I can't do the KIP thing as the signup process for Apache > > Confluence requires sending me a password reset link via E-Mail and said > > E-Mail doesn't seem to reach me for some reason. I've contacted the > Apache > > infrastructure team but haven't yet heard back from them. > > That said, I'd like to push back on the notion that a KIP is really > > necessary for this change. It's certainly not a âmajor new featureâ as it > > adds zero extra functionality, and it doesn't affect binary compatibility > > either as all the currently supported Scala versions are still supported. > > This looks like a routine upgrade to me. Please, let's try to keep the > > administrative overhead to the required minimum, shall we? > > Thanks btw for merging Github PR #15239 (removal of > scala-collection-compat > > dependency from the 2.13 artifact). That will already improve life for > > Scala 3 users. > > > > All the best, > > Matthias > > > > > > Am Do., 8. Feb. 2024 um 18:02 Uhr schrieb Matthias J. Sax < > > mj...@apache.org > > >: > > > > > Josep, > > > > > > thanks for helping with this. I was also thinking if we might need a > KIP > > > for this change. Since you had the same though, I would say, yes, let's > > > do a KIP. > > > > > > @Matthias: can you prepare a KIP? You can read up on the details on the > > > wiki page: > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > > > If you have any questions about the process, please let us know. > > > > > > Thanks for pushing this forward! > > > > > > > > > -Matthias > > > > > > On 2/8/24 8:08 AM, Matthias Berndt wrote: > > > > Hey Josep et al, > > > > > > > > I've created a ticket regarding this. > > > > https://issues.apache.org/jira/browse/KAFKA-16237 > > > > > > > > All the best, > > > > Matthias > > > > > > > > Am Do., 8. Feb. 2024 um 11:42 Uhr schrieb Josep Prat > > > > <josep.p...@aiven.io.invalid>: > > > >> > > > >> Go ahead and ask for a JIRA and Wiki account (Confluence). Let us > know > > > when > > > >> your accounts are created and we'll properly set them up so you can > > > create > > > >> and assign tickets to you. > > > >> > > > >> Best, > > > >> > > > >> On Thu, Feb 8, 2024 at 11:32âŻAM Matthias Berndt < > > > matthias.ber...@ttmzero.com> > > > >> wrote: > > > >> > > > >>> Thanks Josep, I've applied for a JIRA account and addressed your > > > >>> review comments. > > > >>> > > > >>> Am Do., 8. Feb. 2024 um 09:19 Uhr schrieb Josep Prat > > > >>> <josep.p...@aiven.io.invalid>: > > > >>>> > > > >>>> Hi Matthias, > > > >>>> > > > >>>> I think for this particular case it would be worth creating a JIRA > > > ticket > > > >>>> for this as it's a new "feature". > > > >>>> Regarding the change itself, I think we need to clarify how the > > > release > > > >>>> process would work. Right now, the script `gradlewAll` is used > > (which > > > >>>> basically runs the build with Scala version 2.12 and 2.13). If I > > > >>> understand > > > >>>> your changes correctly, we would need to run the build 3 times: > > > >>>> - 1 with property scalaVersion 2.12 > > > >>>> - 1 with scalaVersion 2.13 and streamsScalaVersion 2.13 > > > >>>> - 1 with scalaVersion 2.13 and streamsScalaVersion 3.1 > > > >>>> > > > >>>> I think we should document this and discuss when to have this > > feature. > > > >>> If I > > > >>>> remember correctly from when I tried to update Kafka to Scala 3, > the > > > idea > > > >>>> was to push this to a Kafka 4.0 version because we didn't want to > > > >>> maintain > > > >>>> more than 2 Scala versions at the same time. I would encourage if > > not > > > >>>> having a KIP, at least open up a [DISCUSS] thread to clarify some > of > > > >>> these > > > >>>> points. > > > >>>> > > > >>>> I'll add some feedback on the PR itself regarding the changes. > > > >>>> > > > >>>> Best, > > > >>>> > > > >>>> On Thu, Feb 8, 2024 at 1:57âŻAM Matthias Berndt < > > > >>> matthias.ber...@ttmzero.com> > > > >>>> wrote: > > > >>>> > > > >>>>> Hi Matthias J., Hi Lucas, Hi Josep, > > > >>>>> > > > >>>>> Thank you for your encouraging responses regarding a Scala 3 port > > of > > > >>>>> Kafka-Streams-Scala, and apologies for the late response from my > > > side. > > > >>>>> I have now created a PR to port Kafka-Streams-Scala to Scala 3 > > (while > > > >>>>> retaining support for 2.13 and 2.12). Almost no changes to the > code > > > >>>>> were required and the tests also pass. Please take a look and let > > me > > > >>>>> know what you think :-) > > > >>>>> https://github.com/apache/kafka/pull/15338 > > > >>>>> > > > >>>>> All the best > > > >>>>> Matthias > > > >>>>> > > > >>>>> Am Do., 1. Feb. 2024 um 16:35 Uhr schrieb Josep Prat > > > >>>>> <josep.p...@aiven.io.invalid>: > > > >>>>>> > > > >>>>>> Hi, > > > >>>>>> > > > >>>>>> For reference, prior work on this: > > > >>>>>> https://github.com/apache/kafka/pull/11350 > > > >>>>>> https://github.com/apache/kafka/pull/11432 > > > >>>>>> > > > >>>>>> Best, > > > >>>>>> > > > >>>>>> On Thu, Feb 1, 2024, 15:55 Lucas Brutschy < > lbruts...@confluent.io > > > >>>>> .invalid> > > > >>>>>> wrote: > > > >>>>>> > > > >>>>>>> Hi Matthiases, > > > >>>>>>> > > > >>>>>>> I know Scala 2 fairly well, so I'd be happy to review changes > > that > > > >>> add > > > >>>>>>> Scala 3 support. However, as Matthias S. said, it has to be > > driven > > > >>> by > > > >>>>>>> people who use Scala day-to-day, since I believe most Kafka > > Streams > > > >>>>>>> committers are working with Java. > > > >>>>>>> > > > >>>>>>> Rewriting the tests to not use EmbeddedKafkaCluster seems like > a > > > >>> large > > > >>>>>>> undertaking, so option 1 is the first thing we should explore. > > > >>>>>>> > > > >>>>>>> I don't have any experience with Scala 3 migration topics, but > on > > > >>> the > > > >>>>>>> Scala website it says > > > >>>>>>>> The first piece of good news is that the Scala 3 compiler is > > > >>> able to > > > >>>>>>> read the Scala 2.13 Pickle format and thus it can type check > code > > > >>> that > > > >>>>>>> depends on modules or libraries compiled with Scala 2.13. > > > >>>>>>>> One notable example is the Scala 2.13 library. We have indeed > > > >>> decided > > > >>>>>>> that the Scala 2.13 library is the official standard library > for > > > >>> Scala > > > >>>>> 3. > > > >>>>>>> So wouldn't that mean that we are safe in terms of standard > > library > > > >>>>>>> upgrades if we use core_2.13 in the tests? > > > >>>>>>> > > > >>>>>>> Cheers, > > > >>>>>>> Lucas > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> On Wed, Jan 31, 2024 at 9:20âŻPM Matthias J. Sax < > > mj...@apache.org> > > > >>>>> wrote: > > > >>>>>>>> > > > >>>>>>>> Thanks for raising this. The `kafka-streams-scala` module > seems > > > >>> to > > > >>>>> be an > > > >>>>>>>> important feature for Kafka Streams and I am generally in > favor > > > >>> of > > > >>>>> your > > > >>>>>>>> proposal to add Scala 3 support. However, I am personally no > > > >>> Scala > > > >>>>>>>> person and it sounds like quite some overhead. > > > >>>>>>>> > > > >>>>>>>> If you are willing to drive and own this initiative happy to > > > >>> support > > > >>>>> you > > > >>>>>>>> to the extend I can. > > > >>>>>>>> > > > >>>>>>>> About the concrete proposal: my understanding is that :core > will > > > >>> move > > > >>>>>>>> off Scala long-term (not 100% sure what the timeline is, but > new > > > >>>>> modules > > > >>>>>>>> are written in Java only). Thus, down the road the > compatibility > > > >>>>> issue > > > >>>>>>>> would go away naturally, but it's unclear when. > > > >>>>>>>> > > > >>>>>>>> Thus, if we can test kafak-stream-scala_3 with core_2.13 it > > > >>> seems we > > > >>>>>>>> could add support for Scala 3 now, taking a risk that it might > > > >>> break > > > >>>>> in > > > >>>>>>>> the future assume that the migration off Scala from core is > not > > > >>> fast > > > >>>>>>> enough. > > > >>>>>>>> > > > >>>>>>>> For proposal (2), I don't think that it would be easily > possible > > > >>> for > > > >>>>>>>> unit/integration tests. We could fall back to system tests > > > >>> though, > > > >>>>> but > > > >>>>>>>> they would be much more heavy weight of course. > > > >>>>>>>> > > > >>>>>>>> Might be good to hear from others. We might actually also want > > > >>> to do > > > >>>>> a > > > >>>>>>>> KIP for this? > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> -Matthias > > > >>>>>>>> > > > >>>>>>>> On 1/20/24 10:34 AM, Matthias Berndt wrote: > > > >>>>>>>>> Hey there, > > > >>>>>>>>> > > > >>>>>>>>> I'd like to discuss a Scala 3 port of the kafka-streams-scala > > > >>>>> library. > > > >>>>>>>>> Currently, the build system is set up such that > > > >>> kafka-streams-scala > > > >>>>>>>>> and core (i. e. kafka itself) are compiled with the same > Scala > > > >>>>>>>>> compiler versions. This is not an optimal situation because > it > > > >>>>> means > > > >>>>>>>>> that a Scala 3 release of kafka-streams-scala cannot happen > > > >>>>>>>>> independently of kafka itself. I think this should be changed > > > >>>>>>>>> > > > >>>>>>>>> The production codebase of scala-streams-kafka actually > > > >>> compiles > > > >>>>> just > > > >>>>>>>>> fine on Scala 3.3.1 with two lines of trivial syntax changes. > > > >>> The > > > >>>>>>>>> problem is with the tests. These use the > `EmbeddedKafkaCluster` > > > >>>>> class, > > > >>>>>>>>> which means that kafka is pulled into the classpath, > > > >>> potentially > > > >>>>>>>>> leading to binary compatibility issues. > > > >>>>>>>>> I can see several approaches to fixing this: > > > >>>>>>>>> > > > >>>>>>>>> 1. Run the kafka-streams-scala tests using the compatible > > > >>> version > > > >>>>> of > > > >>>>>>>>> :core if one is available. Currently, this means that > > > >>> everything > > > >>>>> can > > > >>>>>>>>> be tested (test kafka-streams-scala_2.12 using core_2.12, > > > >>>>>>>>> kafka-streams-scala_2.13 using core_2.13 and > > > >>> kafka-streams-scala_3 > > > >>>>>>>>> using core_2.13, as these should be compatible), but when a > new > > > >>>>>>>>> scala-library version is released that is no longer > compatible > > > >>> with > > > >>>>>>>>> 2.13, we won't be able to test that. > > > >>>>>>>>> 2. Rewrite the tests to run without EmbeddedKafkaCluster, > > > >>> instead > > > >>>>>>>>> running the test cluster in a separate JVM or perhaps even a > > > >>>>>>>>> container. > > > >>>>>>>>> > > > >>>>>>>>> I'd be willing to get my hands dirty working on this, but > > > >>> before I > > > >>>>>>>>> start I'd like to get some feedback from the Kafka team > > > >>> regarding > > > >>>>> the > > > >>>>>>>>> approaches outlined above. > > > >>>>>>>>> > > > >>>>>>>>> All the best > > > >>>>>>>>> Matthias Berndt > > > >>>>>>> > > > >>>>>> KJosep Prat > > > >>>>>> Open Source Engineering Director, aivenjosep.p...@aiven.io | > > > >>>>>> +491715557497 | aiven.io > > > >>>>>> Aiven Deutschland GmbH > > > >>>>>> Alexanderufer 3-7, 10117 Berlin > > > >>>>>> GeschäftsfĂźhrer: Oskari Saarenmaa & Hannu Valtonen > > > >>>>>> Amtsgericht Charlottenburg, HRB 209739 B > > > >>>>> > > > >>>> > > > >>>> > > > >>>> -- > > > >>>> [image: Aiven] <https://www.aiven.io> > > > >>>> > > > >>>> *Josep Prat* > > > >>>> Open Source Engineering Director, *Aiven* > > > >>>> josep.p...@aiven.io | +491715557497 > > > >>>> aiven.io <https://www.aiven.io> | < > > > >>> https://www.facebook.com/aivencloud> > > > >>>> <https://www.linkedin.com/company/aiven/> < > > > >>> https://twitter.com/aiven_io> > > > >>>> *Aiven Deutschland GmbH* > > > >>>> Alexanderufer 3-7, 10117 Berlin > > > >>>> GeschäftsfĂźhrer: Oskari Saarenmaa & Hannu Valtonen > > > >>>> Amtsgericht Charlottenburg, HRB 209739 B > > > >>> > > > >> > > > >> > > > >> -- > > > >> [image: Aiven] <https://www.aiven.io> > > > >> > > > >> *Josep Prat* > > > >> Open Source Engineering Director, *Aiven* > > > >> josep.p...@aiven.io | +491715557497 > > > >> aiven.io <https://www.aiven.io> | < > > > https://www.facebook.com/aivencloud> > > > >> <https://www.linkedin.com/company/aiven/> < > > > https://twitter.com/aiven_io> > > > >> *Aiven Deutschland GmbH* > > > >> Alexanderufer 3-7, 10117 Berlin > > > >> GeschäftsfĂźhrer: Oskari Saarenmaa & Hannu Valtonen > > > >> Amtsgericht Charlottenburg, HRB 209739 B > > > > > >
