Hi Matthias, It's not adding a new functionality but it's changing the way to generate artifacts. In the end we are talking about generating a new binary.
I could live with not having a KIP, but a DISCUSS thread I think it's necessary. This signals the community members and maintainers that their input is needed. I could help you with writing the KIP if you want. Best, Josep Prat Open Source Engineering Director, aivenjosep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B On Fri, Feb 9, 2024, 18:02 Matthias Berndt <matthias.ber...@ttmzero.com> wrote: > Hi Matthias, Hi Josep, > > I'm afraid I can't do the KIP thing as the signup process for Apache > Confluence requires sending me a password reset link via E-Mail and said > E-Mail doesn't seem to reach me for some reason. I've contacted the Apache > infrastructure team but haven't yet heard back from them. > That said, I'd like to push back on the notion that a KIP is really > necessary for this change. It's certainly not a “major new feature” as it > adds zero extra functionality, and it doesn't affect binary compatibility > either as all the currently supported Scala versions are still supported. > This looks like a routine upgrade to me. Please, let's try to keep the > administrative overhead to the required minimum, shall we? > Thanks btw for merging Github PR #15239 (removal of scala-collection-compat > dependency from the 2.13 artifact). That will already improve life for > Scala 3 users. > > All the best, > Matthias > > > Am Do., 8. Feb. 2024 um 18:02 Uhr schrieb Matthias J. Sax < > mj...@apache.org > >: > > > Josep, > > > > thanks for helping with this. I was also thinking if we might need a KIP > > for this change. Since you had the same though, I would say, yes, let's > > do a KIP. > > > > @Matthias: can you prepare a KIP? You can read up on the details on the > > wiki page: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > If you have any questions about the process, please let us know. > > > > Thanks for pushing this forward! > > > > > > -Matthias > > > > On 2/8/24 8:08 AM, Matthias Berndt wrote: > > > Hey Josep et al, > > > > > > I've created a ticket regarding this. > > > https://issues.apache.org/jira/browse/KAFKA-16237 > > > > > > All the best, > > > Matthias > > > > > > Am Do., 8. Feb. 2024 um 11:42 Uhr schrieb Josep Prat > > > <josep.p...@aiven.io.invalid>: > > >> > > >> Go ahead and ask for a JIRA and Wiki account (Confluence). Let us know > > when > > >> your accounts are created and we'll properly set them up so you can > > create > > >> and assign tickets to you. > > >> > > >> Best, > > >> > > >> On Thu, Feb 8, 2024 at 11:32 AM Matthias Berndt < > > matthias.ber...@ttmzero.com> > > >> wrote: > > >> > > >>> Thanks Josep, I've applied for a JIRA account and addressed your > > >>> review comments. > > >>> > > >>> Am Do., 8. Feb. 2024 um 09:19 Uhr schrieb Josep Prat > > >>> <josep.p...@aiven.io.invalid>: > > >>>> > > >>>> Hi Matthias, > > >>>> > > >>>> I think for this particular case it would be worth creating a JIRA > > ticket > > >>>> for this as it's a new "feature". > > >>>> Regarding the change itself, I think we need to clarify how the > > release > > >>>> process would work. Right now, the script `gradlewAll` is used > (which > > >>>> basically runs the build with Scala version 2.12 and 2.13). If I > > >>> understand > > >>>> your changes correctly, we would need to run the build 3 times: > > >>>> - 1 with property scalaVersion 2.12 > > >>>> - 1 with scalaVersion 2.13 and streamsScalaVersion 2.13 > > >>>> - 1 with scalaVersion 2.13 and streamsScalaVersion 3.1 > > >>>> > > >>>> I think we should document this and discuss when to have this > feature. > > >>> If I > > >>>> remember correctly from when I tried to update Kafka to Scala 3, the > > idea > > >>>> was to push this to a Kafka 4.0 version because we didn't want to > > >>> maintain > > >>>> more than 2 Scala versions at the same time. I would encourage if > not > > >>>> having a KIP, at least open up a [DISCUSS] thread to clarify some of > > >>> these > > >>>> points. > > >>>> > > >>>> I'll add some feedback on the PR itself regarding the changes. > > >>>> > > >>>> Best, > > >>>> > > >>>> On Thu, Feb 8, 2024 at 1:57 AM Matthias Berndt < > > >>> matthias.ber...@ttmzero.com> > > >>>> wrote: > > >>>> > > >>>>> Hi Matthias J., Hi Lucas, Hi Josep, > > >>>>> > > >>>>> Thank you for your encouraging responses regarding a Scala 3 port > of > > >>>>> Kafka-Streams-Scala, and apologies for the late response from my > > side. > > >>>>> I have now created a PR to port Kafka-Streams-Scala to Scala 3 > (while > > >>>>> retaining support for 2.13 and 2.12). Almost no changes to the code > > >>>>> were required and the tests also pass. Please take a look and let > me > > >>>>> know what you think :-) > > >>>>> https://github.com/apache/kafka/pull/15338 > > >>>>> > > >>>>> All the best > > >>>>> Matthias > > >>>>> > > >>>>> Am Do., 1. Feb. 2024 um 16:35 Uhr schrieb Josep Prat > > >>>>> <josep.p...@aiven.io.invalid>: > > >>>>>> > > >>>>>> Hi, > > >>>>>> > > >>>>>> For reference, prior work on this: > > >>>>>> https://github.com/apache/kafka/pull/11350 > > >>>>>> https://github.com/apache/kafka/pull/11432 > > >>>>>> > > >>>>>> Best, > > >>>>>> > > >>>>>> On Thu, Feb 1, 2024, 15:55 Lucas Brutschy <lbruts...@confluent.io > > >>>>> .invalid> > > >>>>>> wrote: > > >>>>>> > > >>>>>>> Hi Matthiases, > > >>>>>>> > > >>>>>>> I know Scala 2 fairly well, so I'd be happy to review changes > that > > >>> add > > >>>>>>> Scala 3 support. However, as Matthias S. said, it has to be > driven > > >>> by > > >>>>>>> people who use Scala day-to-day, since I believe most Kafka > Streams > > >>>>>>> committers are working with Java. > > >>>>>>> > > >>>>>>> Rewriting the tests to not use EmbeddedKafkaCluster seems like a > > >>> large > > >>>>>>> undertaking, so option 1 is the first thing we should explore. > > >>>>>>> > > >>>>>>> I don't have any experience with Scala 3 migration topics, but on > > >>> the > > >>>>>>> Scala website it says > > >>>>>>>> The first piece of good news is that the Scala 3 compiler is > > >>> able to > > >>>>>>> read the Scala 2.13 Pickle format and thus it can type check code > > >>> that > > >>>>>>> depends on modules or libraries compiled with Scala 2.13. > > >>>>>>>> One notable example is the Scala 2.13 library. We have indeed > > >>> decided > > >>>>>>> that the Scala 2.13 library is the official standard library for > > >>> Scala > > >>>>> 3. > > >>>>>>> So wouldn't that mean that we are safe in terms of standard > library > > >>>>>>> upgrades if we use core_2.13 in the tests? > > >>>>>>> > > >>>>>>> Cheers, > > >>>>>>> Lucas > > >>>>>>> > > >>>>>>> > > >>>>>>> On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax < > mj...@apache.org> > > >>>>> wrote: > > >>>>>>>> > > >>>>>>>> Thanks for raising this. The `kafka-streams-scala` module seems > > >>> to > > >>>>> be an > > >>>>>>>> important feature for Kafka Streams and I am generally in favor > > >>> of > > >>>>> your > > >>>>>>>> proposal to add Scala 3 support. However, I am personally no > > >>> Scala > > >>>>>>>> person and it sounds like quite some overhead. > > >>>>>>>> > > >>>>>>>> If you are willing to drive and own this initiative happy to > > >>> support > > >>>>> you > > >>>>>>>> to the extend I can. > > >>>>>>>> > > >>>>>>>> About the concrete proposal: my understanding is that :core will > > >>> move > > >>>>>>>> off Scala long-term (not 100% sure what the timeline is, but new > > >>>>> modules > > >>>>>>>> are written in Java only). Thus, down the road the compatibility > > >>>>> issue > > >>>>>>>> would go away naturally, but it's unclear when. > > >>>>>>>> > > >>>>>>>> Thus, if we can test kafak-stream-scala_3 with core_2.13 it > > >>> seems we > > >>>>>>>> could add support for Scala 3 now, taking a risk that it might > > >>> break > > >>>>> in > > >>>>>>>> the future assume that the migration off Scala from core is not > > >>> fast > > >>>>>>> enough. > > >>>>>>>> > > >>>>>>>> For proposal (2), I don't think that it would be easily possible > > >>> for > > >>>>>>>> unit/integration tests. We could fall back to system tests > > >>> though, > > >>>>> but > > >>>>>>>> they would be much more heavy weight of course. > > >>>>>>>> > > >>>>>>>> Might be good to hear from others. We might actually also want > > >>> to do > > >>>>> a > > >>>>>>>> KIP for this? > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> -Matthias > > >>>>>>>> > > >>>>>>>> On 1/20/24 10:34 AM, Matthias Berndt wrote: > > >>>>>>>>> Hey there, > > >>>>>>>>> > > >>>>>>>>> I'd like to discuss a Scala 3 port of the kafka-streams-scala > > >>>>> library. > > >>>>>>>>> Currently, the build system is set up such that > > >>> kafka-streams-scala > > >>>>>>>>> and core (i. e. kafka itself) are compiled with the same Scala > > >>>>>>>>> compiler versions. This is not an optimal situation because it > > >>>>> means > > >>>>>>>>> that a Scala 3 release of kafka-streams-scala cannot happen > > >>>>>>>>> independently of kafka itself. I think this should be changed > > >>>>>>>>> > > >>>>>>>>> The production codebase of scala-streams-kafka actually > > >>> compiles > > >>>>> just > > >>>>>>>>> fine on Scala 3.3.1 with two lines of trivial syntax changes. > > >>> The > > >>>>>>>>> problem is with the tests. These use the `EmbeddedKafkaCluster` > > >>>>> class, > > >>>>>>>>> which means that kafka is pulled into the classpath, > > >>> potentially > > >>>>>>>>> leading to binary compatibility issues. > > >>>>>>>>> I can see several approaches to fixing this: > > >>>>>>>>> > > >>>>>>>>> 1. Run the kafka-streams-scala tests using the compatible > > >>> version > > >>>>> of > > >>>>>>>>> :core if one is available. Currently, this means that > > >>> everything > > >>>>> can > > >>>>>>>>> be tested (test kafka-streams-scala_2.12 using core_2.12, > > >>>>>>>>> kafka-streams-scala_2.13 using core_2.13 and > > >>> kafka-streams-scala_3 > > >>>>>>>>> using core_2.13, as these should be compatible), but when a new > > >>>>>>>>> scala-library version is released that is no longer compatible > > >>> with > > >>>>>>>>> 2.13, we won't be able to test that. > > >>>>>>>>> 2. Rewrite the tests to run without EmbeddedKafkaCluster, > > >>> instead > > >>>>>>>>> running the test cluster in a separate JVM or perhaps even a > > >>>>>>>>> container. > > >>>>>>>>> > > >>>>>>>>> I'd be willing to get my hands dirty working on this, but > > >>> before I > > >>>>>>>>> start I'd like to get some feedback from the Kafka team > > >>> regarding > > >>>>> the > > >>>>>>>>> approaches outlined above. > > >>>>>>>>> > > >>>>>>>>> All the best > > >>>>>>>>> Matthias Berndt > > >>>>>>> > > >>>>>> KJosep Prat > > >>>>>> Open Source Engineering Director, aivenjosep.p...@aiven.io | > > >>>>>> +491715557497 | aiven.io > > >>>>>> Aiven Deutschland GmbH > > >>>>>> Alexanderufer 3-7, 10117 Berlin > > >>>>>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > >>>>>> Amtsgericht Charlottenburg, HRB 209739 B > > >>>>> > > >>>> > > >>>> > > >>>> -- > > >>>> [image: Aiven] <https://www.aiven.io> > > >>>> > > >>>> *Josep Prat* > > >>>> Open Source Engineering Director, *Aiven* > > >>>> josep.p...@aiven.io | +491715557497 > > >>>> aiven.io <https://www.aiven.io> | < > > >>> https://www.facebook.com/aivencloud> > > >>>> <https://www.linkedin.com/company/aiven/> < > > >>> https://twitter.com/aiven_io> > > >>>> *Aiven Deutschland GmbH* > > >>>> Alexanderufer 3-7, 10117 Berlin > > >>>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > >>>> Amtsgericht Charlottenburg, HRB 209739 B > > >>> > > >> > > >> > > >> -- > > >> [image: Aiven] <https://www.aiven.io> > > >> > > >> *Josep Prat* > > >> Open Source Engineering Director, *Aiven* > > >> josep.p...@aiven.io | +491715557497 > > >> aiven.io <https://www.aiven.io> | < > > https://www.facebook.com/aivencloud> > > >> <https://www.linkedin.com/company/aiven/> < > > https://twitter.com/aiven_io> > > >> *Aiven Deutschland GmbH* > > >> Alexanderufer 3-7, 10117 Berlin > > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > >> Amtsgericht Charlottenburg, HRB 209739 B > > >
