Josep,
thanks for helping with this. I was also thinking if we might need a KIP
for this change. Since you had the same though, I would say, yes, let's
do a KIP.
@Matthias: can you prepare a KIP? You can read up on the details on the
wiki page:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
If you have any questions about the process, please let us know.
Thanks for pushing this forward!
-Matthias
On 2/8/24 8:08 AM, Matthias Berndt wrote:
Hey Josep et al,
I've created a ticket regarding this.
https://issues.apache.org/jira/browse/KAFKA-16237
All the best,
Matthias
Am Do., 8. Feb. 2024 um 11:42 Uhr schrieb Josep Prat
<josep.p...@aiven.io.invalid>:
Go ahead and ask for a JIRA and Wiki account (Confluence). Let us know when
your accounts are created and we'll properly set them up so you can create
and assign tickets to you.
Best,
On Thu, Feb 8, 2024 at 11:32 AM Matthias Berndt <matthias.ber...@ttmzero.com>
wrote:
Thanks Josep, I've applied for a JIRA account and addressed your
review comments.
Am Do., 8. Feb. 2024 um 09:19 Uhr schrieb Josep Prat
<josep.p...@aiven.io.invalid>:
Hi Matthias,
I think for this particular case it would be worth creating a JIRA ticket
for this as it's a new "feature".
Regarding the change itself, I think we need to clarify how the release
process would work. Right now, the script `gradlewAll` is used (which
basically runs the build with Scala version 2.12 and 2.13). If I
understand
your changes correctly, we would need to run the build 3 times:
- 1 with property scalaVersion 2.12
- 1 with scalaVersion 2.13 and streamsScalaVersion 2.13
- 1 with scalaVersion 2.13 and streamsScalaVersion 3.1
I think we should document this and discuss when to have this feature.
If I
remember correctly from when I tried to update Kafka to Scala 3, the idea
was to push this to a Kafka 4.0 version because we didn't want to
maintain
more than 2 Scala versions at the same time. I would encourage if not
having a KIP, at least open up a [DISCUSS] thread to clarify some of
these
points.
I'll add some feedback on the PR itself regarding the changes.
Best,
On Thu, Feb 8, 2024 at 1:57 AM Matthias Berndt <
matthias.ber...@ttmzero.com>
wrote:
Hi Matthias J., Hi Lucas, Hi Josep,
Thank you for your encouraging responses regarding a Scala 3 port of
Kafka-Streams-Scala, and apologies for the late response from my side.
I have now created a PR to port Kafka-Streams-Scala to Scala 3 (while
retaining support for 2.13 and 2.12). Almost no changes to the code
were required and the tests also pass. Please take a look and let me
know what you think :-)
https://github.com/apache/kafka/pull/15338
All the best
Matthias
Am Do., 1. Feb. 2024 um 16:35 Uhr schrieb Josep Prat
<josep.p...@aiven.io.invalid>:
Hi,
For reference, prior work on this:
https://github.com/apache/kafka/pull/11350
https://github.com/apache/kafka/pull/11432
Best,
On Thu, Feb 1, 2024, 15:55 Lucas Brutschy <lbruts...@confluent.io
.invalid>
wrote:
Hi Matthiases,
I know Scala 2 fairly well, so I'd be happy to review changes that
add
Scala 3 support. However, as Matthias S. said, it has to be driven
by
people who use Scala day-to-day, since I believe most Kafka Streams
committers are working with Java.
Rewriting the tests to not use EmbeddedKafkaCluster seems like a
large
undertaking, so option 1 is the first thing we should explore.
I don't have any experience with Scala 3 migration topics, but on
the
Scala website it says
The first piece of good news is that the Scala 3 compiler is
able to
read the Scala 2.13 Pickle format and thus it can type check code
that
depends on modules or libraries compiled with Scala 2.13.
One notable example is the Scala 2.13 library. We have indeed
decided
that the Scala 2.13 library is the official standard library for
Scala
3.
So wouldn't that mean that we are safe in terms of standard library
upgrades if we use core_2.13 in the tests?
Cheers,
Lucas
On Wed, Jan 31, 2024 at 9:20 PM Matthias J. Sax <mj...@apache.org>
wrote:
Thanks for raising this. The `kafka-streams-scala` module seems
to
be an
important feature for Kafka Streams and I am generally in favor
of
your
proposal to add Scala 3 support. However, I am personally no
Scala
person and it sounds like quite some overhead.
If you are willing to drive and own this initiative happy to
support
you
to the extend I can.
About the concrete proposal: my understanding is that :core will
move
off Scala long-term (not 100% sure what the timeline is, but new
modules
are written in Java only). Thus, down the road the compatibility
issue
would go away naturally, but it's unclear when.
Thus, if we can test kafak-stream-scala_3 with core_2.13 it
seems we
could add support for Scala 3 now, taking a risk that it might
break
in
the future assume that the migration off Scala from core is not
fast
enough.
For proposal (2), I don't think that it would be easily possible
for
unit/integration tests. We could fall back to system tests
though,
but
they would be much more heavy weight of course.
Might be good to hear from others. We might actually also want
to do
a
KIP for this?
-Matthias
On 1/20/24 10:34 AM, Matthias Berndt wrote:
Hey there,
I'd like to discuss a Scala 3 port of the kafka-streams-scala
library.
Currently, the build system is set up such that
kafka-streams-scala
and core (i. e. kafka itself) are compiled with the same Scala
compiler versions. This is not an optimal situation because it
means
that a Scala 3 release of kafka-streams-scala cannot happen
independently of kafka itself. I think this should be changed
The production codebase of scala-streams-kafka actually
compiles
just
fine on Scala 3.3.1 with two lines of trivial syntax changes.
The
problem is with the tests. These use the `EmbeddedKafkaCluster`
class,
which means that kafka is pulled into the classpath,
potentially
leading to binary compatibility issues.
I can see several approaches to fixing this:
1. Run the kafka-streams-scala tests using the compatible
version
of
:core if one is available. Currently, this means that
everything
can
be tested (test kafka-streams-scala_2.12 using core_2.12,
kafka-streams-scala_2.13 using core_2.13 and
kafka-streams-scala_3
using core_2.13, as these should be compatible), but when a new
scala-library version is released that is no longer compatible
with
2.13, we won't be able to test that.
2. Rewrite the tests to run without EmbeddedKafkaCluster,
instead
running the test cluster in a separate JVM or perhaps even a
container.
I'd be willing to get my hands dirty working on this, but
before I
start I'd like to get some feedback from the Kafka team
regarding
the
approaches outlined above.
All the best
Matthias Berndt
KJosep Prat
Open Source Engineering Director, aivenjosep.p...@aiven.io |
+491715557497 | aiven.io
Aiven Deutschland GmbH
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B
--
[image: Aiven] <https://www.aiven.io>
*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io | +491715557497
aiven.io <https://www.aiven.io> | <
https://www.facebook.com/aivencloud>
<https://www.linkedin.com/company/aiven/> <
https://twitter.com/aiven_io>
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B
--
[image: Aiven] <https://www.aiven.io>
*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io | +491715557497
aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud>
<https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io>
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B
