Thanks for raising this. The `kafka-streams-scala` module seems to be an
important feature for Kafka Streams and I am generally in favor of your
proposal to add Scala 3 support. However, I am personally no Scala
person and it sounds like quite some overhead.
If you are willing to drive and own this initiative happy to support you
to the extend I can.
About the concrete proposal: my understanding is that :core will move
off Scala long-term (not 100% sure what the timeline is, but new modules
are written in Java only). Thus, down the road the compatibility issue
would go away naturally, but it's unclear when.
Thus, if we can test kafak-stream-scala_3 with core_2.13 it seems we
could add support for Scala 3 now, taking a risk that it might break in
the future assume that the migration off Scala from core is not fast enough.
For proposal (2), I don't think that it would be easily possible for
unit/integration tests. We could fall back to system tests though, but
they would be much more heavy weight of course.
Might be good to hear from others. We might actually also want to do a
KIP for this?
-Matthias
On 1/20/24 10:34 AM, Matthias Berndt wrote:
Hey there,
I'd like to discuss a Scala 3 port of the kafka-streams-scala library.
Currently, the build system is set up such that kafka-streams-scala
and core (i. e. kafka itself) are compiled with the same Scala
compiler versions. This is not an optimal situation because it means
that a Scala 3 release of kafka-streams-scala cannot happen
independently of kafka itself. I think this should be changed
The production codebase of scala-streams-kafka actually compiles just
fine on Scala 3.3.1 with two lines of trivial syntax changes. The
problem is with the tests. These use the `EmbeddedKafkaCluster` class,
which means that kafka is pulled into the classpath, potentially
leading to binary compatibility issues.
I can see several approaches to fixing this:
1. Run the kafka-streams-scala tests using the compatible version of
:core if one is available. Currently, this means that everything can
be tested (test kafka-streams-scala_2.12 using core_2.12,
kafka-streams-scala_2.13 using core_2.13 and kafka-streams-scala_3
using core_2.13, as these should be compatible), but when a new
scala-library version is released that is no longer compatible with
2.13, we won't be able to test that.
2. Rewrite the tests to run without EmbeddedKafkaCluster, instead
running the test cluster in a separate JVM or perhaps even a
container.
I'd be willing to get my hands dirty working on this, but before I
start I'd like to get some feedback from the Kafka team regarding the
approaches outlined above.
All the best
Matthias Berndt
