[ https://issues.apache.org/jira/browse/SPARK-33913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17255395#comment-17255395 ]
L. C. Hsieh commented on SPARK-33913: ------------------------------------- Seems we cannot upgrade to Kafka 2.7.0. Because Kafka core inlines the Scala library, you can not use a different Scala patch version than what Kafka used to compile its jars: https://github.com/embeddedkafka/embedded-kafka/issues/202 Kafka 2.7.0 uses Scala 2.12.12 and currently Spark uses Scala 2.12.10, so there will be {{java.lang.NoClassDefFoundError: scala/math/Ordering$$anon$7}} errors. Due to an issue in Scala 2.12.12, Spark won't upgrade to Scala 2.12.12: https://github.com/scala/bug/issues/12096, and waits for Scala 2.12.13. So, seems for Kafka, Spark needs to wait for next Kafka version which uses Scala 2.12.13 too. > Upgrade Kafka to 2.7.0 > ---------------------- > > Key: SPARK-33913 > URL: https://issues.apache.org/jira/browse/SPARK-33913 > Project: Spark > Issue Type: Improvement > Components: Build, DStreams > Affects Versions: 3.2.0 > Reporter: dengziming > Priority: Major > > > The Apache Kafka community has released for Apache Kafka 2.7.0, some features > are useful for example the KAFKA-9893 > configurable TCP connection timeout, more details : > https://downloads.apache.org/kafka/2.7.0/RELEASE_NOTES.html -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org