Hi, Need to fix my previous comment in the last reply - it should be totally fine that the POM files for flink-connector-kafka 3.0.1-1.18 point to an older version. For example, in the ongoing flink-connector-opensearch release 1.1.0-1.18, the POM files also still point to Flink 1.17.1 [1].
If the user intends to compile their job for Flink 1.18.0, then they overwrite the versions for flink-streaming-java and flink-clients accordingly in the user POM. The *-1.18 prefix on the externalized connector artifacts simply indicate that they are guaranteed to be compilable and compatible with Flink 1.18.x dependencies. As a sanity check, I’ve also re-done the tests that were validated during the 3.0.1 release process to see why any issues slipped, but it turns out to be working fine: - built a fat uber jar from quickstart with Flink 1.18.0 for flink-streaming-java and flink-clients, and flink-connector-kafka version 3.0.1-1.18 - then submitted to local Flink cluster 1.18.0. Things worked as expected and the job ran fine. Now, looking at mvn dependency:tree of the uber jar (see [3] below for the output of dependency:tree), the only Flink dependency being pulled into the user uber jar is flink-connector-base:1.17.0. FLINK-30400 [2], as Danny pointed out, does intend to address this so that flink-connector-base is not being bundled by user uber jars and should be provided by Flink distributions, but AFAIK there were no breaking changes for the APIs used by classes in flink-connector-base in 1.18.0, so things should still remain compatible (as proven by my local testing). Which leaves me wondering what was the actual issue that @guenterh.lists bumped into in the first place? Am I missing something obvious? Would like to clarify this before I kick off a new release. Thanks, Gordon [1] https://repository.apache.org/content/repositories/orgapacheflink-1666/org/apache/flink[%E2%80%A6].1.0-1.18/flink-connector-opensearch-parent-1.1.0-1.18.pom [2] https://issues.apache.org/jira/browse/FLINK-30400 [3] mvn dependency:tree output for user job jar: ``` [INFO] com.tzulitai:testing-kafka:jar:1.0-SNAPSHOT [INFO] +- org.apache.flink:flink-streaming-java:jar:1.18.0:provided [INFO] | +- org.apache.flink:flink-core:jar:1.18.0:provided [INFO] | | +- org.apache.flink:flink-annotations:jar:1.18.0:provided [INFO] | | +- org.apache.flink:flink-metrics-core:jar:1.18.0:provided [INFO] | | +- org.apache.flink:flink-shaded-asm-9:jar:9.5-17.0:provided [INFO] | | +- org.apache.flink:flink-shaded-jackson:jar:2.14.2-17.0:provided [INFO] | | +- org.apache.commons:commons-lang3:jar:3.12.0:provided [INFO] | | +- org.apache.commons:commons-text:jar:1.10.0:provided [INFO] | | +- com.esotericsoftware.kryo:kryo:jar:2.24.0:provided [INFO] | | | +- com.esotericsoftware.minlog:minlog:jar:1.2:provided [INFO] | | | \- org.objenesis:objenesis:jar:2.1:provided [INFO] | | +- commons-collections:commons-collections:jar:3.2.2:provided [INFO] | | \- org.apache.commons:commons-compress:jar:1.21:provided [INFO] | +- org.apache.flink:flink-file-sink-common:jar:1.18.0:provided [INFO] | +- org.apache.flink:flink-runtime:jar:1.18.0:provided [INFO] | | +- org.apache.flink:flink-rpc-core:jar:1.18.0:provided [INFO] | | +- org.apache.flink:flink-rpc-akka-loader:jar:1.18.0:provided [INFO] | | +- org.apache.flink:flink-queryable-state-client-java:jar:1.18.0:provided [INFO] | | +- org.apache.flink:flink-hadoop-fs:jar:1.18.0:provided [INFO] | | +- commons-io:commons-io:jar:2.11.0:provided [INFO] | | +- org.apache.flink:flink-shaded-netty:jar:4.1.91.Final-17.0:provided [INFO] | | +- org.apache.flink:flink-shaded-zookeeper-3:jar:3.7.1-17.0:provided [INFO] | | +- org.javassist:javassist:jar:3.24.0-GA:provided [INFO] | | +- org.xerial.snappy:snappy-java:jar:1.1.10.4:runtime [INFO] | | \- org.lz4:lz4-java:jar:1.8.0:runtime [INFO] | +- org.apache.flink:flink-java:jar:1.18.0:provided [INFO] | | \- com.twitter:chill-java:jar:0.7.6:provided [INFO] | +- org.apache.flink:flink-shaded-guava:jar:31.1-jre-17.0:provided [INFO] | +- org.apache.commons:commons-math3:jar:3.6.1:provided [INFO] | +- org.slf4j:slf4j-api:jar:1.7.36:runtime [INFO] | \- com.google.code.findbugs:jsr305:jar:1.3.9:provided [INFO] +- org.apache.flink:flink-clients:jar:1.18.0:provided [INFO] | +- org.apache.flink:flink-optimizer:jar:1.18.0:provided [INFO] | \- commons-cli:commons-cli:jar:1.5.0:provided [INFO] +- org.apache.flink:flink-connector-kafka:jar:3.0.1-1.18:compile [INFO] | +- org.apache.flink:flink-connector-base:jar:1.17.0:compile [INFO] | +- org.apache.kafka:kafka-clients:jar:3.2.3:compile [INFO] | | \- com.github.luben:zstd-jni:jar:1.5.2-1:runtime [INFO] | +- com.fasterxml.jackson.core:jackson-core:jar:2.15.2:compile [INFO] | +- com.fasterxml.jackson.core:jackson-databind:jar:2.15.2:compile [INFO] | | \- com.fasterxml.jackson.core:jackson-annotations:jar:2.15.2:compile [INFO] | +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.15.2:compile [INFO] | \- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.15.2:compile [INFO] +- org.apache.logging.log4j:log4j-slf4j-impl:jar:2.17.1:runtime [INFO] +- org.apache.logging.log4j:log4j-api:jar:2.17.1:runtime [INFO] \- org.apache.logging.log4j:log4j-core:jar:2.17.1:runtime ``` On Thu, Nov 23, 2023 at 11:48 AM Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote: > Hi all, > > There seems to be an issue with the connector release scripts used in the > release process that doesn't correctly overwrite the flink.version property > in POMs. > > I'll kick off a new release for 3.0.2 shortly to address this. Sorry for > overlooking this during the previous release. > > Best, > Gordon > > On Thu, Nov 23, 2023 at 7:11 AM guenterh.lists <guenterh.li...@bluewin.ch> > wrote: > >> Hi Danny >> >> thanks for taking a look into it and for the hint. >> >> Your assumption is correct - It compiles when the base connector is >> excluded. >> >> In sbt: >> "org.apache.flink" % "flink-connector-kafka" % "3.0.1-1.18" >> exclude("org.apache.flink", "flink-connector-base"), >> >> Günter >> >> >> On 23.11.23 14:24, Danny Cranmer wrote: >> > Hey all, >> > >> > I believe this is because of FLINK-30400. Looking at the pom I cannot >> see >> > any other dependencies that would cause a problem. To workaround this, >> can >> > you try to remove that dependency from your build? >> > >> > <dependency> >> > <groupId>org.apache.flink</groupId> >> > <artifactId>flink-connector-kafka</artifactId> >> > <version>3.0.1-1.18</version> >> > <exclusions> >> > <exclusion> >> > <groupId>org.apache.flink</groupId> >> > <artifactId>flink-connector-base</artifactId> >> > </exclusion> >> > </exclusions> >> > </dependency> >> > >> > >> > Alternatively you can add it in: >> > >> > <dependency> >> > <groupId>org.apache.flink</groupId> >> > <artifactId>flink-connector-base</artifactId> >> > <version>1.18.0</version> >> > </dependency> >> > >> > Sorry I am not sure how to do this in Scala SBT. >> > >> > Agree we should get this fixed and push a 3.0.2 Kafka connector. >> > >> > Thanks, >> > Danny >> > >> > [1] https://issues.apache.org/jira/browse/FLINK-30400 >> > >> > On Thu, Nov 23, 2023 at 12:39 PM Leonard Xu <xbjt...@gmail.com> wrote: >> > >> >> Hi, Gurnterh >> >> >> >> It seems a bug for me that 3.0.1-1.18 flink Kafka connector use flink >> >> 1.17 dependency which lead to your issue. >> >> >> >> I guess we need propose a new release for Kafka connector for fix this >> >> issue. >> >> >> >> CC: Gordan, Danny, Martijn >> >> >> >> Best, >> >> Leonard >> >> >> >> 2023年11月14日 下午6:53，Alexey Novakov via user <user@flink.apache.org> 写道： >> >> >> >> Hi Günterh, >> >> >> >> It looks like a problem with the Kafka connector release. >> >> >> https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.0.1-1.18 >> >> Compile dependencies are still pointing to Flink 1.17. >> >> >> >> Release person is already contacted about this or will be contacted >> soon. >> >> >> >> Best regards, >> >> Alexey >> >> >> >> On Mon, Nov 13, 2023 at 10:42 PM guenterh.lists < >> guenterh.li...@bluewin.ch> >> >> wrote: >> >> >> >>> Hello >> >>> >> >>> I'm getting a dependency error when using the latest Kafka connector >> in >> >>> a Scala project. >> >>> >> >>> Using the 1.17.1 Kafka connector compilation is ok. >> >>> >> >>> With >> >>> >> >>> "org.apache.flink" % "flink-connector-kafka" % "3.0.1-1.18" >> >>> >> >>> I get >> >>> [error] (update) sbt.librarymanagement.ResolveException: Error >> >>> downloading org.apache.flink:flink-connector-base: >> >>> [error] Not found >> >>> [error] Not found >> >>> [error] not found: >> >>> >> >>> >> /home/swissbib/.ivy2/local/org.apache.flink/flink-connector-base/ivys/ivy.xml >> >>> [error] not found: >> >>> >> >>> >> https://repo1.maven.org/maven2/org/apache/flink/flink-connector-base//flink-connector-base-.pom >> >>> >> >>> Seems Maven packaging is not correct. >> >>> >> >>> My sbt build file: >> >>> >> >>> ThisBuild / scalaVersion := "3.3.0" >> >>> val flinkVersion = "1.18.0" >> >>> val postgresVersion = "42.2.2" >> >>> >> >>> lazy val root = (project in file(".")).settings( >> >>> name := "flink-scala-proj", >> >>> libraryDependencies ++= Seq( >> >>> "org.flinkextended" %% "flink-scala-api" % "1.17.1_1.1.0", >> >>> "org.apache.flink" % "flink-clients" % flinkVersion % Provided, >> >>> "org.apache.flink" % "flink-connector-files" % flinkVersion % >> >>> Provided, >> >>> >> >>> "org.apache.flink" % "flink-connector-kafka" % "1.17.1", >> >>> //"org.apache.flink" % "flink-connector-kafka" % "3.0.1-1.18", >> >>> >> >>> //"org.apache.flink" % "flink-connector-jdbc" % "3.1.1-1.17", >> >>> //"org.postgresql" % "postgresql" % postgresVersion, >> >>> "org.apache.flink" % "flink-connector-files" % flinkVersion % >> Provided, >> >>> //"org.apache.flink" % "flink-connector-base" % flinkVersion % >> Provided >> >>> ) >> >>> ) >> >>> >> >>> >> >>> >> >>> Thanks! >> >>> >> >>> -- >> >>> Günter Hipler >> >>> https://openbiblio.social/@vog61 >> >>> https://twitter.com/vog61 >> >>> >> >>> >> -- >> Günter Hipler >> https://openbiblio.social/@vog61 >> https://twitter.com/vog61 >> >>
