Hi Timo,

Thanks for the reply, I filed https://issues.apache.org/jira/browse/FLINK-24687

The file connector dependency is via flink-table-common and there are
main source references (see below).

Thomas


[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile
(default-compile) on project flink-table-common: Compilation failure:
Compilation failure:
[ERROR] 
/Users/thomas/src/flink/flink-table/flink-table-common/src/main/java/org/apache/flink/table/descriptors/FileSystemValidator.java:[22,44]
package org.apache.flink.connector.file.sink does not exist
[ERROR] 
/Users/thomas/src/flink/flink-table/flink-table-common/src/main/java/org/apache/flink/table/descriptors/FileSystemValidator.java:[23,43]
package org.apache.flink.connector.file.src does not exist
[ERROR] 
/Users/thomas/src/flink/flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/BulkReaderFormatFactory.java:[23,43]
package org.apache.flink.connector.file.src does not exist
[ERROR] 
/Users/thomas/src/flink/flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/BulkReaderFormatFactory.java:[24,50]
package org.apache.flink.connector.file.src.reader does not exist
[ERROR] 
/Users/thomas/src/flink/flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/BulkReaderFormatFactory.java:[35,39]
cannot find symbol
[ERROR]   symbol: class BulkFormat
[ERROR] 
/Users/thomas/src/flink/flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/BulkReaderFormatFactory.java:[35,59]
cannot find symbol
[ERROR]   symbol: class FileSourceSplit
[ERROR] 
/Users/thomas/src/flink/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/format/BulkDecodingFormat.java:[22,43]
package org.apache.flink.connector.file.src does not exist
[ERROR] 
/Users/thomas/src/flink/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/format/BulkDecodingFormat.java:[23,50]
package org.apache.flink.connector.file.src.reader does not exist
[ERROR] 
/Users/thomas/src/flink/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/format/BulkDecodingFormat.java:[31,63]
cannot find symbol
[ERROR]   symbol: class BulkFormat
[ERROR] 
/Users/thomas/src/flink/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/format/BulkDecodingFormat.java:[31,77]
cannot find symbol
[ERROR]   symbol: class FileSourceSplit



On Tue, Oct 26, 2021 at 5:13 AM Timo Walther <twal...@apache.org> wrote:
>
> Hi Thomas,
>
> thanks for your feedback.
>
> The error that you are experiencing is definitely a bug in 1.13.3 and
> the missing method should be reintroduced in the next patch version to
> make code compiled against older patch versions run again.
>
> Regarding the discussion points:
>
> I agree that flink-table-blink uber jar should not contain a dependency
> to flink-connector-base. Even filesystem connectors should be optional
> and put in a dedicated module that is not in /lib by default.
>
> With having the flink-table-blink uber jar in /lib we would like to
> improve the SQL experience as this API is as important as the DataStream
> API nowadays. But the depenencies should be minimal nevertheless.
>
> Regards,
> Timo
>
>
> On 23.10.21 00:59, Thomas Weise wrote:
> > Hi,
> >
> > As part of upgrading to Flink 1.13.3 from 1.13.2 we run into the
> > following problem with KafkaSource (Flink distribution is 1.13.2 and
> > the application was built with 1.13.3):
> >
> > java.lang.NoSuchMethodError: 'void
> > org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager.<init>(org.apache.flink.connector.base.source.reader.synchronization.FutureCompletingBlockingQueue,
> > java.util.function.Supplier, java.util.function.Consumer)'
> > at 
> > org.apache.flink.connector.kafka.source.reader.fetcher.KafkaSourceFetcherManager.<init>(KafkaSourceFetcherManager.java:67)
> > at 
> > org.apache.flink.connector.kafka.source.KafkaSource.createReader(KafkaSource.java:160)
> > at 
> > org.apache.flink.connector.kafka.source.KafkaSource.createReader(KafkaSource.java:127)
> >
> > It turns out that flink-table-blink_2.12-1.13.2.jar contains
> > flink-connector-base and because that jar is under lib the 1.13.2
> > connector base gets picked up instead of the one bundled in the
> > application jar.
> >
> > (The constructor in
> > org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager
> > was added in 1.13.3.)
> >
> > There are a few points I would like to discuss:
> >
> > 1) Version compatibility: A *patch* version should ideally not
> > introduce such a change, it should be forward and backward compatible.
> > Hopefully this will be the case after 1.14 with stable source API.
> > 2) flink-table-blink - if it is meant to be self contained and usable
> > as a library - should not leak its shaded dependencies. It contains
> > FileSource and other deps from Flink, can those be relocated?
> > 3) Do we need flink-table-blink under lib? Can it be bundled with the
> > application instead? It would be great if the dependencies under lib
> > are strictly Flink core.
> >
> > Thanks,
> > Thomas
> >
>

Reply via email to