[
https://issues.apache.org/jira/browse/FLINK-34076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17806720#comment-17806720
]
Khanh Vu edited comment on FLINK-34076 at 1/15/24 10:48 AM:
------------------------------------------------------------
I'm aware of the IntelliJ feature, it wont work without the fix I mentioned
even I enabled it in IntelliJ.
I'm aware of the FLINK-30400 too, and I raised this one as we're creating the
issue for kinesis connector by the change. I was not aware when the decision
was made, but IMO it's the issue with the base, and we should not solve the
issue by transferring it to concrete connectors. For things which likely breaks
compatibility, it should be abstracted away from the base, here the coupling is
still there, and we cut the connection.
bq. The flink-connector-base is already included in flink-dist
You're not declaring `flink-dist` in your app's pom.xml, are you? it wont be
issue with prod environment, but it's an issue with local development. We have
a workaround, but it's a bad user experience to use the connector.
Again, it's programmatically/semantically incorrect as kinesis connector
depends on connector base to execute, but it's not in the dependency tree of
kinesis connector?! How am I (as a user) suppose to know I need to declare
flink-connector-base in my app pom.xml before I execute the app?
was (Author: JIRAUSER290685):
I'm aware of the IntelliJ feature, it wont work with the fix I mentioned if I
dont enable it in IntelliJ.
I'm aware of the FLINK-30400 too, and I raised this one as we're creating the
issue for kinesis connector by the change. I was not aware when the decision
was made, but IMO it's the issue with the base, and we should not solve the
issue by transferring it to concrete connectors. For things which likely breaks
compatibility, it should be abstracted away from the base, here the coupling is
still there, and we cut the connection.
bq. The flink-connector-base is already included in flink-dist
You're not declaring `flink-dist` in your app's pom.xml, are you? it wont be
issue with prod environment, but it's an issue with local development. We have
a workaround, but it's a bad user experience to use the connector.
Again, it's programmatically/semantically incorrect as kinesis connector
depends on connector base to execute, but it's not in the dependency tree of
kinesis connector?! How am I (as a user) suppose to know I need to declare
flink-connector-base in my app pom.xml before I execute the app?
> flink-connector-base missing fails kinesis table sink to create
> ---------------------------------------------------------------
>
> Key: FLINK-34076
> URL: https://issues.apache.org/jira/browse/FLINK-34076
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
> Affects Versions: aws-connector-4.2.0
> Reporter: Khanh Vu
> Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> The following issue encounters with flink-kinesis-connector v4.2.0, Flink
> 1.17, it's working properly with kinesis connector v4.1.0 (I have not tested
> version pre v4.1.0).
> The
> [commit|https://github.com/apache/flink-connector-aws/commit/01f112bd5a69f95cd5d2a4bc7e08d1ba9a81d56a]
> which stops bundling `flink-connector-base` with `flink-connector-kinesis`
> has caused kinesis sink failing to create when using Table API as required
> classes from `flink-connector-base` are not loaded in runtime.
> E.g. with following depenency only in pom.xml
> {code:java}
> <dependency>
> <groupId>org.apache.flink</groupId>
> <artifactId>flink-connector-kinesis</artifactId>
> <version>${flink.connector.kinesis.version}</version>
> </dependency>
> {code}
> and a minimal job definition:
> {code:java}
> public static void main(String[] args) throws Exception {
> // create data stream environment
> StreamExecutionEnvironment sEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> sEnv.setRuntimeMode(RuntimeExecutionMode.STREAMING);
> StreamTableEnvironment tEnv =
> StreamTableEnvironment.create(sEnv);
> Schema a = Schema.newBuilder().column("a",
> DataTypes.STRING()).build();
> TableDescriptor descriptor =
> TableDescriptor.forConnector("kinesis")
> .schema(a)
> .format("json")
> .build();
> tEnv.createTemporaryTable("sinkTable", descriptor);
> tEnv.executeSql("CREATE TABLE sinkTable " +
> descriptor.toString()).print();
> }
> {code}
> following exception will be thrown:
> {code:java}
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.connector.base.table.AsyncDynamicTableSinkFactory
> at
> jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
> ~[?:?]
> at
> jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> ~[?:?]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:527) ~[?:?]
> ... 28 more
> {code}
> The fix is to explicitly specify `flink-connector-base` as dependency of the
> project:
> {code:java}
> <dependency>
> <groupId>org.apache.flink</groupId>
> <artifactId>flink-connector-kinesis</artifactId>
> <version>${flink.connector.kinesis.version}</version>
> </dependency>
> <dependency>
> <groupId>org.apache.flink</groupId>
> <artifactId>flink-connector-base</artifactId>
> <version>${flink.version}</version>
> <scope>provided</scope>
> </dependency>
> {code}
> In general, `flink-connector-base` should be pulled in by default when
> pulling in the kinesis connector, the current separation adds unnecessary
> hassle to use the connector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
