Hi, Ram. Thanks for reaching out. 1: About Hive dialect issue, may be you're using JDK11? There's a known issue in FLINK-27450[1]. The main reason that Hive dosen't fully support JDK11. More specific to your case, it has been tracked in HIVE-21584[2]. Flink has upgrade the Hive 2.x version to 2.3.9 to include this patch. But unfortunately, IIRC, this patch is still not available in Hive 3.x.
2: About the creating table issue, thanks for reporting it. I tried it and it turns out that it's a bug. I have created FLINK-32596 [3] to trace it. It only happen with Flink dialect & partitioned table & Hive Catalog. In most case, we recommend user to use Hive dialect to created hive tables, then we miss the test to cover use Flink dialect to create partitioed table in Hive Catalog. So this bug has been hiden for a while. For your case, as a work around, I think you can try to create the table in Hive itself with the following SQL: CREATE TABLE testsource( `geo_altitude` FLOAT ) PARTITIONED by ( `date` STRING) tblproperties ( 'sink.partition-commit.delay'='1 s', 'sink.partition-commit.policy.kind'='metastore,success-file'); [1] https://issues.apache.org/jira/browse/FLINK-27450 [2] https://issues.apache.org/jira/browse/HIVE-21584 [3] https://issues.apache.org/jira/browse/FLINK-32596 Best regards, Yuxia 发件人: "ramkrishna vasudevan" <ramvasu.fl...@gmail.com> 收件人: "User" <u...@flink.apache.org>, "dev" <dev@flink.apache.org> 发送时间: 星期五, 2023年 7 月 14日 下午 8:46:20 主题: Issue with flink 1.16 and hive dialect Hi All, I am not sure if this was already discussed in this forum. In our set up with 1.16.0 flink we have ensured that the setup has all the necessary things for Hive catalog to work. The flink dialect works fine functionally (with some issues will come to that later). But when i follow the steps here in [ https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/hive-compatibility/hive-dialect/queries/overview/#examples | https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/hive-compatibility/hive-dialect/queries/overview/#examples ] I am getting an exception once i set to hive dialect at org.apache.flink.table.client.SqlClient.main(SqlClient.java:161) [flink-sql-client-1.16.0-0.0-SNAPSHOT.jar:1.16.0-0.0-SNAPSHOT] Caused by: java.lang.ClassCastException: class jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and java.net.URLClassLoader are in module java.base of loader 'bootstrap') at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:413) ~[flink-sql-connector-hive-3.1.2_2.12-1.16.0-0.0-SNAPSHOT.jar:1.16.0-0.0-SNAPSHOT] at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:389) ~[flink-sql-connector-hive-3.1.2_2.12-1.16.0-0.0-SNAPSHOT.jar:1.16.0-0.0-SNAPSHOT] at org.apache.flink.table.planner.delegation.hive.HiveSessionState.<init>(HiveSessionState.java:80) ~[flink-sql-connector-hive-3.1.2_2.12-1.16.0-0.0-SNAPSHOT.jar:1.16.0-0.0-SNAPSHOT] at org.apache.flink.table.planner.delegation.hive.HiveSessionState.startSessionState(HiveSessionState.java:128) ~[flink-sql-connector-hive-3.1.2_2.12-1.16.0-0.0-SNAPSHOT.jar:1.16.0-0.0-SNAPSHOT] at org.apache.flink.table.planner.delegation.hive.HiveParser.parse(HiveParser.java:210) ~[flink-sql-connector-hive-3.1.2_2.12-1.16.0-0.0-SNAPSHOT.jar:1.16.0-0.0-SNAPSHOT] at org.apache.flink.table.client.gateway.local.LocalExecutor.parseStatement(LocalExecutor.java:172) ~[flink-sql-client-1.16.0-0.0-SNAPSHOT.jar:1.16.0-0.0-SNAPSHOT] I have ensured the dialect related steps are completed followed including [ https://issues.apache.org/jira/browse/FLINK-25128 | https://issues.apache.org/jira/browse/FLINK-25128 ] In the flink catalog - if we create a table > CREATE TABLE testsource( > > `date` STRING, > `geo_altitude` FLOAT > ) > PARTITIONED by ( `date`) > > WITH ( > > 'connector' = 'hive', > 'sink.partition-commit.delay'='1 s', > 'sink.partition-commit.policy.kind'='metastore,success-file' > ); The parition always gets created on the last set of columns and not on the columns that we specify. Is this a known bug? Regards Ram
