[
https://issues.apache.org/jira/browse/FLINK-40839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramin Gharib updated FLINK-40839:
---------------------------------
Description:
An INSERT with an explicit column list fails during validation when the target
table has a VARIANT column that is not listed.
{code:sql}
CREATE TABLE src (k INT) WITH ('connector' = 'datagen', 'number-of-rows' = '1');
CREATE TABLE snk (k INT, v VARIANT) WITH ('connector' = 'blackhole');
INSERT INTO snk SELECT k, CAST(NULL AS VARIANT) FROM src; -- works
INSERT INTO snk (k) SELECT k FROM src; -- fails
{code}
{noformat}
org.apache.flink.table.api.ValidationException: SQL validation failed.
Unsupported type when convertTypeToSpec: VARIANT
Caused by: java.lang.UnsupportedOperationException: Unsupported type when
convertTypeToSpec: VARIANT
at
org.apache.calcite.sql.type.SqlTypeUtil.convertTypeToSpec(SqlTypeUtil.java:1326)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.maybeCast(SqlValidatorImpl.java:819)
at
org.apache.flink.table.planner.calcite.FlinkCalciteSqlValidator.maybeCast(FlinkCalciteSqlValidator.java:491)
at
org.apache.flink.table.planner.calcite.PreValidateReWriter$.appendPartitionAndNullsProjects(PreValidateReWriter.scala:187)
{noformat}
Nested types fail too, for example {{{}ARRAY<VARIANT>{}}}, {{ROW<f VARIANT>}}
and {{{}MAP<STRING, VARIANT>{}}}.
*Cause*
{{PreValidateReWriter}} pads every omitted column with {{{}CAST(NULL AS <column
type>){}}}. The cast target is built by {{{}SqlTypeUtil.convertTypeToSpec{}}}.
VARIANT is not covered by {{{}SqlTypeUtil.isAtomic{}}}, so no branch matches.
Calcite fixed this in CALCITE-7293, which ships with Calcite 1.42.0. Flink is
on 1.41.0.
*Proposed fix*
Backport CALCITE-7293 to Flink's copy of {{SqlTypeUtil}} by adding VARIANT to
{{{}isAtomic{}}}. The backport can be dropped once FLINK-40001 upgrades to
Calcite 1.42.0.
Flink disables Calcite's implicit type coercion, so the {{isAtomic}} change has
no effect outside {{{}convertTypeToSpec{}}}.
BITMAP, RAW and STRUCTURED columns fail the same way. They are Flink types that
Calcite does not know, so they are fixed separately in FLINK-40840
The workaround is to list every column and write {{CAST(NULL AS VARIANT)}} for
the omitted ones.
{{CAST(NULL AS VARIANT)}} will work once FLINK-40825 is resolved
was:
An INSERT with an explicit column list fails during validation when the target
table has a VARIANT column that is not listed.
{code:sql}
CREATE TABLE src (k INT) WITH ('connector' = 'datagen', 'number-of-rows' = '1');
CREATE TABLE snk (k INT, v VARIANT) WITH ('connector' = 'blackhole');
INSERT INTO snk SELECT k, CAST(NULL AS VARIANT) FROM src; -- works
INSERT INTO snk (k) SELECT k FROM src; -- fails
{code}
{noformat}
org.apache.flink.table.api.ValidationException: SQL validation failed.
Unsupported type when convertTypeToSpec: VARIANT
Caused by: java.lang.UnsupportedOperationException: Unsupported type when
convertTypeToSpec: VARIANT
at
org.apache.calcite.sql.type.SqlTypeUtil.convertTypeToSpec(SqlTypeUtil.java:1326)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.maybeCast(SqlValidatorImpl.java:819)
at
org.apache.flink.table.planner.calcite.FlinkCalciteSqlValidator.maybeCast(FlinkCalciteSqlValidator.java:491)
at
org.apache.flink.table.planner.calcite.PreValidateReWriter$.appendPartitionAndNullsProjects(PreValidateReWriter.scala:187)
{noformat}
Nested types fail too, for example \{{ARRAY<VARIANT>}}, \{{ROW<f VARIANT>}} and
\{{MAP<STRING, VARIANT>}}.
*Cause*
{\{PreValidateReWriter}} pads every omitted column with \{{CAST(NULL AS <column
type>)}}. The cast target is built by \{{SqlTypeUtil.convertTypeToSpec}}.
VARIANT is not covered by \{{SqlTypeUtil.isAtomic}}, so no branch matches.
Calcite fixed this in CALCITE-7293, which ships with Calcite 1.42.0. Flink is
on 1.41.0.
*Proposed fix*
Backport CALCITE-7293 to Flink's copy of \{{SqlTypeUtil}} by adding VARIANT to
\{{isAtomic}}. The backport can be dropped once FLINK-40001 upgrades to Calcite
1.42.0.
Flink disables Calcite's implicit type coercion, so the \{{isAtomic}} change
has no effect outside \{{convertTypeToSpec}}.
BITMAP, RAW and STRUCTURED columns fail the same way. They are Flink types that
Calcite does not know, so they are fixed separately in FLINK-XXXXX.
The workaround is to list every column and write \{{CAST(NULL AS VARIANT)}} for
the omitted ones.
{{CAST(NULL AS VARIANT)}} will work once FLINK-40825 is resolved
> INSERT with a column list fails when an omitted column is of type VARIANT or
> BITMAP
> -----------------------------------------------------------------------------------
>
> Key: FLINK-40839
> URL: https://issues.apache.org/jira/browse/FLINK-40839
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
> Reporter: Ramin Gharib
> Assignee: Ramin Gharib
> Priority: Major
> Labels: pull-request-available
>
> An INSERT with an explicit column list fails during validation when the
> target table has a VARIANT column that is not listed.
> {code:sql}
> CREATE TABLE src (k INT) WITH ('connector' = 'datagen', 'number-of-rows' =
> '1');
> CREATE TABLE snk (k INT, v VARIANT) WITH ('connector' = 'blackhole');
> INSERT INTO snk SELECT k, CAST(NULL AS VARIANT) FROM src; -- works
> INSERT INTO snk (k) SELECT k FROM src; -- fails
> {code}
> {noformat}
> org.apache.flink.table.api.ValidationException: SQL validation failed.
> Unsupported type when convertTypeToSpec: VARIANT
> Caused by: java.lang.UnsupportedOperationException: Unsupported type when
> convertTypeToSpec: VARIANT
> at
> org.apache.calcite.sql.type.SqlTypeUtil.convertTypeToSpec(SqlTypeUtil.java:1326)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.maybeCast(SqlValidatorImpl.java:819)
> at
> org.apache.flink.table.planner.calcite.FlinkCalciteSqlValidator.maybeCast(FlinkCalciteSqlValidator.java:491)
> at
> org.apache.flink.table.planner.calcite.PreValidateReWriter$.appendPartitionAndNullsProjects(PreValidateReWriter.scala:187)
> {noformat}
> Nested types fail too, for example {{{}ARRAY<VARIANT>{}}}, {{ROW<f VARIANT>}}
> and {{{}MAP<STRING, VARIANT>{}}}.
> *Cause*
> {{PreValidateReWriter}} pads every omitted column with {{{}CAST(NULL AS
> <column type>){}}}. The cast target is built by
> {{{}SqlTypeUtil.convertTypeToSpec{}}}. VARIANT is not covered by
> {{{}SqlTypeUtil.isAtomic{}}}, so no branch matches. Calcite fixed this in
> CALCITE-7293, which ships with Calcite 1.42.0. Flink is on 1.41.0.
> *Proposed fix*
> Backport CALCITE-7293 to Flink's copy of {{SqlTypeUtil}} by adding VARIANT to
> {{{}isAtomic{}}}. The backport can be dropped once FLINK-40001 upgrades to
> Calcite 1.42.0.
> Flink disables Calcite's implicit type coercion, so the {{isAtomic}} change
> has no effect outside {{{}convertTypeToSpec{}}}.
> BITMAP, RAW and STRUCTURED columns fail the same way. They are Flink types
> that Calcite does not know, so they are fixed separately in FLINK-40840
> The workaround is to list every column and write {{CAST(NULL AS VARIANT)}}
> for the omitted ones.
> {{CAST(NULL AS VARIANT)}} will work once FLINK-40825 is resolved
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
