[
https://issues.apache.org/jira/browse/FLINK-40839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramin Gharib updated FLINK-40839:
---------------------------------
Description:
An INSERT with an explicit column list fails during validation when the target
table has a VARIANT or BITMAP column that is not listed.
{code:sql}
CREATE TABLE src (k INT) WITH ('connector' = 'datagen', 'number-of-rows' = '1');
CREATE TABLE snk (k INT, v VARIANT, b BITMAP) WITH ('connector' = 'blackhole');
INSERT INTO snk SELECT k, CAST(NULL AS VARIANT), CAST(NULL AS BITMAP) FROM src;
-- works
INSERT INTO snk (k) SELECT k FROM src;
-- fails
{code}
{noformat}
org.apache.flink.table.api.ValidationException: SQL validation failed.
Unsupported type when convertTypeToSpec: VARIANT
Caused by: java.lang.UnsupportedOperationException: Unsupported type when
convertTypeToSpec: VARIANT
at
org.apache.calcite.sql.type.SqlTypeUtil.convertTypeToSpec(SqlTypeUtil.java:1326)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.maybeCast(SqlValidatorImpl.java:819)
at
org.apache.flink.table.planner.calcite.FlinkCalciteSqlValidator.maybeCast(FlinkCalciteSqlValidator.java:491)
at
org.apache.flink.table.planner.calcite.PreValidateReWriter$.appendPartitionAndNullsProjects(PreValidateReWriter.scala:187)
{noformat}
BITMAP fails the same way with {{{}Unsupported type when convertTypeToSpec:
OTHER{}}}. Nested types fail too, for example {{{}ARRAY<VARIANT>{}}}, {{{}ROW<f
VARIANT>{}}}, {{MAP<STRING, VARIANT>}} and {{{}ARRAY<BITMAP>{}}}.
{{CAST(NULL AS VARIANT)}} will work once FLINK-40825 is resolved
*Cause*
{{PreValidateReWriter}} pads every omitted column with {{{}CAST(NULL AS <column
type>){}}}. The cast target is built by {{{}SqlTypeUtil.convertTypeToSpec{}}},
which has no branch for either type:
* VARIANT is not covered by {{{}SqlTypeUtil.isAtomic{}}}. Calcite fixed this
in CALCITE-7293, which ships with Calcite 1.42.0. Flink is on 1.41.0.
* BITMAP is a Flink type. {{BitmapRelDataType}} reports
{{{}SqlTypeName.OTHER{}}}, so no branch matches. Calcite cannot fix this one.
*Proposed fix*
In Flink's copy of {{{}SqlTypeUtil{}}}:
* Backport CALCITE-7293 by adding VARIANT to {{{}isAtomic{}}}. This part
becomes a no-op once FLINK-40001 upgrades to Calcite 1.42.0.
* Map {{BitmapRelDataType}} to {{SqlBitmapTypeNameSpec}} in
{{{}convertTypeToSpec{}}}.
Flink disables Calcite's implicit type coercion, so the {{isAtomic}} change has
no effect outside {{{}convertTypeToSpec{}}}.
The workaround is to list every column and write {{CAST(NULL AS VARIANT)}} or
{{CAST(NULL AS BITMAP)}} for the omitted ones.
was:
An INSERT with an explicit column list fails during validation when the target
table has a VARIANT or BITMAP column that is not listed.
{code:sql}
CREATE TABLE src (k INT) WITH ('connector' = 'datagen', 'number-of-rows' = '1');
CREATE TABLE snk (k INT, v VARIANT, b BITMAP) WITH ('connector' = 'blackhole');
INSERT INTO snk SELECT k, CAST(NULL AS VARIANT), CAST(NULL AS BITMAP) FROM src;
-- works
INSERT INTO snk (k) SELECT k FROM src;
-- fails
{code}
{noformat}
org.apache.flink.table.api.ValidationException: SQL validation failed.
Unsupported type when convertTypeToSpec: VARIANT
Caused by: java.lang.UnsupportedOperationException: Unsupported type when
convertTypeToSpec: VARIANT
at
org.apache.calcite.sql.type.SqlTypeUtil.convertTypeToSpec(SqlTypeUtil.java:1326)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.maybeCast(SqlValidatorImpl.java:819)
at
org.apache.flink.table.planner.calcite.FlinkCalciteSqlValidator.maybeCast(FlinkCalciteSqlValidator.java:491)
at
org.apache.flink.table.planner.calcite.PreValidateReWriter$.appendPartitionAndNullsProjects(PreValidateReWriter.scala:187)
{noformat}
BITMAP fails the same way with \{{Unsupported type when convertTypeToSpec:
OTHER}}. Nested types fail too, for example \{{ARRAY<VARIANT>}}, \{{ROW<f
VARIANT>}}, \{{MAP<STRING, VARIANT>}} and \{{ARRAY<BITMAP>}}.
*Cause*
{\{PreValidateReWriter}} pads every omitted column with \{{CAST(NULL AS <column
type>)}}. The cast target is built by \{{SqlTypeUtil.convertTypeToSpec}}, which
has no branch for either type:
* VARIANT is not covered by \{{SqlTypeUtil.isAtomic}}. Calcite fixed this in
CALCITE-7293, which ships with Calcite 1.42.0. Flink is on 1.41.0.
* BITMAP is a Flink type. \{{BitmapRelDataType}} reports
\{{SqlTypeName.OTHER}}, so no branch matches. Calcite cannot fix this one.
*Proposed fix*
In Flink's copy of \{{SqlTypeUtil}}:
* Backport CALCITE-7293 by adding VARIANT to \{{isAtomic}}. This part becomes a
no-op once FLINK-40001 upgrades to Calcite 1.42.0.
* Map \{{BitmapRelDataType}} to \{{SqlBitmapTypeNameSpec}} in
\{{convertTypeToSpec}}.
Flink disables Calcite's implicit type coercion, so the \{{isAtomic}} change
has no effect outside \{{convertTypeToSpec}}.
The workaround is to list every column and write \{{CAST(NULL AS VARIANT)}} or
\{{CAST(NULL AS BITMAP)}} for the omitted ones.
> INSERT with a column list fails when an omitted column is of type VARIANT or
> BITMAP
> -----------------------------------------------------------------------------------
>
> Key: FLINK-40839
> URL: https://issues.apache.org/jira/browse/FLINK-40839
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
> Reporter: Ramin Gharib
> Assignee: Ramin Gharib
> Priority: Major
>
> An INSERT with an explicit column list fails during validation when the
> target table has a VARIANT or BITMAP column that is not listed.
> {code:sql}
> CREATE TABLE src (k INT) WITH ('connector' = 'datagen', 'number-of-rows' =
> '1');
> CREATE TABLE snk (k INT, v VARIANT, b BITMAP) WITH ('connector' =
> 'blackhole');
> INSERT INTO snk SELECT k, CAST(NULL AS VARIANT), CAST(NULL AS BITMAP) FROM
> src; -- works
> INSERT INTO snk (k) SELECT k FROM src;
> -- fails
> {code}
> {noformat}
> org.apache.flink.table.api.ValidationException: SQL validation failed.
> Unsupported type when convertTypeToSpec: VARIANT
> Caused by: java.lang.UnsupportedOperationException: Unsupported type when
> convertTypeToSpec: VARIANT
> at
> org.apache.calcite.sql.type.SqlTypeUtil.convertTypeToSpec(SqlTypeUtil.java:1326)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.maybeCast(SqlValidatorImpl.java:819)
> at
> org.apache.flink.table.planner.calcite.FlinkCalciteSqlValidator.maybeCast(FlinkCalciteSqlValidator.java:491)
> at
> org.apache.flink.table.planner.calcite.PreValidateReWriter$.appendPartitionAndNullsProjects(PreValidateReWriter.scala:187)
> {noformat}
> BITMAP fails the same way with {{{}Unsupported type when convertTypeToSpec:
> OTHER{}}}. Nested types fail too, for example {{{}ARRAY<VARIANT>{}}},
> {{{}ROW<f VARIANT>{}}}, {{MAP<STRING, VARIANT>}} and {{{}ARRAY<BITMAP>{}}}.
> {{CAST(NULL AS VARIANT)}} will work once FLINK-40825 is resolved
> *Cause*
> {{PreValidateReWriter}} pads every omitted column with {{{}CAST(NULL AS
> <column type>){}}}. The cast target is built by
> {{{}SqlTypeUtil.convertTypeToSpec{}}}, which has no branch for either type:
> * VARIANT is not covered by {{{}SqlTypeUtil.isAtomic{}}}. Calcite fixed this
> in CALCITE-7293, which ships with Calcite 1.42.0. Flink is on 1.41.0.
> * BITMAP is a Flink type. {{BitmapRelDataType}} reports
> {{{}SqlTypeName.OTHER{}}}, so no branch matches. Calcite cannot fix this one.
> *Proposed fix*
> In Flink's copy of {{{}SqlTypeUtil{}}}:
> * Backport CALCITE-7293 by adding VARIANT to {{{}isAtomic{}}}. This part
> becomes a no-op once FLINK-40001 upgrades to Calcite 1.42.0.
> * Map {{BitmapRelDataType}} to {{SqlBitmapTypeNameSpec}} in
> {{{}convertTypeToSpec{}}}.
> Flink disables Calcite's implicit type coercion, so the {{isAtomic}} change
> has no effect outside {{{}convertTypeToSpec{}}}.
> The workaround is to list every column and write {{CAST(NULL AS VARIANT)}} or
> {{CAST(NULL AS BITMAP)}} for the omitted ones.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
