Ramin Gharib created FLINK-40839:
------------------------------------
Summary: INSERT with a column list fails when an omitted column is
of type VARIANT or BITMAP
Key: FLINK-40839
URL: https://issues.apache.org/jira/browse/FLINK-40839
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Reporter: Ramin Gharib
Assignee: Ramin Gharib
An INSERT with an explicit column list fails during validation when the target
table has a VARIANT or BITMAP column that is not listed.
{code:sql}
CREATE TABLE src (k INT) WITH ('connector' = 'datagen', 'number-of-rows' = '1');
CREATE TABLE snk (k INT, v VARIANT, b BITMAP) WITH ('connector' = 'blackhole');
INSERT INTO snk SELECT k, CAST(NULL AS VARIANT), CAST(NULL AS BITMAP) FROM src;
-- works
INSERT INTO snk (k) SELECT k FROM src;
-- fails
{code}
{noformat}
org.apache.flink.table.api.ValidationException: SQL validation failed.
Unsupported type when convertTypeToSpec: VARIANT
Caused by: java.lang.UnsupportedOperationException: Unsupported type when
convertTypeToSpec: VARIANT
at
org.apache.calcite.sql.type.SqlTypeUtil.convertTypeToSpec(SqlTypeUtil.java:1326)
at
org.apache.calcite.sql.validate.SqlValidatorImpl.maybeCast(SqlValidatorImpl.java:819)
at
org.apache.flink.table.planner.calcite.FlinkCalciteSqlValidator.maybeCast(FlinkCalciteSqlValidator.java:491)
at
org.apache.flink.table.planner.calcite.PreValidateReWriter$.appendPartitionAndNullsProjects(PreValidateReWriter.scala:187)
{noformat}
BITMAP fails the same way with \{{Unsupported type when convertTypeToSpec:
OTHER}}. Nested types fail too, for example \{{ARRAY<VARIANT>}}, \{{ROW<f
VARIANT>}}, \{{MAP<STRING, VARIANT>}} and \{{ARRAY<BITMAP>}}.
*Cause*
{\{PreValidateReWriter}} pads every omitted column with \{{CAST(NULL AS <column
type>)}}. The cast target is built by \{{SqlTypeUtil.convertTypeToSpec}}, which
has no branch for either type:
* VARIANT is not covered by \{{SqlTypeUtil.isAtomic}}. Calcite fixed this in
CALCITE-7293, which ships with Calcite 1.42.0. Flink is on 1.41.0.
* BITMAP is a Flink type. \{{BitmapRelDataType}} reports
\{{SqlTypeName.OTHER}}, so no branch matches. Calcite cannot fix this one.
*Proposed fix*
In Flink's copy of \{{SqlTypeUtil}}:
* Backport CALCITE-7293 by adding VARIANT to \{{isAtomic}}. This part becomes a
no-op once FLINK-40001 upgrades to Calcite 1.42.0.
* Map \{{BitmapRelDataType}} to \{{SqlBitmapTypeNameSpec}} in
\{{convertTypeToSpec}}.
Flink disables Calcite's implicit type coercion, so the \{{isAtomic}} change
has no effect outside \{{convertTypeToSpec}}.
The workaround is to list every column and write \{{CAST(NULL AS VARIANT)}} or
\{{CAST(NULL AS BITMAP)}} for the omitted ones.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
