david radley created CALCITE-6579:
-------------------------------------
Summary: Unexpected nullable checks for nullable types with
non-nullable children
Key: CALCITE-6579
URL: https://issues.apache.org/jira/browse/CALCITE-6579
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.37.0
Reporter: david radley
This Jira is raised for the issue discussed in
[https://lists.apache.org/thread/x857njnd42gcvrldbcd7y85to6fb58dn]
I am using flink and have a table definition like this.
_CREATE TABLE source_1 ( `order_id` STRING NOT NULL, `order_time` STRING NOT
NULL, `buyer` ROW< `first_name` STRING, `last_name` STRING NOT NULL, `title`
STRING NOT NULL_
_) WITH ( 'connector' = 'kafka', 'topic' = 'mytopic',
'properties.bootstrap.servers' = 'localhost:9092', 'value.format' = 'avro',
'value.fields-include' = 'ALL', 'scan.startup.mode' = 'earliest-offset' );_
This errors as code in Calcite does not create the correct schema.
It sets`last_name` STRING NOT NULL, `title` STRING NOT NULL to both be nullable.
The cause of this is in the table planner when we convert SqlDataTypeSpec to
the RelDataType. We push the nullable of buyer onto all of its fields, losing
the children’s nullable status . In the debugger I see that it is taking the
nullable true (from buyer) and putting this on all the children in
[fixupNullability|http://example.com]https://github.com/davidradl/calcite/blob/ad2e843c5d9b3bec001d22e680ebe6b5de4e2078/core/src/main/java/org/apache/calcite/sql/SqlDataTypeSpec.java#L238.
I can see that the SqlDataTypeSpec has this information but it is not used.
Flink maps this directly to an Avro schema that incorreclty polices the Avro
payload.
In terms of fixing this, I wonder if the type could be created with the correct
nullable so it doesn’t need to be fixed up or whether we should pass down the
SqlDataTypeSpec so it has the right information to set the nullable status.
This subject has previously been raised on the dev list
[https://lists.apache.org/thread/s4nd9rk0fzckoctokl7kjdbtfbvxncy7] by
[~dwysakowicz]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
