Well done finding that thread. First step, before we jump into a discussion, please log a jira case. (The original thread made great progress analyzing the problem, and Ruben asked the OP to log a case, but it never happened.)
> On Sep 11, 2024, at 5:03 AM, David Radley <[email protected]> wrote: > > I notice https://lists.apache.org/thread/s4nd9rk0fzckoctokl7kjdbtfbvxncy7 > that talks about this and poses some questions on how we can get round this. > It sounds like this is a current limitation of Calcite, is there appetite in > the community for this to be changed? > >> On 2024/09/11 08:27:14 David Radley wrote: >> Hi, >> good question. Without this method, I guess the children would always be >> mandatory. So for nullable parent it is making all the children optional. >> For our case we want to be able to honour the nullable specified in the SQL >> for each child, so the existing approach is not sufficient, >> kind regards, David. >> >> >>> On 2024/09/10 15:15:14 Julian Hyde wrote: >>> I haven’t read the code in a while. What is fixupNullabilty trying to >>> achieve? >>> >>>> On Sep 10, 2024, at 3:40 AM, David Radley <[email protected]> wrote: >>>> >>>> >>>> Hello, >>>> I am using flink and have a table definition like this. >>>> CREATE TABLE source_1 >>>> ( >>>> `order_id` STRING NOT NULL, >>>> `order_time` STRING NOT NULL, >>>> `buyer` ROW< >>>> `first_name` STRING, >>>> `last_name` STRING NOT NULL, >>>> `title` STRING NOT NULL >>>>> >>>> ) >>>> >>>> WITH ( >>>> 'connector' = 'kafka', >>>> 'topic' = 'mytopic', >>>> 'properties.bootstrap.servers' = 'localhost:9092', >>>> 'value.format' = 'avro', >>>> 'value.fields-include' = 'ALL', >>>> 'scan.startup.mode' = 'earliest-offset' >>>> ); >>>> >>>> This errors as code in Calcite does not create the correct schema. It >>>> `last_name` STRING NOT NULL, `title` STRING NOT NULL to both be >>>> nullable. >>>> >>>> The cause of this is in the table planner when we convert SqlDataTypeSpec >>>> to the RelDataType. We push the nullable of buyer onto all of its fields, >>>> losing the children’s nullable status . >>>> >>>> In the debugger I see that it is taking the nullable true (from buyer) and >>>> putting this on all the children in fixupNullability. I can see that the >>>> SqlDataTypeSpec has this information but it is not used. >>>> >>>> In terms of fixing this, I wonder if the type could be created with the >>>> correct nullable so it doesn’t need to be fixed up or whether we should >>>> pass down the SqlDataTypeSpec so it has the right information to set the >>>> nullable status. I am new with the cee base so would appreciate your >>>> thoughts around this – I am happy to implement the change if we get >>>> consensus. WDYT? >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> Unless otherwise stated above: >>>> >>>> IBM United Kingdom Limited >>>> Registered in England and Wales with number 741598 >>>> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU >>> >>
