I notice https://lists.apache.org/thread/s4nd9rk0fzckoctokl7kjdbtfbvxncy7 that talks about this and poses some questions on how we can get round this. It sounds like this is a current limitation of Calcite, is there appetite in the community for this to be changed?
On 2024/09/11 08:27:14 David Radley wrote: > Hi, > good question. Without this method, I guess the children would always be > mandatory. So for nullable parent it is making all the children optional. For > our case we want to be able to honour the nullable specified in the SQL for > each child, so the existing approach is not sufficient, > kind regards, David. > > > On 2024/09/10 15:15:14 Julian Hyde wrote: > > I haven’t read the code in a while. What is fixupNullabilty trying to > > achieve? > > > > > On Sep 10, 2024, at 3:40 AM, David Radley <david_rad...@uk.ibm.com> wrote: > > > > > > > > > Hello, > > > I am using flink and have a table definition like this. > > > CREATE TABLE source_1 > > > ( > > > `order_id` STRING NOT NULL, > > > `order_time` STRING NOT NULL, > > > `buyer` ROW< > > > `first_name` STRING, > > > `last_name` STRING NOT NULL, > > > `title` STRING NOT NULL > > > > > > > ) > > > > > > WITH ( > > > 'connector' = 'kafka', > > > 'topic' = 'mytopic', > > > 'properties.bootstrap.servers' = 'localhost:9092', > > > 'value.format' = 'avro', > > > 'value.fields-include' = 'ALL', > > > 'scan.startup.mode' = 'earliest-offset' > > > ); > > > > > > This errors as code in Calcite does not create the correct schema. It > > > `last_name` STRING NOT NULL, `title` STRING NOT NULL to both be > > > nullable. > > > > > > The cause of this is in the table planner when we convert SqlDataTypeSpec > > > to the RelDataType. We push the nullable of buyer onto all of its fields, > > > losing the children’s nullable status . > > > > > > In the debugger I see that it is taking the nullable true (from buyer) > > > and putting this on all the children in fixupNullability. I can see that > > > the SqlDataTypeSpec has this information but it is not used. > > > > > > In terms of fixing this, I wonder if the type could be created with the > > > correct nullable so it doesn’t need to be fixed up or whether we should > > > pass down the SqlDataTypeSpec so it has the right information to set the > > > nullable status. I am new with the cee base so would appreciate your > > > thoughts around this – I am happy to implement the change if we get > > > consensus. WDYT? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Unless otherwise stated above: > > > > > > IBM United Kingdom Limited > > > Registered in England and Wales with number 741598 > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU > > >
