Hi, I am looking to get the Avro format to support Non Nullable fields within objects or arrays. It works with Confluent Avro. I notice https://issues.apache.org/jira/browse/CALCITE-4085 where it looks like Calcite was changed to allow this capability to work with Flink. https://github.com/apache/flink/pull/24916/files#diff-c2ed59d02b6bec354790442fd97c7694676eaa4199425353d7ef6cde1304c2e0 might effect this also.
I have a table definition of CREATE TABLE source_7 ( `order_id` STRING, `order_time` STRING, `buyer` ROW< `first_name` STRING, `last_name` STRING NOT NULL, `title` STRING NOT NULL > ) WITH ( 'connector' = 'kafka', 'topic' = 'vivekavro', 'properties.bootstrap.servers' = 'localhost:9092', 'value.format' = 'avro', 'value.fields-include' = 'ALL', 'scan.startup.mode' = 'earliest-offset' ); And an event of shape: { "order_id": "12345", "order_time": "1234", "buyer": { "first_name": "hvcwc", "last_name": "hvcwc2", "title": "hvcwc3" } } When I issue a select it fails deserialize, internally the writer schema has "name" : "last_name", "type" : [ "null", "string" ], "name" : "titie", "type" : [ "null", "string" ], So it has lost the non-nullable. I would have expected "type" : "string", for last_name and title. I have done a little digging. It appears that the issue is in the createSqlTableConverter. In the debugger I see: Buyer has a nullable followed by 2 non nullable fields. The FieldsDataType are all nullable. This looks like it has lost the nullable hint. LogicalType does not have the concept of nullable. fromLogicalTypeToDataType creates a DataType from LogicalType and results in the fields being set as nullables. This looks like it could be cause of the behaviour we are seeing or am I missing something? WDYT? Kind regards, David. Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU