Hi,
I am looking to get the Avro format to support Non Nullable fields within
objects or arrays. It works with Confluent Avro. I notice
https://issues.apache.org/jira/browse/CALCITE-4085 where it looks like Calcite
was changed to allow this capability to work with Flink.
https://github.com/apache/flink/pull/24916/files#diff-c2ed59d02b6bec354790442fd97c7694676eaa4199425353d7ef6cde1304c2e0
might effect this also.
I have a table definition of
CREATE TABLE source_7
(
`order_id` STRING,
`order_time` STRING,
`buyer` ROW<
`first_name` STRING,
`last_name` STRING NOT NULL,
`title` STRING NOT NULL
>
)
WITH (
'connector' = 'kafka',
'topic' = 'vivekavro',
'properties.bootstrap.servers' = 'localhost:9092',
'value.format' = 'avro',
'value.fields-include' = 'ALL',
'scan.startup.mode' = 'earliest-offset'
);
And an event of shape:
{
"order_id": "12345",
"order_time": "1234",
"buyer": {
"first_name": "hvcwc",
"last_name": "hvcwc2",
"title": "hvcwc3"
}
}
When I issue a select it fails deserialize, internally the writer schema has
"name" : "last_name", "type" : [ "null", "string" ],
"name" : "titie", "type" : [ "null", "string" ],
So it has lost the non-nullable. I would have expected "type" : "string", for
last_name and title.
I have done a little digging. It appears that the issue is in the
createSqlTableConverter.
In the debugger I see:
Buyer has a nullable followed by 2 non nullable fields.
The FieldsDataType are all nullable. This looks like it has lost the nullable
hint.
LogicalType does not have the concept of nullable.
fromLogicalTypeToDataType creates a DataType from LogicalType and results in
the fields being set as nullables.
This looks like it could be cause of the behaviour we are seeing or am I
missing something?
WDYT?
Kind regards, David.
Unless otherwise stated above:
IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
