[ https://issues.apache.org/jira/browse/KAFKA-8713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheng Pan updated KAFKA-8713: ----------------------------- Description: Class JsonConverter line: 582 {code:java} private static JsonNode convertToJson(Schema schema, Object logicalValue) { if (logicalValue == null) { if (schema == null) // Any schema is valid and we don't have a default, so treat this as an optional schema return null; if (schema.defaultValue() != null) return convertToJson(schema, schema.defaultValue()); if (schema.isOptional()) return JsonNodeFactory.instance.nullNode(); throw new DataException("Conversion error: null value for field that is required and has no default value"); } .... } {code} h1.The Excepted: Value `null` is valid for an optional filed, even though the filed has a default value. Only when field is required, the converter return default value fallback when value is `null`. h1.The Current: Always return default value if `null` was given. h1. Addition: I'm not sure if the current behavior is the exactly expected, but at least on MySQL, a table define as {code:sql} create table t1 { name varchar(40) not null, create_time datetime default '1999-01-01 11:11:11' null, update_time datetime default '1999-01-01 11:11:11' null } {code} Just insert a record: {code:sql} INSERT INTO `t1` (`name`, `update_time`) VALUES ('kafka', null); {code} The result is: {code:json} { "name": "kafka", "create_time": "1999-01-01 11:11:11", "update_time": null } {code} But when I use debezium pull binlog and send the record to Kafka with JsonConverter, the result changed to: {code:json} { "name": "kafka", "create_time": "1999-01-01 11:11:11", "update_time": "1999-01-01 11:11:11" } {code} For details, see: https://issues.jboss.org/browse/DBZ-1064 was: Class JsonConverter line: 582 {code:java} private static JsonNode convertToJson(Schema schema, Object logicalValue) { if (logicalValue == null) { if (schema == null) // Any schema is valid and we don't have a default, so treat this as an optional schema return null; if (schema.defaultValue() != null) return convertToJson(schema, schema.defaultValue()); if (schema.isOptional()) return JsonNodeFactory.instance.nullNode(); throw new DataException("Conversion error: null value for field that is required and has no default value"); } .... } {code} h1.The Excepted: Value `null` is valid for an optional filed, even though the filed has a default value. Only when field is required, the converter return default value fallback when value is `null`. h1.The Current: Always return default value if `null` was given. h1. Addition: I'm not sure if the current behavior is the exactly expected, but at least on MySQL, a table define as {code:sql} create table t1 { name varchar(40) not null, create_time datetime default '1999-01-01 11:11:11' null, update_time datetime default '1999-01-01 11:11:11' null } {code} Just insert a record: {code:sql} INSERT INTO `t1` (`name`, `update_time`) VALUES ('kafka', null); {code} The result is: {code:json} { "name": "kafka", "create_time": "1999-01-01 11:11:11", "update_time": null } {code} But when I use debezium pull binlog and send the record to Kafka with JsonConverter, the result changed to: {code:json} { "name": "kafka", "create_time": "1999-01-01 11:11:11", "update_time": "1999-01-01 11:11:11" } {code} > Connect JsonConverter swallow null value on optional filed which has default > value > ---------------------------------------------------------------------------------- > > Key: KAFKA-8713 > URL: https://issues.apache.org/jira/browse/KAFKA-8713 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect > Affects Versions: 2.0.0, 2.0.1, 2.1.0, 2.2.0, 2.3.0, 2.2.1 > Reporter: Cheng Pan > Priority: Major > > Class JsonConverter line: 582 > {code:java} > private static JsonNode convertToJson(Schema schema, Object logicalValue) > { > if (logicalValue == null) { > if (schema == null) // Any schema is valid and we don't have a > default, so treat this as an optional schema > return null; > if (schema.defaultValue() != null) > return convertToJson(schema, schema.defaultValue()); > if (schema.isOptional()) > return JsonNodeFactory.instance.nullNode(); > throw new DataException("Conversion error: null value for field > that is required and has no default value"); > } > .... > } > {code} > h1.The Excepted: > Value `null` is valid for an optional filed, even though the filed has a > default value. > Only when field is required, the converter return default value fallback > when value is `null`. > h1.The Current: > Always return default value if `null` was given. > h1. Addition: > I'm not sure if the current behavior is the exactly expected, but at least on > MySQL, a table define as > {code:sql} > create table t1 { > name varchar(40) not null, > create_time datetime default '1999-01-01 11:11:11' null, > update_time datetime default '1999-01-01 11:11:11' null > } > {code} > Just insert a record: > {code:sql} > INSERT INTO `t1` (`name`, `update_time`) VALUES ('kafka', null); > {code} > The result is: > {code:json} > { > "name": "kafka", > "create_time": "1999-01-01 11:11:11", > "update_time": null > } > {code} > But when I use debezium pull binlog and send the record to Kafka with > JsonConverter, the result changed to: > {code:json} > { > "name": "kafka", > "create_time": "1999-01-01 11:11:11", > "update_time": "1999-01-01 11:11:11" > } > {code} > For details, see: https://issues.jboss.org/browse/DBZ-1064 -- This message was sent by Atlassian Jira (v8.3.4#803005)