Andy Bryant created KAFKA-9390:
----------------------------------
Summary: Non-key joining of KTable not compatible with confluent
avro serdes
Key: KAFKA-9390
URL: https://issues.apache.org/jira/browse/KAFKA-9390
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 2.4.0
Reporter: Andy Bryant
I was trying out the new one-to-many KTable joins against some CDC data in Avro
format and kept getting serialisation errors.
{code:java}
org.apache.kafka.common.errors.SerializationException: Error registering Avro
schema:
{"type":"record","name":"Key","namespace":"dbserver1.inventory.orders","fields":[
{"name":"order_number","type":"int"}
],"connect.name":"dbserver1.inventory.orders.Key"}
Caused by:
io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException:
Schema being registered is incompatible with an earlier schema; error code: 409
{code}
Both tables have avro keys of different types (one is an order key, the other a
customer key).
This looks like it will cause issues.
[https://github.com/apache/kafka/blob/2.4/streams/src/main/java/org/apache/kafka/streams/kstream/internals/foreignkeyjoin/CombinedKeySchema.java#L57-L60]
They will both attempt to register schemas with the same subject to the schema
registry which will fail a backward compatibility check.
I also noticed in the schema registry there were some subjects that didn't have
the application id prefix. This is probably caused by this...
[https://github.com/apache/kafka/blob/2.4/streams/src/main/java/org/apache/kafka/streams/kstream/internals/foreignkeyjoin/ForeignJoinSubscriptionSendProcessorSupplier.java#L88]
Where here {{repartitionTopicName}} doesn't have the application prefix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
