kasparjarek opened a new pull request, #31: URL: https://github.com/apache/pulsar-connectors/pull/31
Pulsar Kafka Connect adaptor previously used the same converters for both data and offset storage. This could cause various issues. For example when data were using AvroConverter, offsets were serialized using MockSchemaRegistryClient (in memory only). After a connector restart, the fresh MockSchemaRegistryClient had no schema records, causing deserialization to fail with "Subject Not Found; error code: 40401" and the connector losing its offset position. Kafka Connect do not reuse the data converters for offset, but creates new JSON converters configured with `schema.enable` set to `false`. Thus the adaptor was changed to have the same behavior. This is a breaking change for connectors that previously stored offsets in a non-JSON format, those offsets do not have to be readable after upgrade. The offsets probably weren't readable even before this fix (as we can see for the Avro converter). But since we cannot be sure what converters users used and how they behaved, this change should be probably included in a major release. Note: The tests for adaptor currently do not compile. We are waiting for matching pulsar artifact to be published. The fix was verified E2E by manual testing. Fixes #30 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
