fqshopify commented on PR #174: URL: https://github.com/apache/flink-connector-kafka/pull/174#issuecomment-4252157290
> By the way, do you run this in production already? Yes! We do run this in production and have been for the last year with multiple pipelines. > If you have any benchmarks or observations on the deserialization savings, I'd be interested to see them. I think all of our pipelines that are making use of this feature use the `debezium-avro-confluent` format. The way that format works [internally](https://github.com/apache/flink/blob/d8010df8795e1a8e4895d66b7b4861a85c8e0e24/flink-formats/flink-avro-confluent-registry/src/main/java/org/apache/flink/formats/avro/registry/confluent/debezium/DebeziumAvroDeserializationSchema.java#L102) is it first deserializes the message fully (using the schema-id encoded in the first 4 bytes of the message) before projection pushdown kicks in to select specific columns. So in this case the deserialization savings would be miniscule, if any. The big unlock for us was really the schema-evolution-tolerance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
