[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry

Github user medcv commented on the issue: https://github.com/apache/flink/pull/6083 @tillrohrmann PR has been updated! I had to do some changes: - map `User` object to `String` in `TestAvroConsumerConfluent` class - use `read_messages_from_kafka` instead of `read_messages_fr

[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry

Github user medcv commented on the issue: https://github.com/apache/flink/pull/6083 @tillrohrmann make sense. I created this ticket https://issues.apache.org/jira/browse/FLINK-9679 for implementing `AvroSerializationConfluentSchema`. I will make a PR for it shortly. Mean time

[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry

Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/6083 This is a good point @medcv. However, I think we should tackle adding a `AvroSerializationConfluentSchema` as an orthogonal step. What about removing it from this PR which covers the existing in

[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry

Github user medcv commented on the issue: https://github.com/apache/flink/pull/6083 @tillrohrmann here is my two cents: By converting the events to `String` and sending them to Kafka there is a high risk to produce a `bad events` and we will lose the benifit of Schema Registr

[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry

Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/6083 Why don't we output the events using the `SimpleStringSchema` without verification of the schema. That way we would no longer need `AvroSerializationConfluentSchema` and could get rid of the `ka

[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry

Github user medcv commented on the issue: https://github.com/apache/flink/pull/6083 @tillrohrmann I made the changes and used `ConfluentRegistryAvroDeserializationSchema` for the Deserializations. I still using `AvroSerializationConfluentSchema` as we need to compare each income

[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry

Github user medcv commented on the issue: https://github.com/apache/flink/pull/6083 @dawidwys do u have an example using `ConfluentRegistryAvroDeserializationSchema`. I have some difficulties to make it work ---

[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry

Github user medcv commented on the issue: https://github.com/apache/flink/pull/6083 @tillrohrmann PR is updated please review! I will also look into `ConfluentRegistryAvroDeserializationSchema` ---

[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry

Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/6083 I agree with @dawidwys. We should use `ConfluentRegistryAvroDeserializationSchema` for the test. It is ok that this end-to-end test would then only go to Flink `1.6`. The benefit is that we only

[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry

Github user medcv commented on the issue: https://github.com/apache/flink/pull/6083 @dawidwys Thanks for the review! I will look into that PR. This test is for Release 1.4 and 1.5 and provide a proof that Flink V4.0 and V5.0 can work with ConfluentSchema Registry properly. As PR you m

[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry

Github user dawidwys commented on the issue: https://github.com/apache/flink/pull/6083 Just my two cents, we've recently introduced a `DeserializationSchema` that uses Confluent Schema Registry to decode data. Could you reuse it rather than introduce a new one. The PR in which it was

[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry

Github user medcv commented on the issue: https://github.com/apache/flink/pull/6083 @tillrohrmann Thanks for the review! I will go through them and will make the changes shortly. ---

[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry

Github user dmpour23 commented on the issue: https://github.com/apache/flink/pull/6083 @medcv Regarding KeyedDeserialization/KeyedSerializationSchema. I assume that the blocking part is the casting in deserializeKey, serializeKey, ---

[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry

Github user medcv commented on the issue: https://github.com/apache/flink/pull/6083 @dmpour23 Thanks! I updated the import as`*.util.serialization` was Deprecated! ---

[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry

Github user dmpour23 commented on the issue: https://github.com/apache/flink/pull/6083 Hi would this work for KeyedDeserialization/KeyedSerializationSchema. Should the import be: > org.apache.flink.api.common.serialization.DeserializationSchema ---

[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry

Github user medcv commented on the issue: https://github.com/apache/flink/pull/6083 @tillrohrmann Thanks a lot for doing the review! ---

[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry

Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/6083 Thanks for opening the PR @medcv. I will try to take a look next week. ---

[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry

Github user medcv commented on the issue: https://github.com/apache/flink/pull/6083 @tillrohrmann I would appreciate if you review or assign a reviewer to this PR. ---

[GitHub] flink issue #6083: [FLINK-8983] End-to-end test: Confluent schema registry

Github user medcv commented on the issue: https://github.com/apache/flink/pull/6083 @tillrohrmann please review ---