Re: Flink Kafka connector consume from a single kafka partition

2020-02-21 Thread Robert Metzger
Hey Hemant, Are you able to reconstruct the ordering of the event, for example based on time or some sequence number? If so, you could create as many Kafka partitions as you need (for proper load distribution), disregarding any ordering at that point. Then you keyBy your stream in Flink, and

Re: Flink Kafka connector consume from a single kafka partition

2020-02-19 Thread hemant singh
Hi Arvid, Thanks for your response. I think I did not word my question properly. I wanted to confirm that if the data is distributed to more than one partition then the ordering cannot be maintained (which is documented). According to your response I understand if I set the parallelism to number

Re: Flink Kafka connector consume from a single kafka partition

2020-02-19 Thread Arvid Heise
Hi Hemant, Flink passes your configurations to the Kafka consumer, so you could check if you can subscribe to only one partition there. However, I would discourage that approach. I don't see the benefit to just subscribing to the topic entirely and have dedicated processing for the different

Flink Kafka connector consume from a single kafka partition

2020-02-18 Thread hemant singh
Hello Flink Users, I have a use case where I am processing metrics from different type of sources(one source will have multiple devices) and for aggregations as well as build alerts order of messages is important. To maintain customer data segregation I plan to have single topic for each customer