Thanks for the response Thomas. My quick doubt is..
I have around 30 partitions of kafka topic, And all of them have messages ordered at partition level. So, when I consume those messages using single consumer[with ONE_TO_MANY strategy set], still the ordering doesn’t work ? My messages in topic are guaranteed to be ordered at partition level. Thanks a lot in advance for your response. Regards, Raja. From: Thomas Weise <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Tuesday, June 7, 2016 at 5:52 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: kafka input is processing records in a jumbled order Raja, Are you expecting ordering across multiple Kafka partitions? All messages from a given Kafka partition are received by the same consumer and thus will be ordered. However, when messages come from multiple partitions there is no such guarantee. Thomas On Tue, Jun 7, 2016 at 3:34 PM, Raja.Aravapalli <[email protected]<mailto:[email protected]>> wrote: Hi I have built a DAG, that reads from kafka and in the next operators, does lookup to a hbase table and update hbase table based on some business logic. Some times my operator which does hbase lookup and update in the same operator(Custom written), is processing the records it receives from kafka in a jumbled order, which is causing, many records being ignored from processing!! I am not using any parallel partitions/instance, and with KafkaInputOperator I am using only partition strategy ONE_TO_MANY. I am very new to Apex. I expected, Apex will guarantee the ordering. Can someone pls share your knowledge on the issue…? Thanks a lot in advance… Regards, Raja.
