Hi guys,
I have a dataflow in a Nifi 3-node clustered environment reading from a kafka topic and writing to a mongodb collection target1. We are not filtering any messages in the dataflow as well. The groupid for this consumer is test1 and this consumer has been active since two years >From at least a month, the business customers have been reporting us that they >are missing data in the target which are they sure to be publishing. Even the >kafka team helped us searching the message(s) on the kafka brokers which the >customers claim that they were sure to be publishing. So its evident that the >consumer is not picking them up. Now, I did set-up a new dataflow in nifi duplicating the original dataflow. I made two differences. New group-id test2 for reading messages and new target collection target2 for writing the data. Apparently this duplicate dataflow is consuming the expected number of messages. Now, I changed the groupid in the original dataflow from test1 to newgrouped and the rest of the consumekafka processor configuration remains same including the offset reset which is latest. Both the original and duplicate dataflows are running from quite some time but still the issue exists with the original dataflow. The duplicate dataflow is keep on doing good consuming the expected number of messages, parsing and loading the processed data to the target. Please advise what could be the issue and how to resolve this. Additional note: Nifi Version: 1.21.0 Number of concurrent threads on the consumekafka processor: 2 Number of kafka partitions on the topic: 5 Thanks The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
