We are using kafka for auditing services. We have a scenario where in there is bulk data and one in a list can fail. What we require is, we want to keep a queue of failed offsets and retry only the failed offsets. We tried to revert to the previous offset in case of failure, but that has either of the two issues:
1) Either the offset stays at the last successful one, and never processes the future offsets. 2) All the messages after the failed one is tried again, even if they were processed before. Basically is there a way in which we maintain a failed offset queue in kafka itself and retry that queue at preset intervals. Thanks, Roshni Thomas
