hello, I'm user in kafka
I want find commit method from consumer to zookeeper, but can't find it
I want to make structure like that,
hadoop-consumer: 1. it get message
2. it write message to hadoop hdfs
3. it have to get message again from what it read ,when it is fault and recover.
( for example, consumer was reading message
suddenly it was fault. during fault time, message is producing.
after recovering, consumer have to read message from what it didn't read
offset, not from first)
problem is 3.
I read about offset, written that
after consumer read message, it commit to zookeeper. using that offset ( ex,
getbeforeoffset() ), can read message again.
But, using this method offer that reading from first(offset-1) to last. And if
use not it, consumer can't get messages produced during fault
So, I want find any source or method to satisfy my structure, is there any
thing to help me?
thank you for reading
please give some help
