Cliff Rhyne created KAFKA-2725:
----------------------------------
Summary: high level consumer rebalances with auto-commit disabled
should throw an exception
Key: KAFKA-2725
URL: https://issues.apache.org/jira/browse/KAFKA-2725
Project: Kafka
Issue Type: Bug
Components: clients
Affects Versions: 0.8.2.1
Environment: Experienced on Java running in linux
Reporter: Cliff Rhyne
Auto-commit is a very resilient mode. Drops in zookeeper sessions due to JVM
garbage collection, network, rebalance or other interference are handled
gracefully within the kafka client.
Systems still can drop due to unexpected gc or network behavior. My proposal
is to handle this drop better when auto-commit is turned off:
- If a rebalance or similar occur (which cause the offset to get reverted in
the client), check and see if the client was assigned back to the same
partition or a different one. If it's the same partition, find the place last
consumed (it doesn't do this today for us). This is to make a graceful
recovery.
- If the partition assignment changes (which can mean duplicate data is getting
processed), throw an exception back to the application code. This lets the
application code handle this exception-case with respect to the work it's doing
(with might be transactional). Failing "silently" (yes it's still getting
logged) is very dangerous in our situation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
