Yes, that sentence is the source of all that confusion.
I learned it was misleading when reading part of the source code.
There are also some stack overflow answers that clarify auto-commit.
I think you are right, the documentation probably does not say
you must process all records in the batch
Hi Clement & Colin
Thanks for the discussion on this.
I think the biggest confusion is how the enable.auto.commit documentation
says "If true the consumer's offset will be periodically committed in the
background."
@Clement, you mention that there is documentation about what you must do
with
Indeed, but this is what the documentation says you must do with auto-commit.
I say it is a user error if you don't.
Regardless, I think this is a fairly common misconception, so it would not hurt
to debunk it explicitly in the documentation.
-Original Message-
From: Colin McCabe
Hi Clement,
You are assuming that the client application is single-threaded-- or at least
processes all the records before polling for more. This may or may not be the
case. But that is a fair point-- in this case, auto-commit would be safe.
best,
Colin
On Tue, Jan 29, 2019, at 16:23,
I had the same impression at some point but this is not how auto-commit works.
Auto-commit can only commit when the application comes back to poll and
if it decides to commit at that time, it will only commit the previous batch.
In your example, the app might come back and have to re-execute all
This is a very fair question. I suspect the answer is that people feel that
automatic offset commit is easier to get started with, even though it has the
error handling shortcomings you mention. It would be very hard to change the
default now, since it would break such a huge amount of code.
As the question indicates.
Should this not be default false? I think this is a bit nefarious to
someone launching their application into production without testing it
extensively around failure modes. I can see a scenario where a consumer
polls for events, processes them, produces to output
