Hi Clement & Colin Thanks for the discussion on this.
I think the biggest confusion is how the enable.auto.commit documentation says "If true the consumer's offset will be periodically committed in the background." @Clement, you mention that there is documentation about what you must do with auto-commit. I am looking at https://kafka.apache.org/documentation and I cannot seem to locate it. Could you point me to it? I would be fine with making a simple PR to improve the documentation on this - I think it will help straighten things out. Thanks again, Adam On Tue, Jan 29, 2019 at 7:35 PM Pellerin, Clement <clement_pelle...@ibi.com> wrote: > Indeed, but this is what the documentation says you must do with > auto-commit. > I say it is a user error if you don't. > > Regardless, I think this is a fairly common misconception, so it would not > hurt > to debunk it explicitly in the documentation. > > -----Original Message----- > From: Colin McCabe [mailto:cmcc...@apache.org] > Sent: Tuesday, January 29, 2019 7:26 PM > To: dev@kafka.apache.org > Subject: Re: Why is enable.auto.commit=true the default value for consumer? > > Hi Clement, > > You are assuming that the client application is single-threaded-- or at > least processes all the records before polling for more. This may or may > not be the case. But that is a fair point-- in this case, auto-commit > would be safe. > > best, > Colin > > On Tue, Jan 29, 2019, at 16:23, Pellerin, Clement wrote: > > I had the same impression at some point but this is not how auto-commit > works. > > Auto-commit can only commit when the application comes back to poll and > > if it decides to commit at that time, it will only commit the previous > batch. > > In your example, the app might come back and have to re-execute all > > the records in the uncommitted batch but it will never skip over > > unprocessed records. > > > > -----Original Message----- > > From: Adam Bellemare [mailto:adam.bellem...@gmail.com] > > Sent: Tuesday, January 29, 2019 3:54 PM > > To: dev@kafka.apache.org > > Subject: Why is enable.auto.commit=true the default value for consumer? > > > > As the question indicates. > > > > Should this not be default false? I think this is a bit nefarious to > > someone launching their application into production without testing it > > extensively around failure modes. I can see a scenario where a consumer > > polls for events, processes them, produces to output topic, and commits > the > > offsets. Say it takes 30 seconds for a batch. If it fails halfway > through, > > upon restarting it will skip everything that was unprocessed/unpublished > up > > to the committed offset. > > > > Is there a historic reason why it's set to default true? Is it because to > > change it to default false it could affect the upgrade path of previous > > implementations? > > > > Adam > > >
