Hi Jason,
Thanks for additional clarification.
So the next version of the OffsetCommit API will return an
INVALID_GROUP_ID error for empty group ids; but on the client side we call
the older version of the client until the next major release.
The table below should summarize this.
+-----------------------------------------------------+
| Client (group.id="") |
+-----------------------------------------------------+
| pre-2.1 | 2.1 | 3.0 |
+-----+-----------+---------+------------------------+------------------+
| | V5 (cur.) | works | works | works |
+ API +-----------+---------+------------------------+------------------+
| | V6 | N/A | N/A (calls V5/warning) | INVALID_GROUP_ID |
+-----+-----------+---------+------------------------+------------------+
Assumptions:
* 2.1: The target release version for this KIP
* 3.0: The next major release
Please advise if you see an issue; otherwise, I'll update the KIP
accordingly.
Thanks!
--Vahid
From: Jason Gustafson <ja...@confluent.io>
To: dev <dev@kafka.apache.org>
Date: 07/25/2018 12:08 AM
Subject: ***UNCHECKED*** Re: [DISCUSS] KIP-289: Improve the default
group id behavior in KafkaConsumer
Hey Vahid,
Sorry for the confusion. I think we all agree that going forward, we
shouldn't support the empty group id, so the question is just around
compatibility. I think we have to bump the OffsetCommit API version so
that
old clients which are unknowingly depending on the default empty group id
will continue to work with new brokers. For new versions of the client, we
can either drop support for the empty group id immediately or we can give
users a grace period. I was thinking we would do the latter. We can change
the default group.id, but in the case that a user has explicitly
configured
the empty group, then we can just use an old version of the OffsetCommit
API which still supports it. In a future release, we can drop this support
and only use the latest OffsetCommit version. Does that make sense?
Thanks,
Jason
On Tue, Jul 24, 2018 at 12:36 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:
> Hi Jason,
>
> Thanks for clarifying.
>
> So if we are going to continue supporting the empty group id as before
> (with only an addition of a deprecation warning), and disable
> enable.auto.commit for the new default (null) group id on the client
side,
> do we really need to bump up the OffsetCommit version?
>
> You mentioned "If an explicit empty string is configured for the group
id,
> then maybe we keep the current behavior for compatibility" which makes
> sense to me, but I find it in conflict with your earlier suggestion "we
> just need to bump the OffsetCommit request API and only accept the
offset
> commit for older versions.". Maybe I'm missing something?
>
> Thanks!
> --Vahid
>
>
>
>
> From: Jason Gustafson <ja...@confluent.io>
> To: dev <dev@kafka.apache.org>
> Date: 07/23/2018 10:52 PM
> Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> behavior in KafkaConsumer
>
>
>
> Hey Vahid,
>
> Thanks for the updates. Just to clarify, I was suggesting that we
disable
> enable.auto.commit only if no explicit group.id is configured. If an
> explicit empty string is configured for the group id, then maybe we keep
> the current behavior for compatibility. We can log a warning mentioning
> the
> deprecation and we can use the old version of the OffsetCommit API that
> allows the empty group id. In a later release, we can drop this support
in
> the client. Does that seem reasonable?
>
> By the way, instead of using the new ILLEGAL_OFFSET_COMMIT error code,
> couldn't we use INVALID_GROUP_ID?
>
> Thanks,
> Jason
>
>
>
> On Mon, Jul 23, 2018 at 5:14 PM, Stanislav Kozlovski
> <stanis...@confluent.io
> > wrote:
>
> > Hey Vahid,
> >
> > No I don't see an issue with it. I believe it to be the best approach.
> >
> > Best,
> > Stanisav
> >
> > On Mon, Jul 23, 2018 at 12:41 PM Vahid S Hashemian <
> > vahidhashem...@us.ibm.com> wrote:
> >
> > > Hi Stanislav,
> > >
> > > Thanks for the feedback.
> > > Do you see an issue with using `null` as the default group id (as
> > > addressed by Jason in his response)?
> > > This default group id would not support offset commits and consumers
> > would
> > > use `auto.offset.reset` config when there is no current offset.
> > >
> > > Thanks.
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From: Stanislav Kozlovski <stanis...@confluent.io>
> > > To: dev@kafka.apache.org
> > > Date: 07/20/2018 11:09 AM
> > > Subject: Re: [DISCUSS] KIP-289: Improve the default group id
> > > behavior in KafkaConsumer
> > >
> > >
> > >
> > > I agree with Jason's notion that
> > > > implicit use of the empty group.id to commit offsets is more
likely
> > to
> > > be causing users unexpected problems than actually providing a
useful
> > > capability.
> > > I was initially confused that this is the behavior when
investigating
> a
> > > new-ish JIRA issue <
> > >
>
https://issues.apache.org/jira/browse/KAFKA-6758
>
> > > > about
> > > the same topic.
> > > So, +1 to deprecating "" as a group.id
> > >
> > > The question after that becomes what the *default* value should be -
> > > should
> > > we:
> > > a) treat an unconfigured group.id consumer as a sort of intermittent
> > > consumer where you don't store offsets at all (thereby making the
user
> > > explicitly sign up for them)
> > > b) have a default value which makes use of them? I sort of like the
> > > former.
> > >
> > > @Dhruvil, thinking about it at a high-level - yes. I can't think of
a
> > > situation where it makes sense to name something an empty string as
> far
> > as
> > > I'm aware - to me it seems like potential for confusion
> > >
> > >
> > > On Fri, Jul 20, 2018 at 10:22 AM Rajini Sivaram
> <rajinisiva...@gmail.com
> > >
> > > wrote:
> > >
> > > > +1 to deprecate use of "" as group.id since it is odd to have a
> > resource
> > > > name that you cannot set ACLs for. Agree, we have to support older
> > > clients
> > > > though.
> > > >
> > > > Thanks,
> > > >
> > > > Rajini
> > > >
> > > > On Fri, Jul 20, 2018 at 5:25 PM, Jason Gustafson
> <ja...@confluent.io>
> > > > wrote:
> > > >
> > > > > Hi Vahid,
> > > > >
> > > > > Sorry for getting to this so late. I think there are two things
> here:
> > > > >
> > > > > 1. The use of "" as a groupId has always been a dubious practice
> at
> > > best.
> > > > > We definitely ought to deprecate its use in the client. Perhaps
in
> > the
> > > > next
> > > > > major release, we can remove support completely. However, since
> older
> > > > > clients depend on it, we may have to continue letting the broker
> > > support
> > > > it
> > > > > to some extent. Perhaps we just need to bump the OffsetCommit
> request
> > > API
> > > > > and only accept the offset commit for older versions. You
probably
> > > have
> > > > to
> > > > > do this anyway if you want to introduce the new error code since
> old
> > > > > clients will not expect it.
> > > > >
> > > > > 2. There should be a way for the consumer to indicate that it
has
> no
> > > > group
> > > > > id and will not commit offsets. This is an explicit instruction
> that
> > > the
> > > > > consumer should not bother with coordinator lookup and such. We
> > > currently
> > > > > have some brittle logic in place to let users avoid the
> coordinator
> > > > lookup,
> > > > > but it is a bit error-prone. I was hoping that we could change
the
> > > > default
> > > > > value of group.id to be null so that the user had to take an
> > explicit
> > > > > action to opt into coordinator management (groups or offsets).
> > > However,
> > > > it
> > > > > is true that some users may be unknowingly depending on offset
> > storage
> > > if
> > > > > they are using both the default group.id and the default
> > > > > enable.auto.commit. Perhaps one option is to disable
> > > enable.auto.commit
> > > > > automatically if no group.id is specified? I am not sure if
there
> > are
> > > > any
> > > > > drawbacks, but my feeling is that implicit use of the empty
> group.id
> > > to
> > > > > commit offsets is more likely to be causing users unexpected
> problems
> > > > than
> > > > > actually providing a useful capability.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Thanks,
> > > > > Jason
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, May 28, 2018 at 9:50 AM, Vahid S Hashemian <
> > > > > vahidhashem...@us.ibm.com> wrote:
> > > > >
> > > > > > Hi Viktor,
> > > > > >
> > > > > > Thanks for sharing your opinion.
> > > > > > So you're in favor of disallowing the empty ("") group id
> > altogether
> > > > > (even
> > > > > > for fetching).
> > > > > > Given that ideally no one should be using the empty group id
(at
> > > least
> > > > in
> > > > > > a production setting) I think the impact would be minimal in
> either
> > > > case.
> > > > > >
> > > > > > But as you said, let's hear what others think and I'd be happy
> to
> > > > modify
> > > > > > the KIP if needed.
> > > > > >
> > > > > > Regards.
> > > > > > --Vahid
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > From: Viktor Somogyi <viktorsomo...@gmail.com>
> > > > > > To: dev <dev@kafka.apache.org>
> > > > > > Date: 05/28/2018 05:18 AM
> > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
group
> id
> > > > > > behavior in KafkaConsumer
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hi Vahid,
> > > > > >
> > > > > > (with the argument that using the default group id for offset
> > commit
> > > > > > should not be the user's intention in practice).
> > > > > >
> > > > > > Yea, so in my opinion too this use case doesn't seem too
> practical.
> > > > Also
> > > > > I
> > > > > > think breaking the offset commit is not smaller from this
> > > perspective
> > > > > than
> > > > > > breaking fetch and offset fetch. If we suppose that someone
uses
> > the
> > > > > > default group id and we break the offset commit then that
might
> be
> > > > harder
> > > > > > to detect than breaking the whole thing altogether. (If we
think
> > > about
> > > > an
> > > > > > upgrade situation.)
> > > > > > So since we think it is not a practical use case, I think it
> would
> > > be
> > > > > > better to break altogether but ofc that's just my 2 cents :).
> Let's
> > > > > gather
> > > > > > other's input as well.
> > > > > >
> > > > > > Cheers,
> > > > > > Viktor
> > > > > >
> > > > > > On Fri, May 25, 2018 at 5:43 PM, Vahid S Hashemian <
> > > > > > vahidhashem...@us.ibm.com> wrote:
> > > > > >
> > > > > > > Hi Victor,
> > > > > > >
> > > > > > > Thanks for reviewing the KIP.
> > > > > > >
> > > > > > > Yes, to minimize the backward compatibility impact, there
> would
> > be
> > > no
> > > > > > harm
> > > > > > > in letting a stand-alone consumer consume messages under a
""
> > > group
> > > > id
> > > > > > (as
> > > > > > > long as there is no offset commit).
> > > > > > > It would have to knowingly seek to an offset or rely on the
> > > > > > > auto.offset.reset config for the starting offset.
> > > > > > > This way the existing functionality would be preserved for
the
> > > most
> > > > > part
> > > > > > > (with the argument that using the default group id for
offset
> > > commit
> > > > > > > should not be the user's intention in practice).
> > > > > > >
> > > > > > > Does it seem reasonable?
> > > > > > >
> > > > > > > Thanks.
> > > > > > > --Vahid
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > From: Viktor Somogyi <viktorsomo...@gmail.com>
> > > > > > > To: dev <dev@kafka.apache.org>
> > > > > > > Date: 05/25/2018 04:56 AM
> > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
> group
> > > id
> > > > > > > behavior in KafkaConsumer
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hi Vahid,
> > > > > > >
> > > > > > > When reading your KIP I coldn't fully understand why did you
> > > decide
> > > > at
> > > > > > > failing with "offset_commit" in case #2? Can't we fail with
an
> > > empty
> > > > > > group
> > > > > > > id even in "fetch" or "fetch_offset"? What was the reason
for
> > > > deciding
> > > > > > to
> > > > > > > fail at "offset_commit"? Was it because of upgrade
> compatibility
> > > > > > reasons?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Viktor
> > > > > > >
> > > > > > > On Thu, May 24, 2018 at 12:06 AM, Ted Yu
<yuzhih...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Looks good to me.
> > > > > > > > -------- Original message --------From: Vahid S Hashemian
<
> > > > > > > > vahidhashem...@us.ibm.com> Date: 5/23/18 11:19 AM
> > (GMT-08:00)
> > > > To:
> > > > > > > > dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-289:
Improve
> > the
> > > > > > default
> > > > > > > > group id behavior in KafkaConsumer
> > > > > > > > Hi Ted,
> > > > > > > >
> > > > > > > > Thanks for reviewing the KIP. I updated the KIP and
> introduced
> > > an
> > > > > > error
> > > > > > > > code for the scenario described.
> > > > > > > >
> > > > > > > > --Vahid
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > From: Ted Yu <yuzhih...@gmail.com>
> > > > > > > > To: dev@kafka.apache.org
> > > > > > > > Date: 04/27/2018 04:31 PM
> > > > > > > > Subject: Re: [DISCUSS] KIP-289: Improve the default
> > group
> > > id
> > > > > > > > behavior in KafkaConsumer
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > bq. If they attempt an offset commit they will receive an
> > error.
> > > > > > > >
> > > > > > > > Can you outline what specific error would be encountered ?
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > On Fri, Apr 27, 2018 at 2:17 PM, Vahid S Hashemian <
> > > > > > > > vahidhashem...@us.ibm.com> wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > I have drafted a proposal for improving the behavior of
> > > > > > KafkaConsumer
> > > > > > > > when
> > > > > > > > > using the default group id:
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > >
>
https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>
> > >
> > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > > 289%3A+Improve+the+default+group+id+behavior+in+
> > KafkaConsumer
> > > > > > > > > The proposal based on the issue and suggestion reported
in
> > > > > > KAFKA-6774.
> > > > > > > > >
> > > > > > > > > Your feedback is welcome!
> > > > > > > > >
> > > > > > > > > Thanks.
> > > > > > > > > --Vahid
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best,
> > > Stanislav
> > >
> > >
> > >
> > >
> > >
> >
> > --
> > Best,
> > Stanislav
> >
>
>
>
>
>
