[DISCUSS] Add missing API to old high level consumer

[Jiangjie Qin]

Add the DISCUSS prefix to the email title : )

From: Jiangjie Qin mailto:j...@linkedin.com>>
Date: Tuesday, May 12, 2015 at 4:51 PM
To: "dev@kafka.apache.org" 
mailto:dev@kafka.apache.org>>
Subject: Add missing API to old high level consumer

Hi,

I just noticed that in KAFKA-1650 (which is before we use KIP) we added an 
offset commit method in high level consumer that commits offsets using a user 
provided offset map.

public void commitOffsets(Map 
offsetsToCommit, boolean retryOnFailure);

This method was added to all the Scala classes but I forgot to add it to Java 
API of ConsumerConnector. (Already regretting now. . .)
This method is very useful in several cases and has been asked for from time to 
time. For example, people have several threads consuming messages and 
processing them. Without this method, one thread will unexpectedly commit 
offsets for another thread, thus might lose some messages if something goes 
wrong.

I created KAFKA-2186 and hope we can add this missing method into the Java API 
of old high level consumer (literarily one line change).
Although this method should have been there since KAFKA-1650,  adding this 
method to Java API now is a public API change, just want to see if people think 
we need a KIP for this.

Thanks.

Jiangjie (Becket) Qin

Re: [DISCUSS] Add missing API to old high level consumer

[Joe Stein]

My gut reaction is that this isn't that important for folks otherwise they
would have complained already. If it is a blocker for folks upgrading to
0.8.2.1 then we should do a 0.8.2.2 release with this fix in it. For 0.9.0.
we are pushing for folks to start using the new consumer and that is the
upgrade path we should continue on, imho. If we are going to phase out the
scala clients then we need to strive to not be making changes to them on
trunk.

~ Joe Stein
- - - - - - - - - - - - - - - - -

  http://www.stealth.ly
- - - - - - - - - - - - - - - - -

On Wed, May 13, 2015 at 6:01 PM, Jiangjie Qin 
wrote:

> Add the DISCUSS prefix to the email title : )
>
> From: Jiangjie Qin mailto:j...@linkedin.com>>
> Date: Tuesday, May 12, 2015 at 4:51 PM
> To: "dev@kafka.apache.org" <
> dev@kafka.apache.org>
> Subject: Add missing API to old high level consumer
>
> Hi,
>
> I just noticed that in KAFKA-1650 (which is before we use KIP) we added an
> offset commit method in high level consumer that commits offsets using a
> user provided offset map.
>
> public void commitOffsets(Map
> offsetsToCommit, boolean retryOnFailure);
>
> This method was added to all the Scala classes but I forgot to add it to
> Java API of ConsumerConnector. (Already regretting now. . .)
> This method is very useful in several cases and has been asked for from
> time to time. For example, people have several threads consuming messages
> and processing them. Without this method, one thread will unexpectedly
> commit offsets for another thread, thus might lose some messages if
> something goes wrong.
>
> I created KAFKA-2186 and hope we can add this missing method into the Java
> API of old high level consumer (literarily one line change).
> Although this method should have been there since KAFKA-1650,  adding this
> method to Java API now is a public API change, just want to see if people
> think we need a KIP for this.
>
> Thanks.
>
> Jiangjie (Becket) Qin
>

Re: [DISCUSS] Add missing API to old high level consumer

[Jiangjie Qin]

Hey Joe,

Actually this API was asked for before, and we have several use cases in
LinkedIn as well. I thought we have added that in KAFKA-1650 but obviously
I forgot to do that.

My understanding is that we won¹t really deprecate high level consumer
until we move to 0.9.0. So we can have this API either in 0.8.3 or
0.8.2.2. Do you mean we only add them to those releases but not put it
into trunk? Any specific concern on that?

Considering this API has already been provided in new consumer. Adding
this method probably won¹t cause any API compatibility issue even if
people move to new consumer later.
Given it is both backward and forward compatible and is a one line change,
I think it is probably OK to have it added.

Thanks,

Jiangjie (Becket) Qin



On 5/13/15, 3:18 PM, "Joe Stein"  wrote:

>My gut reaction is that this isn't that important for folks otherwise they
>would have complained already. If it is a blocker for folks upgrading to
>0.8.2.1 then we should do a 0.8.2.2 release with this fix in it. For
>0.9.0.
>we are pushing for folks to start using the new consumer and that is the
>upgrade path we should continue on, imho. If we are going to phase out the
>scala clients then we need to strive to not be making changes to them on
>trunk.
>
>~ Joe Stein
>- - - - - - - - - - - - - - - - -
>
>  http://www.stealth.ly
>- - - - - - - - - - - - - - - - -
>
>On Wed, May 13, 2015 at 6:01 PM, Jiangjie Qin 
>wrote:
>
>> Add the DISCUSS prefix to the email title : )
>>
>> From: Jiangjie Qin mailto:j...@linkedin.com>>
>> Date: Tuesday, May 12, 2015 at 4:51 PM
>> To: "dev@kafka.apache.org" <
>> dev@kafka.apache.org>
>> Subject: Add missing API to old high level consumer
>>
>> Hi,
>>
>> I just noticed that in KAFKA-1650 (which is before we use KIP) we added
>>an
>> offset commit method in high level consumer that commits offsets using a
>> user provided offset map.
>>
>> public void commitOffsets(Map
>> offsetsToCommit, boolean retryOnFailure);
>>
>> This method was added to all the Scala classes but I forgot to add it to
>> Java API of ConsumerConnector. (Already regretting now. . .)
>> This method is very useful in several cases and has been asked for from
>> time to time. For example, people have several threads consuming
>>messages
>> and processing them. Without this method, one thread will unexpectedly
>> commit offsets for another thread, thus might lose some messages if
>> something goes wrong.
>>
>> I created KAFKA-2186 and hope we can add this missing method into the
>>Java
>> API of old high level consumer (literarily one line change).
>> Although this method should have been there since KAFKA-1650,  adding
>>this
>> method to Java API now is a public API change, just want to see if
>>people
>> think we need a KIP for this.
>>
>> Thanks.
>>
>> Jiangjie (Becket) Qin
>>

Re: [DISCUSS] Add missing API to old high level consumer

[Joe Stein]

Hey Becket, yeah good point. Officially there is no 0.8.3
https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan release
planned.

I agree we should have the new consumer beta and a patch for the old one.
If we do that in 0.8.3 that makes good sense, yup. We should also include
https://issues.apache.org/jira/browse/KAFKA-1694 server side admin in 0.8.3
too. We are testing that next week in a few languages to get it through
testing first before committing.

Maybe we branch 0.8.3 in the near future so that can go through getting
released while the security changes for 0.9 get on trunk.

~ Joe Stein

On Thu, May 14, 2015 at 8:00 PM, Jiangjie Qin 
wrote:

> Hey Joe,
>
> Actually this API was asked for before, and we have several use cases in
> LinkedIn as well. I thought we have added that in KAFKA-1650 but obviously
> I forgot to do that.
>
> My understanding is that we won¹t really deprecate high level consumer
> until we move to 0.9.0. So we can have this API either in 0.8.3 or
> 0.8.2.2. Do you mean we only add them to those releases but not put it
> into trunk? Any specific concern on that?
>
> Considering this API has already been provided in new consumer. Adding
> this method probably won¹t cause any API compatibility issue even if
> people move to new consumer later.
> Given it is both backward and forward compatible and is a one line change,
> I think it is probably OK to have it added.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
>
> On 5/13/15, 3:18 PM, "Joe Stein"  wrote:
>
> >My gut reaction is that this isn't that important for folks otherwise they
> >would have complained already. If it is a blocker for folks upgrading to
> >0.8.2.1 then we should do a 0.8.2.2 release with this fix in it. For
> >0.9.0.
> >we are pushing for folks to start using the new consumer and that is the
> >upgrade path we should continue on, imho. If we are going to phase out the
> >scala clients then we need to strive to not be making changes to them on
> >trunk.
> >
> >~ Joe Stein
> >- - - - - - - - - - - - - - - - -
> >
> >  http://www.stealth.ly
> >- - - - - - - - - - - - - - - - -
> >
> >On Wed, May 13, 2015 at 6:01 PM, Jiangjie Qin 
> >wrote:
> >
> >> Add the DISCUSS prefix to the email title : )
> >>
> >> From: Jiangjie Qin mailto:j...@linkedin.com>>
> >> Date: Tuesday, May 12, 2015 at 4:51 PM
> >> To: "dev@kafka.apache.org" <
> >> dev@kafka.apache.org>
> >> Subject: Add missing API to old high level consumer
> >>
> >> Hi,
> >>
> >> I just noticed that in KAFKA-1650 (which is before we use KIP) we added
> >>an
> >> offset commit method in high level consumer that commits offsets using a
> >> user provided offset map.
> >>
> >> public void commitOffsets(Map
> >> offsetsToCommit, boolean retryOnFailure);
> >>
> >> This method was added to all the Scala classes but I forgot to add it to
> >> Java API of ConsumerConnector. (Already regretting now. . .)
> >> This method is very useful in several cases and has been asked for from
> >> time to time. For example, people have several threads consuming
> >>messages
> >> and processing them. Without this method, one thread will unexpectedly
> >> commit offsets for another thread, thus might lose some messages if
> >> something goes wrong.
> >>
> >> I created KAFKA-2186 and hope we can add this missing method into the
> >>Java
> >> API of old high level consumer (literarily one line change).
> >> Although this method should have been there since KAFKA-1650,  adding
> >>this
> >> method to Java API now is a public API change, just want to see if
> >>people
> >> think we need a KIP for this.
> >>
> >> Thanks.
> >>
> >> Jiangjie (Becket) Qin
> >>
>
>

Re: [DISCUSS] Add missing API to old high level consumer

[Gwen Shapira]

I thought we wanted security on 0.8.3 too... the SSL + Authz patches seem
close to ready, no?

On Fri, May 15, 2015 at 3:56 AM, Joe Stein  wrote:

> Hey Becket, yeah good point. Officially there is no 0.8.3
> https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan
> release
> planned.
>
> I agree we should have the new consumer beta and a patch for the old one.
> If we do that in 0.8.3 that makes good sense, yup. We should also include
> https://issues.apache.org/jira/browse/KAFKA-1694 server side admin in
> 0.8.3
> too. We are testing that next week in a few languages to get it through
> testing first before committing.
>
> Maybe we branch 0.8.3 in the near future so that can go through getting
> released while the security changes for 0.9 get on trunk.
>
> ~ Joe Stein
>
> On Thu, May 14, 2015 at 8:00 PM, Jiangjie Qin 
> wrote:
>
> > Hey Joe,
> >
> > Actually this API was asked for before, and we have several use cases in
> > LinkedIn as well. I thought we have added that in KAFKA-1650 but
> obviously
> > I forgot to do that.
> >
> > My understanding is that we won¹t really deprecate high level consumer
> > until we move to 0.9.0. So we can have this API either in 0.8.3 or
> > 0.8.2.2. Do you mean we only add them to those releases but not put it
> > into trunk? Any specific concern on that?
> >
> > Considering this API has already been provided in new consumer. Adding
> > this method probably won¹t cause any API compatibility issue even if
> > people move to new consumer later.
> > Given it is both backward and forward compatible and is a one line
> change,
> > I think it is probably OK to have it added.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> >
> >
> > On 5/13/15, 3:18 PM, "Joe Stein"  wrote:
> >
> > >My gut reaction is that this isn't that important for folks otherwise
> they
> > >would have complained already. If it is a blocker for folks upgrading to
> > >0.8.2.1 then we should do a 0.8.2.2 release with this fix in it. For
> > >0.9.0.
> > >we are pushing for folks to start using the new consumer and that is the
> > >upgrade path we should continue on, imho. If we are going to phase out
> the
> > >scala clients then we need to strive to not be making changes to them on
> > >trunk.
> > >
> > >~ Joe Stein
> > >- - - - - - - - - - - - - - - - -
> > >
> > >  http://www.stealth.ly
> > >- - - - - - - - - - - - - - - - -
> > >
> > >On Wed, May 13, 2015 at 6:01 PM, Jiangjie Qin  >
> > >wrote:
> > >
> > >> Add the DISCUSS prefix to the email title : )
> > >>
> > >> From: Jiangjie Qin mailto:j...@linkedin.com>>
> > >> Date: Tuesday, May 12, 2015 at 4:51 PM
> > >> To: "dev@kafka.apache.org" <
> > >> dev@kafka.apache.org>
> > >> Subject: Add missing API to old high level consumer
> > >>
> > >> Hi,
> > >>
> > >> I just noticed that in KAFKA-1650 (which is before we use KIP) we
> added
> > >>an
> > >> offset commit method in high level consumer that commits offsets
> using a
> > >> user provided offset map.
> > >>
> > >> public void commitOffsets(Map
> > >> offsetsToCommit, boolean retryOnFailure);
> > >>
> > >> This method was added to all the Scala classes but I forgot to add it
> to
> > >> Java API of ConsumerConnector. (Already regretting now. . .)
> > >> This method is very useful in several cases and has been asked for
> from
> > >> time to time. For example, people have several threads consuming
> > >>messages
> > >> and processing them. Without this method, one thread will unexpectedly
> > >> commit offsets for another thread, thus might lose some messages if
> > >> something goes wrong.
> > >>
> > >> I created KAFKA-2186 and hope we can add this missing method into the
> > >>Java
> > >> API of old high level consumer (literarily one line change).
> > >> Although this method should have been there since KAFKA-1650,  adding
> > >>this
> > >> method to Java API now is a public API change, just want to see if
> > >>people
> > >> think we need a KIP for this.
> > >>
> > >> Thanks.
> > >>
> > >> Jiangjie (Becket) Qin
> > >>
> >
> >
>

Re: [DISCUSS] Add missing API to old high level consumer

[Joe Stein]

I peeked at the Java producer SSL changes, haven't tried it yet though. I
can see about getting a Go version to help testing compatibility done in
the next few weeks.

I still don't understand the Auth pieces, I haven't been able to make the
KIP lately I need to try to attend like every other or something.

I will re-read Auth this weekend. I guess my question still is how do I
test it. With SSL it is configure and it works or it doesn't. Auth is not
so straightforward and more opinionated.

~ Joe Stein
- - - - - - - - - - - - - - - - -

  http://www.stealth.ly
- - - - - - - - - - - - - - - - -

On Fri, May 15, 2015 at 2:27 AM, Gwen Shapira  wrote:

> I thought we wanted security on 0.8.3 too... the SSL + Authz patches seem
> close to ready, no?
>
> On Fri, May 15, 2015 at 3:56 AM, Joe Stein  wrote:
>
> > Hey Becket, yeah good point. Officially there is no 0.8.3
> > https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan
> > release
> > planned.
> >
> > I agree we should have the new consumer beta and a patch for the old one.
> > If we do that in 0.8.3 that makes good sense, yup. We should also include
> > https://issues.apache.org/jira/browse/KAFKA-1694 server side admin in
> > 0.8.3
> > too. We are testing that next week in a few languages to get it through
> > testing first before committing.
> >
> > Maybe we branch 0.8.3 in the near future so that can go through getting
> > released while the security changes for 0.9 get on trunk.
> >
> > ~ Joe Stein
> >
> > On Thu, May 14, 2015 at 8:00 PM, Jiangjie Qin  >
> > wrote:
> >
> > > Hey Joe,
> > >
> > > Actually this API was asked for before, and we have several use cases
> in
> > > LinkedIn as well. I thought we have added that in KAFKA-1650 but
> > obviously
> > > I forgot to do that.
> > >
> > > My understanding is that we won¹t really deprecate high level consumer
> > > until we move to 0.9.0. So we can have this API either in 0.8.3 or
> > > 0.8.2.2. Do you mean we only add them to those releases but not put it
> > > into trunk? Any specific concern on that?
> > >
> > > Considering this API has already been provided in new consumer. Adding
> > > this method probably won¹t cause any API compatibility issue even if
> > > people move to new consumer later.
> > > Given it is both backward and forward compatible and is a one line
> > change,
> > > I think it is probably OK to have it added.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > >
> > >
> > > On 5/13/15, 3:18 PM, "Joe Stein"  wrote:
> > >
> > > >My gut reaction is that this isn't that important for folks otherwise
> > they
> > > >would have complained already. If it is a blocker for folks upgrading
> to
> > > >0.8.2.1 then we should do a 0.8.2.2 release with this fix in it. For
> > > >0.9.0.
> > > >we are pushing for folks to start using the new consumer and that is
> the
> > > >upgrade path we should continue on, imho. If we are going to phase out
> > the
> > > >scala clients then we need to strive to not be making changes to them
> on
> > > >trunk.
> > > >
> > > >~ Joe Stein
> > > >- - - - - - - - - - - - - - - - -
> > > >
> > > >  http://www.stealth.ly
> > > >- - - - - - - - - - - - - - - - -
> > > >
> > > >On Wed, May 13, 2015 at 6:01 PM, Jiangjie Qin
>  > >
> > > >wrote:
> > > >
> > > >> Add the DISCUSS prefix to the email title : )
> > > >>
> > > >> From: Jiangjie Qin mailto:j...@linkedin.com>>
> > > >> Date: Tuesday, May 12, 2015 at 4:51 PM
> > > >> To: "dev@kafka.apache.org" <
> > > >> dev@kafka.apache.org>
> > > >> Subject: Add missing API to old high level consumer
> > > >>
> > > >> Hi,
> > > >>
> > > >> I just noticed that in KAFKA-1650 (which is before we use KIP) we
> > added
> > > >>an
> > > >> offset commit method in high level consumer that commits offsets
> > using a
> > > >> user provided offset map.
> > > >>
> > > >> public void commitOffsets(Map
> > > >> offsetsToCommit, boolean retryOnFailure);
> > > >>
> > > >> This method was added to all the Scala classes but I forgot to add
> it
> > to
> > > >> Java API of ConsumerConnector. (Already regretting now. . .)
> > > >> This method is very useful in several cases and has been asked for
> > from
> > > >> time to time. For example, people have several threads consuming
> > > >>messages
> > > >> and processing them. Without this method, one thread will
> unexpectedly
> > > >> commit offsets for another thread, thus might lose some messages if
> > > >> something goes wrong.
> > > >>
> > > >> I created KAFKA-2186 and hope we can add this missing method into
> the
> > > >>Java
> > > >> API of old high level consumer (literarily one line change).
> > > >> Although this method should have been there since KAFKA-1650,
> adding
> > > >>this
> > > >> method to Java API now is a public API change, just want to see if
> > > >>people
> > > >> think we need a KIP for this.
> > > >>
> > > >> Thanks.
> > > >>
> > > >> Jiangjie (Becket) Qin
> > > >>
> > >
> > >
> 

Re: [DISCUSS] Add missing API to old high level consumer

[Gwen Shapira]

Makes sense, thanks!

When we added Authz to Sqoop, we tested with "mock" authorizers that would
either allow everything or deny everything and would log every call.
Perhaps this will make sense for Kafka too. Parth has the patch up, so we
can see how he tested it.

Gwen

On Fri, May 15, 2015 at 10:02 AM, Joe Stein  wrote:

> I peeked at the Java producer SSL changes, haven't tried it yet though. I
> can see about getting a Go version to help testing compatibility done in
> the next few weeks.
>
> I still don't understand the Auth pieces, I haven't been able to make the
> KIP lately I need to try to attend like every other or something.
>
> I will re-read Auth this weekend. I guess my question still is how do I
> test it. With SSL it is configure and it works or it doesn't. Auth is not
> so straightforward and more opinionated.
>
> ~ Joe Stein
> - - - - - - - - - - - - - - - - -
>
>   http://www.stealth.ly
> - - - - - - - - - - - - - - - - -
>
> On Fri, May 15, 2015 at 2:27 AM, Gwen Shapira 
> wrote:
>
> > I thought we wanted security on 0.8.3 too... the SSL + Authz patches seem
> > close to ready, no?
> >
> > On Fri, May 15, 2015 at 3:56 AM, Joe Stein  wrote:
> >
> > > Hey Becket, yeah good point. Officially there is no 0.8.3
> > > https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan
> > > release
> > > planned.
> > >
> > > I agree we should have the new consumer beta and a patch for the old
> one.
> > > If we do that in 0.8.3 that makes good sense, yup. We should also
> include
> > > https://issues.apache.org/jira/browse/KAFKA-1694 server side admin in
> > > 0.8.3
> > > too. We are testing that next week in a few languages to get it through
> > > testing first before committing.
> > >
> > > Maybe we branch 0.8.3 in the near future so that can go through getting
> > > released while the security changes for 0.9 get on trunk.
> > >
> > > ~ Joe Stein
> > >
> > > On Thu, May 14, 2015 at 8:00 PM, Jiangjie Qin
>  > >
> > > wrote:
> > >
> > > > Hey Joe,
> > > >
> > > > Actually this API was asked for before, and we have several use cases
> > in
> > > > LinkedIn as well. I thought we have added that in KAFKA-1650 but
> > > obviously
> > > > I forgot to do that.
> > > >
> > > > My understanding is that we won¹t really deprecate high level
> consumer
> > > > until we move to 0.9.0. So we can have this API either in 0.8.3 or
> > > > 0.8.2.2. Do you mean we only add them to those releases but not put
> it
> > > > into trunk? Any specific concern on that?
> > > >
> > > > Considering this API has already been provided in new consumer.
> Adding
> > > > this method probably won¹t cause any API compatibility issue even if
> > > > people move to new consumer later.
> > > > Given it is both backward and forward compatible and is a one line
> > > change,
> > > > I think it is probably OK to have it added.
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > >
> > > >
> > > > On 5/13/15, 3:18 PM, "Joe Stein"  wrote:
> > > >
> > > > >My gut reaction is that this isn't that important for folks
> otherwise
> > > they
> > > > >would have complained already. If it is a blocker for folks
> upgrading
> > to
> > > > >0.8.2.1 then we should do a 0.8.2.2 release with this fix in it. For
> > > > >0.9.0.
> > > > >we are pushing for folks to start using the new consumer and that is
> > the
> > > > >upgrade path we should continue on, imho. If we are going to phase
> out
> > > the
> > > > >scala clients then we need to strive to not be making changes to
> them
> > on
> > > > >trunk.
> > > > >
> > > > >~ Joe Stein
> > > > >- - - - - - - - - - - - - - - - -
> > > > >
> > > > >  http://www.stealth.ly
> > > > >- - - - - - - - - - - - - - - - -
> > > > >
> > > > >On Wed, May 13, 2015 at 6:01 PM, Jiangjie Qin
> >  > > >
> > > > >wrote:
> > > > >
> > > > >> Add the DISCUSS prefix to the email title : )
> > > > >>
> > > > >> From: Jiangjie Qin mailto:j...@linkedin.com>>
> > > > >> Date: Tuesday, May 12, 2015 at 4:51 PM
> > > > >> To: "dev@kafka.apache.org" <
> > > > >> dev@kafka.apache.org>
> > > > >> Subject: Add missing API to old high level consumer
> > > > >>
> > > > >> Hi,
> > > > >>
> > > > >> I just noticed that in KAFKA-1650 (which is before we use KIP) we
> > > added
> > > > >>an
> > > > >> offset commit method in high level consumer that commits offsets
> > > using a
> > > > >> user provided offset map.
> > > > >>
> > > > >> public void commitOffsets(Map
> > > > >> offsetsToCommit, boolean retryOnFailure);
> > > > >>
> > > > >> This method was added to all the Scala classes but I forgot to add
> > it
> > > to
> > > > >> Java API of ConsumerConnector. (Already regretting now. . .)
> > > > >> This method is very useful in several cases and has been asked for
> > > from
> > > > >> time to time. For example, people have several threads consuming
> > > > >>messages
> > > > >> and processing them. Without this method, one thread will
> > unexpect

Re: [DISCUSS] Add missing API to old high level consumer

[Jun Rao]

Hi, Jiangjie,

It seems that api is already in
kafka.javaapi.consumer.ZookeeperConsumerConnector. We just need to add it
to kafka.javaapi.consumer.ConsumerConnector. This is fine and I don't think
we need a KIP. Since this is only added in trunk, it will be part of the
0.8.3 release. I agree with Joe that since we are developing the new java
consumer, it would be good not to make changes to the old scala consumer.

As for the the 0.8.3 release, we agreed that it will include at least the
new java consumer. If other features (e.g., security, admin, etc) can be
done at or before that, they will be part of the 0.8.3 release too. Updated
the release plan wiki.

Thanks,

Jun


On Wed, May 13, 2015 at 3:01 PM, Jiangjie Qin 
wrote:

> Add the DISCUSS prefix to the email title : )
>
> From: Jiangjie Qin mailto:j...@linkedin.com>>
> Date: Tuesday, May 12, 2015 at 4:51 PM
> To: "dev@kafka.apache.org" <
> dev@kafka.apache.org>
> Subject: Add missing API to old high level consumer
>
> Hi,
>
> I just noticed that in KAFKA-1650 (which is before we use KIP) we added an
> offset commit method in high level consumer that commits offsets using a
> user provided offset map.
>
> public void commitOffsets(Map
> offsetsToCommit, boolean retryOnFailure);
>
> This method was added to all the Scala classes but I forgot to add it to
> Java API of ConsumerConnector. (Already regretting now. . .)
> This method is very useful in several cases and has been asked for from
> time to time. For example, people have several threads consuming messages
> and processing them. Without this method, one thread will unexpectedly
> commit offsets for another thread, thus might lose some messages if
> something goes wrong.
>
> I created KAFKA-2186 and hope we can add this missing method into the Java
> API of old high level consumer (literarily one line change).
> Although this method should have been there since KAFKA-1650,  adding this
> method to Java API now is a public API change, just want to see if people
> think we need a KIP for this.
>
> Thanks.
>
> Jiangjie (Becket) Qin
>

Re: [DISCUSS] Add missing API to old high level consumer

[Jiangjie Qin]

Sounds good. Thanks, Jun.

Jiangjie (Becket) Qin

On 5/15/15, 5:21 PM, "Jun Rao"  wrote:

>Hi, Jiangjie,
>
>It seems that api is already in
>kafka.javaapi.consumer.ZookeeperConsumerConnector. We just need to add it
>to kafka.javaapi.consumer.ConsumerConnector. This is fine and I don't
>think
>we need a KIP. Since this is only added in trunk, it will be part of the
>0.8.3 release. I agree with Joe that since we are developing the new java
>consumer, it would be good not to make changes to the old scala consumer.
>
>As for the the 0.8.3 release, we agreed that it will include at least the
>new java consumer. If other features (e.g., security, admin, etc) can be
>done at or before that, they will be part of the 0.8.3 release too.
>Updated
>the release plan wiki.
>
>Thanks,
>
>Jun
>
>
>On Wed, May 13, 2015 at 3:01 PM, Jiangjie Qin 
>wrote:
>
>> Add the DISCUSS prefix to the email title : )
>>
>> From: Jiangjie Qin mailto:j...@linkedin.com>>
>> Date: Tuesday, May 12, 2015 at 4:51 PM
>> To: "dev@kafka.apache.org" <
>> dev@kafka.apache.org>
>> Subject: Add missing API to old high level consumer
>>
>> Hi,
>>
>> I just noticed that in KAFKA-1650 (which is before we use KIP) we added
>>an
>> offset commit method in high level consumer that commits offsets using a
>> user provided offset map.
>>
>> public void commitOffsets(Map
>> offsetsToCommit, boolean retryOnFailure);
>>
>> This method was added to all the Scala classes but I forgot to add it to
>> Java API of ConsumerConnector. (Already regretting now. . .)
>> This method is very useful in several cases and has been asked for from
>> time to time. For example, people have several threads consuming
>>messages
>> and processing them. Without this method, one thread will unexpectedly
>> commit offsets for another thread, thus might lose some messages if
>> something goes wrong.
>>
>> I created KAFKA-2186 and hope we can add this missing method into the
>>Java
>> API of old high level consumer (literarily one line change).
>> Although this method should have been there since KAFKA-1650,  adding
>>this
>> method to Java API now is a public API change, just want to see if
>>people
>> think we need a KIP for this.
>>
>> Thanks.
>>
>> Jiangjie (Becket) Qin
>>