Re: [DISCUSS] Add missing API to old high level consumer
I thought we wanted security on 0.8.3 too... the SSL + Authz patches seem close to ready, no? On Fri, May 15, 2015 at 3:56 AM, Joe Stein joe.st...@stealth.ly wrote: Hey Becket, yeah good point. Officially there is no 0.8.3 https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan release planned. I agree we should have the new consumer beta and a patch for the old one. If we do that in 0.8.3 that makes good sense, yup. We should also include https://issues.apache.org/jira/browse/KAFKA-1694 server side admin in 0.8.3 too. We are testing that next week in a few languages to get it through testing first before committing. Maybe we branch 0.8.3 in the near future so that can go through getting released while the security changes for 0.9 get on trunk. ~ Joe Stein On Thu, May 14, 2015 at 8:00 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: Hey Joe, Actually this API was asked for before, and we have several use cases in LinkedIn as well. I thought we have added that in KAFKA-1650 but obviously I forgot to do that. My understanding is that we won¹t really deprecate high level consumer until we move to 0.9.0. So we can have this API either in 0.8.3 or 0.8.2.2. Do you mean we only add them to those releases but not put it into trunk? Any specific concern on that? Considering this API has already been provided in new consumer. Adding this method probably won¹t cause any API compatibility issue even if people move to new consumer later. Given it is both backward and forward compatible and is a one line change, I think it is probably OK to have it added. Thanks, Jiangjie (Becket) Qin On 5/13/15, 3:18 PM, Joe Stein joe.st...@stealth.ly wrote: My gut reaction is that this isn't that important for folks otherwise they would have complained already. If it is a blocker for folks upgrading to 0.8.2.1 then we should do a 0.8.2.2 release with this fix in it. For 0.9.0. we are pushing for folks to start using the new consumer and that is the upgrade path we should continue on, imho. If we are going to phase out the scala clients then we need to strive to not be making changes to them on trunk. ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Wed, May 13, 2015 at 6:01 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: Add the DISCUSS prefix to the email title : ) From: Jiangjie Qin j...@linkedin.commailto:j...@linkedin.com Date: Tuesday, May 12, 2015 at 4:51 PM To: dev@kafka.apache.orgmailto:dev@kafka.apache.org dev@kafka.apache.orgmailto:dev@kafka.apache.org Subject: Add missing API to old high level consumer Hi, I just noticed that in KAFKA-1650 (which is before we use KIP) we added an offset commit method in high level consumer that commits offsets using a user provided offset map. public void commitOffsets(MapTopicPartition, OffsetAndMetadata offsetsToCommit, boolean retryOnFailure); This method was added to all the Scala classes but I forgot to add it to Java API of ConsumerConnector. (Already regretting now. . .) This method is very useful in several cases and has been asked for from time to time. For example, people have several threads consuming messages and processing them. Without this method, one thread will unexpectedly commit offsets for another thread, thus might lose some messages if something goes wrong. I created KAFKA-2186 and hope we can add this missing method into the Java API of old high level consumer (literarily one line change). Although this method should have been there since KAFKA-1650, adding this method to Java API now is a public API change, just want to see if people think we need a KIP for this. Thanks. Jiangjie (Becket) Qin
Re: [DISCUSS] Add missing API to old high level consumer
I peeked at the Java producer SSL changes, haven't tried it yet though. I can see about getting a Go version to help testing compatibility done in the next few weeks. I still don't understand the Auth pieces, I haven't been able to make the KIP lately I need to try to attend like every other or something. I will re-read Auth this weekend. I guess my question still is how do I test it. With SSL it is configure and it works or it doesn't. Auth is not so straightforward and more opinionated. ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Fri, May 15, 2015 at 2:27 AM, Gwen Shapira gshap...@cloudera.com wrote: I thought we wanted security on 0.8.3 too... the SSL + Authz patches seem close to ready, no? On Fri, May 15, 2015 at 3:56 AM, Joe Stein joe.st...@stealth.ly wrote: Hey Becket, yeah good point. Officially there is no 0.8.3 https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan release planned. I agree we should have the new consumer beta and a patch for the old one. If we do that in 0.8.3 that makes good sense, yup. We should also include https://issues.apache.org/jira/browse/KAFKA-1694 server side admin in 0.8.3 too. We are testing that next week in a few languages to get it through testing first before committing. Maybe we branch 0.8.3 in the near future so that can go through getting released while the security changes for 0.9 get on trunk. ~ Joe Stein On Thu, May 14, 2015 at 8:00 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: Hey Joe, Actually this API was asked for before, and we have several use cases in LinkedIn as well. I thought we have added that in KAFKA-1650 but obviously I forgot to do that. My understanding is that we won¹t really deprecate high level consumer until we move to 0.9.0. So we can have this API either in 0.8.3 or 0.8.2.2. Do you mean we only add them to those releases but not put it into trunk? Any specific concern on that? Considering this API has already been provided in new consumer. Adding this method probably won¹t cause any API compatibility issue even if people move to new consumer later. Given it is both backward and forward compatible and is a one line change, I think it is probably OK to have it added. Thanks, Jiangjie (Becket) Qin On 5/13/15, 3:18 PM, Joe Stein joe.st...@stealth.ly wrote: My gut reaction is that this isn't that important for folks otherwise they would have complained already. If it is a blocker for folks upgrading to 0.8.2.1 then we should do a 0.8.2.2 release with this fix in it. For 0.9.0. we are pushing for folks to start using the new consumer and that is the upgrade path we should continue on, imho. If we are going to phase out the scala clients then we need to strive to not be making changes to them on trunk. ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Wed, May 13, 2015 at 6:01 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: Add the DISCUSS prefix to the email title : ) From: Jiangjie Qin j...@linkedin.commailto:j...@linkedin.com Date: Tuesday, May 12, 2015 at 4:51 PM To: dev@kafka.apache.orgmailto:dev@kafka.apache.org dev@kafka.apache.orgmailto:dev@kafka.apache.org Subject: Add missing API to old high level consumer Hi, I just noticed that in KAFKA-1650 (which is before we use KIP) we added an offset commit method in high level consumer that commits offsets using a user provided offset map. public void commitOffsets(MapTopicPartition, OffsetAndMetadata offsetsToCommit, boolean retryOnFailure); This method was added to all the Scala classes but I forgot to add it to Java API of ConsumerConnector. (Already regretting now. . .) This method is very useful in several cases and has been asked for from time to time. For example, people have several threads consuming messages and processing them. Without this method, one thread will unexpectedly commit offsets for another thread, thus might lose some messages if something goes wrong. I created KAFKA-2186 and hope we can add this missing method into the Java API of old high level consumer (literarily one line change). Although this method should have been there since KAFKA-1650, adding this method to Java API now is a public API change, just want to see if people think we need a KIP for this. Thanks. Jiangjie (Becket) Qin
Re: [DISCUSS] Add missing API to old high level consumer
Makes sense, thanks! When we added Authz to Sqoop, we tested with mock authorizers that would either allow everything or deny everything and would log every call. Perhaps this will make sense for Kafka too. Parth has the patch up, so we can see how he tested it. Gwen On Fri, May 15, 2015 at 10:02 AM, Joe Stein joe.st...@stealth.ly wrote: I peeked at the Java producer SSL changes, haven't tried it yet though. I can see about getting a Go version to help testing compatibility done in the next few weeks. I still don't understand the Auth pieces, I haven't been able to make the KIP lately I need to try to attend like every other or something. I will re-read Auth this weekend. I guess my question still is how do I test it. With SSL it is configure and it works or it doesn't. Auth is not so straightforward and more opinionated. ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Fri, May 15, 2015 at 2:27 AM, Gwen Shapira gshap...@cloudera.com wrote: I thought we wanted security on 0.8.3 too... the SSL + Authz patches seem close to ready, no? On Fri, May 15, 2015 at 3:56 AM, Joe Stein joe.st...@stealth.ly wrote: Hey Becket, yeah good point. Officially there is no 0.8.3 https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan release planned. I agree we should have the new consumer beta and a patch for the old one. If we do that in 0.8.3 that makes good sense, yup. We should also include https://issues.apache.org/jira/browse/KAFKA-1694 server side admin in 0.8.3 too. We are testing that next week in a few languages to get it through testing first before committing. Maybe we branch 0.8.3 in the near future so that can go through getting released while the security changes for 0.9 get on trunk. ~ Joe Stein On Thu, May 14, 2015 at 8:00 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: Hey Joe, Actually this API was asked for before, and we have several use cases in LinkedIn as well. I thought we have added that in KAFKA-1650 but obviously I forgot to do that. My understanding is that we won¹t really deprecate high level consumer until we move to 0.9.0. So we can have this API either in 0.8.3 or 0.8.2.2. Do you mean we only add them to those releases but not put it into trunk? Any specific concern on that? Considering this API has already been provided in new consumer. Adding this method probably won¹t cause any API compatibility issue even if people move to new consumer later. Given it is both backward and forward compatible and is a one line change, I think it is probably OK to have it added. Thanks, Jiangjie (Becket) Qin On 5/13/15, 3:18 PM, Joe Stein joe.st...@stealth.ly wrote: My gut reaction is that this isn't that important for folks otherwise they would have complained already. If it is a blocker for folks upgrading to 0.8.2.1 then we should do a 0.8.2.2 release with this fix in it. For 0.9.0. we are pushing for folks to start using the new consumer and that is the upgrade path we should continue on, imho. If we are going to phase out the scala clients then we need to strive to not be making changes to them on trunk. ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Wed, May 13, 2015 at 6:01 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: Add the DISCUSS prefix to the email title : ) From: Jiangjie Qin j...@linkedin.commailto:j...@linkedin.com Date: Tuesday, May 12, 2015 at 4:51 PM To: dev@kafka.apache.orgmailto:dev@kafka.apache.org dev@kafka.apache.orgmailto:dev@kafka.apache.org Subject: Add missing API to old high level consumer Hi, I just noticed that in KAFKA-1650 (which is before we use KIP) we added an offset commit method in high level consumer that commits offsets using a user provided offset map. public void commitOffsets(MapTopicPartition, OffsetAndMetadata offsetsToCommit, boolean retryOnFailure); This method was added to all the Scala classes but I forgot to add it to Java API of ConsumerConnector. (Already regretting now. . .) This method is very useful in several cases and has been asked for from time to time. For example, people have several threads consuming messages and processing them. Without this method, one thread will unexpectedly commit offsets for another thread, thus might lose some messages if something goes wrong. I created KAFKA-2186 and hope we can add this missing method into the Java API of old high level consumer (literarily one line change). Although this method should have been
Re: [DISCUSS] Add missing API to old high level consumer
Hi, Jiangjie, It seems that api is already in kafka.javaapi.consumer.ZookeeperConsumerConnector. We just need to add it to kafka.javaapi.consumer.ConsumerConnector. This is fine and I don't think we need a KIP. Since this is only added in trunk, it will be part of the 0.8.3 release. I agree with Joe that since we are developing the new java consumer, it would be good not to make changes to the old scala consumer. As for the the 0.8.3 release, we agreed that it will include at least the new java consumer. If other features (e.g., security, admin, etc) can be done at or before that, they will be part of the 0.8.3 release too. Updated the release plan wiki. Thanks, Jun On Wed, May 13, 2015 at 3:01 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: Add the DISCUSS prefix to the email title : ) From: Jiangjie Qin j...@linkedin.commailto:j...@linkedin.com Date: Tuesday, May 12, 2015 at 4:51 PM To: dev@kafka.apache.orgmailto:dev@kafka.apache.org dev@kafka.apache.orgmailto:dev@kafka.apache.org Subject: Add missing API to old high level consumer Hi, I just noticed that in KAFKA-1650 (which is before we use KIP) we added an offset commit method in high level consumer that commits offsets using a user provided offset map. public void commitOffsets(MapTopicPartition, OffsetAndMetadata offsetsToCommit, boolean retryOnFailure); This method was added to all the Scala classes but I forgot to add it to Java API of ConsumerConnector. (Already regretting now. . .) This method is very useful in several cases and has been asked for from time to time. For example, people have several threads consuming messages and processing them. Without this method, one thread will unexpectedly commit offsets for another thread, thus might lose some messages if something goes wrong. I created KAFKA-2186 and hope we can add this missing method into the Java API of old high level consumer (literarily one line change). Although this method should have been there since KAFKA-1650, adding this method to Java API now is a public API change, just want to see if people think we need a KIP for this. Thanks. Jiangjie (Becket) Qin
Re: [DISCUSS] Add missing API to old high level consumer
Sounds good. Thanks, Jun. Jiangjie (Becket) Qin On 5/15/15, 5:21 PM, Jun Rao j...@confluent.io wrote: Hi, Jiangjie, It seems that api is already in kafka.javaapi.consumer.ZookeeperConsumerConnector. We just need to add it to kafka.javaapi.consumer.ConsumerConnector. This is fine and I don't think we need a KIP. Since this is only added in trunk, it will be part of the 0.8.3 release. I agree with Joe that since we are developing the new java consumer, it would be good not to make changes to the old scala consumer. As for the the 0.8.3 release, we agreed that it will include at least the new java consumer. If other features (e.g., security, admin, etc) can be done at or before that, they will be part of the 0.8.3 release too. Updated the release plan wiki. Thanks, Jun On Wed, May 13, 2015 at 3:01 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: Add the DISCUSS prefix to the email title : ) From: Jiangjie Qin j...@linkedin.commailto:j...@linkedin.com Date: Tuesday, May 12, 2015 at 4:51 PM To: dev@kafka.apache.orgmailto:dev@kafka.apache.org dev@kafka.apache.orgmailto:dev@kafka.apache.org Subject: Add missing API to old high level consumer Hi, I just noticed that in KAFKA-1650 (which is before we use KIP) we added an offset commit method in high level consumer that commits offsets using a user provided offset map. public void commitOffsets(MapTopicPartition, OffsetAndMetadata offsetsToCommit, boolean retryOnFailure); This method was added to all the Scala classes but I forgot to add it to Java API of ConsumerConnector. (Already regretting now. . .) This method is very useful in several cases and has been asked for from time to time. For example, people have several threads consuming messages and processing them. Without this method, one thread will unexpectedly commit offsets for another thread, thus might lose some messages if something goes wrong. I created KAFKA-2186 and hope we can add this missing method into the Java API of old high level consumer (literarily one line change). Although this method should have been there since KAFKA-1650, adding this method to Java API now is a public API change, just want to see if people think we need a KIP for this. Thanks. Jiangjie (Becket) Qin
Re: [DISCUSS] Add missing API to old high level consumer
Hey Joe, Actually this API was asked for before, and we have several use cases in LinkedIn as well. I thought we have added that in KAFKA-1650 but obviously I forgot to do that. My understanding is that we won¹t really deprecate high level consumer until we move to 0.9.0. So we can have this API either in 0.8.3 or 0.8.2.2. Do you mean we only add them to those releases but not put it into trunk? Any specific concern on that? Considering this API has already been provided in new consumer. Adding this method probably won¹t cause any API compatibility issue even if people move to new consumer later. Given it is both backward and forward compatible and is a one line change, I think it is probably OK to have it added. Thanks, Jiangjie (Becket) Qin On 5/13/15, 3:18 PM, Joe Stein joe.st...@stealth.ly wrote: My gut reaction is that this isn't that important for folks otherwise they would have complained already. If it is a blocker for folks upgrading to 0.8.2.1 then we should do a 0.8.2.2 release with this fix in it. For 0.9.0. we are pushing for folks to start using the new consumer and that is the upgrade path we should continue on, imho. If we are going to phase out the scala clients then we need to strive to not be making changes to them on trunk. ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Wed, May 13, 2015 at 6:01 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: Add the DISCUSS prefix to the email title : ) From: Jiangjie Qin j...@linkedin.commailto:j...@linkedin.com Date: Tuesday, May 12, 2015 at 4:51 PM To: dev@kafka.apache.orgmailto:dev@kafka.apache.org dev@kafka.apache.orgmailto:dev@kafka.apache.org Subject: Add missing API to old high level consumer Hi, I just noticed that in KAFKA-1650 (which is before we use KIP) we added an offset commit method in high level consumer that commits offsets using a user provided offset map. public void commitOffsets(MapTopicPartition, OffsetAndMetadata offsetsToCommit, boolean retryOnFailure); This method was added to all the Scala classes but I forgot to add it to Java API of ConsumerConnector. (Already regretting now. . .) This method is very useful in several cases and has been asked for from time to time. For example, people have several threads consuming messages and processing them. Without this method, one thread will unexpectedly commit offsets for another thread, thus might lose some messages if something goes wrong. I created KAFKA-2186 and hope we can add this missing method into the Java API of old high level consumer (literarily one line change). Although this method should have been there since KAFKA-1650, adding this method to Java API now is a public API change, just want to see if people think we need a KIP for this. Thanks. Jiangjie (Becket) Qin
Re: [DISCUSS] Add missing API to old high level consumer
Hey Becket, yeah good point. Officially there is no 0.8.3 https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan release planned. I agree we should have the new consumer beta and a patch for the old one. If we do that in 0.8.3 that makes good sense, yup. We should also include https://issues.apache.org/jira/browse/KAFKA-1694 server side admin in 0.8.3 too. We are testing that next week in a few languages to get it through testing first before committing. Maybe we branch 0.8.3 in the near future so that can go through getting released while the security changes for 0.9 get on trunk. ~ Joe Stein On Thu, May 14, 2015 at 8:00 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: Hey Joe, Actually this API was asked for before, and we have several use cases in LinkedIn as well. I thought we have added that in KAFKA-1650 but obviously I forgot to do that. My understanding is that we won¹t really deprecate high level consumer until we move to 0.9.0. So we can have this API either in 0.8.3 or 0.8.2.2. Do you mean we only add them to those releases but not put it into trunk? Any specific concern on that? Considering this API has already been provided in new consumer. Adding this method probably won¹t cause any API compatibility issue even if people move to new consumer later. Given it is both backward and forward compatible and is a one line change, I think it is probably OK to have it added. Thanks, Jiangjie (Becket) Qin On 5/13/15, 3:18 PM, Joe Stein joe.st...@stealth.ly wrote: My gut reaction is that this isn't that important for folks otherwise they would have complained already. If it is a blocker for folks upgrading to 0.8.2.1 then we should do a 0.8.2.2 release with this fix in it. For 0.9.0. we are pushing for folks to start using the new consumer and that is the upgrade path we should continue on, imho. If we are going to phase out the scala clients then we need to strive to not be making changes to them on trunk. ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Wed, May 13, 2015 at 6:01 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: Add the DISCUSS prefix to the email title : ) From: Jiangjie Qin j...@linkedin.commailto:j...@linkedin.com Date: Tuesday, May 12, 2015 at 4:51 PM To: dev@kafka.apache.orgmailto:dev@kafka.apache.org dev@kafka.apache.orgmailto:dev@kafka.apache.org Subject: Add missing API to old high level consumer Hi, I just noticed that in KAFKA-1650 (which is before we use KIP) we added an offset commit method in high level consumer that commits offsets using a user provided offset map. public void commitOffsets(MapTopicPartition, OffsetAndMetadata offsetsToCommit, boolean retryOnFailure); This method was added to all the Scala classes but I forgot to add it to Java API of ConsumerConnector. (Already regretting now. . .) This method is very useful in several cases and has been asked for from time to time. For example, people have several threads consuming messages and processing them. Without this method, one thread will unexpectedly commit offsets for another thread, thus might lose some messages if something goes wrong. I created KAFKA-2186 and hope we can add this missing method into the Java API of old high level consumer (literarily one line change). Although this method should have been there since KAFKA-1650, adding this method to Java API now is a public API change, just want to see if people think we need a KIP for this. Thanks. Jiangjie (Becket) Qin
Re: [DISCUSS] Add missing API to old high level consumer
My gut reaction is that this isn't that important for folks otherwise they would have complained already. If it is a blocker for folks upgrading to 0.8.2.1 then we should do a 0.8.2.2 release with this fix in it. For 0.9.0. we are pushing for folks to start using the new consumer and that is the upgrade path we should continue on, imho. If we are going to phase out the scala clients then we need to strive to not be making changes to them on trunk. ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Wed, May 13, 2015 at 6:01 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: Add the DISCUSS prefix to the email title : ) From: Jiangjie Qin j...@linkedin.commailto:j...@linkedin.com Date: Tuesday, May 12, 2015 at 4:51 PM To: dev@kafka.apache.orgmailto:dev@kafka.apache.org dev@kafka.apache.orgmailto:dev@kafka.apache.org Subject: Add missing API to old high level consumer Hi, I just noticed that in KAFKA-1650 (which is before we use KIP) we added an offset commit method in high level consumer that commits offsets using a user provided offset map. public void commitOffsets(MapTopicPartition, OffsetAndMetadata offsetsToCommit, boolean retryOnFailure); This method was added to all the Scala classes but I forgot to add it to Java API of ConsumerConnector. (Already regretting now. . .) This method is very useful in several cases and has been asked for from time to time. For example, people have several threads consuming messages and processing them. Without this method, one thread will unexpectedly commit offsets for another thread, thus might lose some messages if something goes wrong. I created KAFKA-2186 and hope we can add this missing method into the Java API of old high level consumer (literarily one line change). Although this method should have been there since KAFKA-1650, adding this method to Java API now is a public API change, just want to see if people think we need a KIP for this. Thanks. Jiangjie (Becket) Qin
Add missing API to old high level consumer
Hi, I just noticed that in KAFKA-1650 (which is before we use KIP) we added an offset commit method in high level consumer that commits offsets using a user provided offset map. public void commitOffsets(MapTopicPartition, OffsetAndMetadata offsetsToCommit, boolean retryOnFailure); This method was added to all the Scala classes but I forgot to add it to Java API of ConsumerConnector. (Already regretting now. . .) This method is very useful in several cases and has been asked for from time to time. For example, people have several threads consuming messages and processing them. Without this method, one thread will unexpectedly commit offsets for another thread, thus might lose some messages if something goes wrong. I created KAFKA-2186 and hope we can add this missing method into the Java API of old high level consumer (literarily one line change). Although this method should have been there since KAFKA-1650, adding this method to Java API now is a public API change, just want to see if people think we need a KIP for this. Thanks. Jiangjie (Becket) Qin
