[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tong Li updated KAFKA-1926: --- Attachment: KAFKA-1926_2015-04-05_23:45:13.patch Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.0 Reporter: Jay Kreps Assignee: Tong Li Labels: newbie, patch Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926_2015-04-01_22:16:46.patch, KAFKA-1926_2015-04-05_23:45:13.patch There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14480907#comment-14480907 ] Tong Li commented on KAFKA-1926: [~junrao] Jun, thanks again. Noticed you have done a lot over the weekend. what an inspiration! So here I did a bit. Since the patch 1809 is in, I rebased the patch set and addressed the couple new comments. Here it is in again for your review. Really appreciate all the work you did for the project. Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.0 Reporter: Jay Kreps Assignee: Tong Li Labels: newbie, patch Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926_2015-04-01_22:16:46.patch, KAFKA-1926_2015-04-05_23:45:13.patch There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14480906#comment-14480906 ] Tong Li commented on KAFKA-1926: Updated reviewboard https://reviews.apache.org/r/32740/ against branch origin/trunk Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.0 Reporter: Jay Kreps Assignee: Tong Li Labels: newbie, patch Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926_2015-04-01_22:16:46.patch, KAFKA-1926_2015-04-05_23:45:13.patch There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391999#comment-14391999 ] Tong Li commented on KAFKA-1926: [~junrao]Thanks so much for the quick review. All the new comments were addressed in the updated patch set.. Please review when you have time. Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.0 Reporter: Jay Kreps Assignee: Tong Li Labels: newbie, patch Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926_2015-04-01_22:16:46.patch There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391990#comment-14391990 ] Tong Li commented on KAFKA-1926: Updated reviewboard https://reviews.apache.org/r/32740/ against branch origin/trunk Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.0 Reporter: Jay Kreps Assignee: Tong Li Labels: newbie, patch Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926_2015-04-01_22:16:46.patch There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tong Li updated KAFKA-1926: --- Attachment: KAFKA-1926_2015-04-01_22:16:46.patch Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.0 Reporter: Jay Kreps Assignee: Tong Li Labels: newbie, patch Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926_2015-04-01_22:16:46.patch There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391295#comment-14391295 ] Tong Li commented on KAFKA-1926: [~junrao], all comments have been addressed in the latest patch set other than the readByte movement. The move of readByte from core to client will cause many other changes since the method uses scala.Byte, any thing uses that method will have to have the signature changes. Since this patch set is already huge, I would very much like to do that refactor in a later patch set. Thanks. Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.0 Reporter: Jay Kreps Assignee: Tong Li Labels: newbie, patch Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tong Li updated KAFKA-1926: --- Attachment: KAFKA-1926.patch Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.0 Reporter: Jay Kreps Assignee: Tong Li Labels: newbie, patch Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363164#comment-14363164 ] Tong Li commented on KAFKA-1926: @Jun Rao, awesome comments. I will be following the directions and provide new patch set. This also confirms the direction that I am going. Thanks. Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.0 Reporter: Jay Kreps Labels: newbie, patch Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352488#comment-14352488 ] Tong Li commented on KAFKA-1926: Created reviewboard https://reviews.apache.org/r/31844/ against branch origin/trunk Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.0 Reporter: Jay Kreps Labels: newbie, patch Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tong Li updated KAFKA-1926: --- Attachment: KAFKA-1926.patch Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.0 Reporter: Jay Kreps Labels: newbie, patch Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348038#comment-14348038 ] Tong Li commented on KAFKA-1926: Jay, I finally put together a new patch set just for this issue. (the SystemTime issue we can address in a separate issue since this patch set is already very big). In the new patch set, I will have the following changes made: This patch set is an attempt to refactor core utils class and remove the namespace overlap. Utils class was defined by both clients and core projects. Many methods are duplicate. This patch did the following: 1. Renames the core kafka.utils.Utils class to be kafka.utils.CoreUtils 2. Removed the abs method in kafka.utils.CoreUtils, anywhere using abs method will be replaced with the method in o.a.k.c.u.Utils.abs 3. Removed the Crc32 class in kafka.utils package. Also use the one defined in o.a.k.c.u.Utils class. 4. Removed asString method in CoreUtils since this was not used. 5. Removed ReadUnisnedInt (two methods), WriteUnsignedInt (two methods) since they are completely duplicate of the client methods, both signature and implementation are absolutely identical. I will submit the patch set for you to review. Since this patch set touches so many files, very time consuming. if we can get this in faster, then I won't need to do so many rounds of rebase. Thanks so much. Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.0 Reporter: Jay Kreps Labels: newbie, patch Attachments: KAFKA-1926.patch, KAFKA-1926.patch There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tong Li updated KAFKA-1988: --- Attachment: KAFKA-1988_2015-03-03_19:00:21.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. Key: KAFKA-1988 URL: https://issues.apache.org/jira/browse/KAFKA-1988 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.0 Reporter: Tong Li Assignee: Tong Li Priority: Blocker Fix For: 0.8.2.1 Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988_2015-03-03_19:00:21.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. The method only returns intended value for positive numbers. All negative numbers except the Integer.Min_Value will be returned an unsigned integer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346038#comment-14346038 ] Tong Li commented on KAFKA-1988: Updated reviewboard https://reviews.apache.org/r/31566/diff/ against branch origin/trunk org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. Key: KAFKA-1988 URL: https://issues.apache.org/jira/browse/KAFKA-1988 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.0 Reporter: Tong Li Assignee: Tong Li Priority: Blocker Fix For: 0.8.2.1 Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988_2015-03-03_19:00:21.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. The method only returns intended value for positive numbers. All negative numbers except the Integer.Min_Value will be returned an unsigned integer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tong Li updated KAFKA-1988: --- Attachment: KAFKA-1988_2015-03-03_19:03:31.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. Key: KAFKA-1988 URL: https://issues.apache.org/jira/browse/KAFKA-1988 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.0 Reporter: Tong Li Assignee: Tong Li Priority: Blocker Fix For: 0.8.2.1 Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988_2015-03-03_19:00:21.patch, KAFKA-1988_2015-03-03_19:03:31.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. The method only returns intended value for positive numbers. All negative numbers except the Integer.Min_Value will be returned an unsigned integer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346045#comment-14346045 ] Tong Li commented on KAFKA-1988: Updated reviewboard https://reviews.apache.org/r/31566/diff/ against branch origin/trunk org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. Key: KAFKA-1988 URL: https://issues.apache.org/jira/browse/KAFKA-1988 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.0 Reporter: Tong Li Assignee: Tong Li Priority: Blocker Fix For: 0.8.2.1 Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988_2015-03-03_19:00:21.patch, KAFKA-1988_2015-03-03_19:03:31.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. The method only returns intended value for positive numbers. All negative numbers except the Integer.Min_Value will be returned an unsigned integer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343109#comment-14343109 ] Tong Li commented on KAFKA-1988: [~guozhang] Yes, the o.a.k.c.c.u.Utils.abs used in few places. in patch set for issue 1926, I will consolidate both Utils modules from clients and core into one. So that we do not have name conflict all over the place. The patch set for issue 1926 will be quite big. I would like to get this thing fixed for coming up release first, then we can address issue 1926. Thanks. org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. Key: KAFKA-1988 URL: https://issues.apache.org/jira/browse/KAFKA-1988 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.0 Reporter: Tong Li Assignee: Tong Li Priority: Blocker Fix For: 0.8.2.1 Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. The method only returns intended value for positive numbers. All negative numbers except the Integer.Min_Value will be returned an unsigned integer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tong Li updated KAFKA-1988: --- Attachment: KAFKA-1988.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. Key: KAFKA-1988 URL: https://issues.apache.org/jira/browse/KAFKA-1988 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.0 Reporter: Tong Li Assignee: Tong Li Priority: Blocker Fix For: 0.8.2.1 Attachments: KAFKA-1988.patch, KAFKA-1988.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. The method only returns intended value for positive numbers. All negative numbers except the Integer.Min_Value will be returned an unsigned integer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339921#comment-14339921 ] Tong Li commented on KAFKA-1988: Created reviewboard https://reviews.apache.org/r/31533/diff/ against branch origin/trunk org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. Key: KAFKA-1988 URL: https://issues.apache.org/jira/browse/KAFKA-1988 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.0 Reporter: Tong Li Assignee: Tong Li Priority: Blocker Fix For: 0.8.2.1 Attachments: KAFKA-1988.patch, KAFKA-1988.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. The method only returns intended value for positive numbers. All negative numbers except the Integer.Min_Value will be returned an unsigned integer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.
[
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339924#comment-14339924
]
Tong Li commented on KAFKA-1988:
Jun Rao,
Thanks for taking time reviewing the patch. Would like to confirm here
is what you suggested to do:
1. Utils.abs will be renamed to Utils.toPositive
2. Make changes in these methods of class DefaultPartitioner and
ByteArrayPartitioner so that toPositive gets called in partition method.
def partition(key: Any, numPartitions: Int): Int = {
Utils.abs(key.hashCode) % numPartitions
}
I can certainly make the above changes. however method abs gets used
many other places and making the case worse is that there are two abs methods
both in core and clients. I found this problem when I was in an effort of
addressing issue 1926. For now, what if I keep that method in clients, remove
the one in core Utils module and making changes to all other modules where abs
method gets called so that all use of abs method will point to the clients
Utils.abs instead of two locations?
The patch set reflects the above thoughts is here
https://reviews.apache.org/r/31533/diff/
org.apache.kafka.common.utils.Utils.abs method returns wrong value for
negative numbers.
Key: KAFKA-1988
URL: https://issues.apache.org/jira/browse/KAFKA-1988
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8.2.0
Reporter: Tong Li
Assignee: Tong Li
Priority: Blocker
Fix For: 0.8.2.1
Attachments: KAFKA-1988.patch, KAFKA-1988.patch
org.apache.kafka.common.utils.Utils.abs method returns wrong value for
negative numbers. The method only returns intended value for positive
numbers. All negative numbers except the Integer.Min_Value will be returned
an unsigned integer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340959#comment-14340959 ] Tong Li commented on KAFKA-1988: Run Rao, I got the point now. Thanks so much for your very detailed explanation. A patch is coming in few minutes with only that. org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. Key: KAFKA-1988 URL: https://issues.apache.org/jira/browse/KAFKA-1988 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.0 Reporter: Tong Li Assignee: Tong Li Priority: Blocker Fix For: 0.8.2.1 Attachments: KAFKA-1988.patch, KAFKA-1988.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. The method only returns intended value for positive numbers. All negative numbers except the Integer.Min_Value will be returned an unsigned integer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tong Li updated KAFKA-1988: --- Attachment: KAFKA-1988.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. Key: KAFKA-1988 URL: https://issues.apache.org/jira/browse/KAFKA-1988 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.0 Reporter: Tong Li Assignee: Tong Li Priority: Blocker Fix For: 0.8.2.1 Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. The method only returns intended value for positive numbers. All negative numbers except the Integer.Min_Value will be returned an unsigned integer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341004#comment-14341004 ] Tong Li commented on KAFKA-1988: Created reviewboard https://reviews.apache.org/r/31566/diff/ against branch origin/trunk org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. Key: KAFKA-1988 URL: https://issues.apache.org/jira/browse/KAFKA-1988 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.0 Reporter: Tong Li Assignee: Tong Li Priority: Blocker Fix For: 0.8.2.1 Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. The method only returns intended value for positive numbers. All negative numbers except the Integer.Min_Value will be returned an unsigned integer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tong Li updated KAFKA-1988: --- Status: Patch Available (was: Open) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. Key: KAFKA-1988 URL: https://issues.apache.org/jira/browse/KAFKA-1988 Project: Kafka Issue Type: Bug Reporter: Tong Li Assignee: Tong Li Attachments: KAFKA-1988.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. The method only returns intended value for positive numbers. All negative numbers except the Integer.Min_Value will be returned an unsigned integer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tong Li updated KAFKA-1988: --- Attachment: KAFKA-1988.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. Key: KAFKA-1988 URL: https://issues.apache.org/jira/browse/KAFKA-1988 Project: Kafka Issue Type: Bug Reporter: Tong Li Assignee: Tong Li Attachments: KAFKA-1988.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. The method only returns intended value for positive numbers. All negative numbers except the Integer.Min_Value will be returned an unsigned integer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.
[ https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339478#comment-14339478 ] Tong Li commented on KAFKA-1988: Created reviewboard https://reviews.apache.org/r/31517/diff/ against branch origin/trunk org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. Key: KAFKA-1988 URL: https://issues.apache.org/jira/browse/KAFKA-1988 Project: Kafka Issue Type: Bug Reporter: Tong Li Assignee: Tong Li Attachments: KAFKA-1988.patch org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. The method only returns intended value for positive numbers. All negative numbers except the Integer.Min_Value will be returned an unsigned integer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.
Tong Li created KAFKA-1988: -- Summary: org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. Key: KAFKA-1988 URL: https://issues.apache.org/jira/browse/KAFKA-1988 Project: Kafka Issue Type: Bug Reporter: Tong Li Assignee: Tong Li org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers. The method only returns intended value for positive numbers. All negative numbers except the Integer.Min_Value will be returned an unsigned integer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1959) Class CommitThread overwrite group of Thread class causing compile errors
[
https://issues.apache.org/jira/browse/KAFKA-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14322966#comment-14322966
]
Tong Li commented on KAFKA-1959:
Created reviewboard https://reviews.apache.org/r/31088/diff/
against branch origin/trunk
Class CommitThread overwrite group of Thread class causing compile errors
-
Key: KAFKA-1959
URL: https://issues.apache.org/jira/browse/KAFKA-1959
Project: Kafka
Issue Type: Bug
Components: core
Environment: scala 2.10.4
Reporter: Tong Li
Labels: newbie
Attachments: KAFKA-1959.patch
class CommitThread(id: Int, partitionCount: Int, commitIntervalMs: Long,
zkClient: ZkClient)
extends ShutdownableThread(commit-thread)
with KafkaMetricsGroup {
private val group = group- + id
group overwrite class Thread group member, causing the following compile
error:
overriding variable group in class Thread of type ThreadGroup; value group
has weaker access privileges; it should not be private
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KAFKA-1959) Class CommitThread overwrite group of Thread class causing compile errors
Tong Li created KAFKA-1959:
--
Summary: Class CommitThread overwrite group of Thread class
causing compile errors
Key: KAFKA-1959
URL: https://issues.apache.org/jira/browse/KAFKA-1959
Project: Kafka
Issue Type: Bug
Components: core
Environment: scala 2.10.4
Reporter: Tong Li
class CommitThread(id: Int, partitionCount: Int, commitIntervalMs: Long,
zkClient: ZkClient)
extends ShutdownableThread(commit-thread)
with KafkaMetricsGroup {
private val group = group- + id
group overwrite class Thread group member, causing the following compile error:
overriding variable group in class Thread of type ThreadGroup; value group has
weaker access privileges; it should not be private
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1959) Class CommitThread overwrite group of Thread class causing compile errors
[
https://issues.apache.org/jira/browse/KAFKA-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14322933#comment-14322933
]
Tong Li commented on KAFKA-1959:
Though I saw the error by using scala 2.10.4, I think it will be the same
error in other scala versions.
Class CommitThread overwrite group of Thread class causing compile errors
-
Key: KAFKA-1959
URL: https://issues.apache.org/jira/browse/KAFKA-1959
Project: Kafka
Issue Type: Bug
Components: core
Environment: scala 2.10.4
Reporter: Tong Li
Labels: newbie
class CommitThread(id: Int, partitionCount: Int, commitIntervalMs: Long,
zkClient: ZkClient)
extends ShutdownableThread(commit-thread)
with KafkaMetricsGroup {
private val group = group- + id
group overwrite class Thread group member, causing the following compile
error:
overriding variable group in class Thread of type ThreadGroup; value group
has weaker access privileges; it should not be private
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-1959) Class CommitThread overwrite group of Thread class causing compile errors
[
https://issues.apache.org/jira/browse/KAFKA-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tong Li updated KAFKA-1959:
---
Attachment: KAFKA-1959.patch
Class CommitThread overwrite group of Thread class causing compile errors
-
Key: KAFKA-1959
URL: https://issues.apache.org/jira/browse/KAFKA-1959
Project: Kafka
Issue Type: Bug
Components: core
Environment: scala 2.10.4
Reporter: Tong Li
Labels: newbie
Attachments: KAFKA-1959.patch
class CommitThread(id: Int, partitionCount: Int, commitIntervalMs: Long,
zkClient: ZkClient)
extends ShutdownableThread(commit-thread)
with KafkaMetricsGroup {
private val group = group- + id
group overwrite class Thread group member, causing the following compile
error:
overriding variable group in class Thread of type ThreadGroup; value group
has weaker access privileges; it should not be private
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1960) .gitignore does not exclude test generated files and folders.
[ https://issues.apache.org/jira/browse/KAFKA-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14323347#comment-14323347 ] Tong Li commented on KAFKA-1960: Created reviewboard https://reviews.apache.org/r/31097/diff/ against branch origin/trunk .gitignore does not exclude test generated files and folders. - Key: KAFKA-1960 URL: https://issues.apache.org/jira/browse/KAFKA-1960 Project: Kafka Issue Type: Bug Components: build Reporter: Tong Li Priority: Minor Labels: newbie Attachments: KAFKA-1960.patch gradle test can create quite few folders, .gitignore should exclude these files for an easier git submit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1960) .gitignore does not exclude test generated files and folders.
[ https://issues.apache.org/jira/browse/KAFKA-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tong Li updated KAFKA-1960: --- Attachment: KAFKA-1960.patch .gitignore does not exclude test generated files and folders. - Key: KAFKA-1960 URL: https://issues.apache.org/jira/browse/KAFKA-1960 Project: Kafka Issue Type: Bug Components: build Reporter: Tong Li Priority: Minor Labels: newbie Attachments: KAFKA-1960.patch gradle test can create quite few folders, .gitignore should exclude these files for an easier git submit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tong Li updated KAFKA-1926: --- Attachment: KAFKA-1926.patch Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.0 Reporter: Jay Kreps Labels: newbie, patch Attachments: KAFKA-1926.patch, KAFKA-1926.patch There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14320416#comment-14320416 ] Tong Li commented on KAFKA-1926: [~jkreps] Removal of Time.scala cause a lot of code changes. Since Time.scala defined the SystemTime object and get used all over the place. Though another class SystemTime defined in client util package as a java class. The use of the SystemTime object in the core is like using the property, not method calls, I can rework that part, but it will be a lot of code like this SystemTime.milliseconds To be changed to: new SystemTime().milliseconds() Since the code in client common util gets defined as a regular class. I feel doing that also will cost quite a bit rather than stick with the scala SystemTime and simply calls its property, not many instance gets created. Let me know what you think and we can always improve. Thanks. Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.0 Reporter: Jay Kreps Labels: newbie, patch Attachments: KAFKA-1926.patch, KAFKA-1926.patch There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14320404#comment-14320404 ] Tong Li commented on KAFKA-1926: Created reviewboard https://reviews.apache.org/r/31007/diff/ against branch origin/trunk Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.0 Reporter: Jay Kreps Labels: newbie, patch Attachments: KAFKA-1926.patch, KAFKA-1926.patch There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1948) kafka.api.consumerTests are hanging
[ https://issues.apache.org/jira/browse/KAFKA-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318311#comment-14318311 ] Tong Li commented on KAFKA-1948: Pulled down the latest trunk and run into this problem. It will be nice if someone can take a look. The test does not stuck always at the same place. I did notice there are tons of threads getting created and died. What I did is when the test hung, I ran this command. top -H -p theprocessid of the test java You will notice that many threads come and go. and the state of these threads are always S (meaning sleep). including the parent thread. Not really sure what the deal is. kafka.api.consumerTests are hanging --- Key: KAFKA-1948 URL: https://issues.apache.org/jira/browse/KAFKA-1948 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Noticed today that very often when I run the full test suite, it hangs on kafka.api.consumerTest (not always same test though). It doesn't reproduce 100% of the time, but enough to be very annoying. I also saw it happening on trunk after KAFKA-1333: https://builds.apache.org/view/All/job/Kafka-trunk/389/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316830#comment-14316830 ] Tong Li commented on KAFKA-1926: [~jkreps] Jay, really appreciate your quick review comments. I will see what I can do and will submit another patch based on trunk branch. New patch set will come up real soon. Thanks so much. Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.0 Reporter: Jay Kreps Labels: newbie, patch Attachments: KAFKA-1926.patch There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316380#comment-14316380 ] Tong Li commented on KAFKA-1926: [~harsha_ch]Yeah, will do that. Thanks. Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.0 Reporter: Jay Kreps Labels: newbie, patch Attachments: KAFKA-1926.patch There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tong Li updated KAFKA-1926: --- Reviewer: Neha Narkhede Labels: newbie patch (was: newbie) Affects Version/s: 0.8.2.0 Status: Patch Available (was: Open) The first part of the attempt to fix kafka-1926 Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.0 Reporter: Jay Kreps Labels: newbie, patch There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tong Li updated KAFKA-1926: --- Attachment: KAFKA-1926.patch Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.0 Reporter: Jay Kreps Labels: newbie, patch Attachments: KAFKA-1926-p1.patch, KAFKA-1926.patch There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314948#comment-14314948 ] Tong Li commented on KAFKA-1926: Created reviewboard https://reviews.apache.org/r/30845/diff/ against branch origin/0.8.2 Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.0 Reporter: Jay Kreps Labels: newbie, patch Attachments: KAFKA-1926-p1.patch, KAFKA-1926.patch There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314962#comment-14314962 ] Tong Li commented on KAFKA-1926: [~harsha_ch], yes, new bee mistake. used the tool submitted the patch set for review. Seems it is working fine now. Thanks. Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.0 Reporter: Jay Kreps Labels: newbie, patch Attachments: KAFKA-1926.patch There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tong Li updated KAFKA-1926: --- Attachment: KAFKA-1926-p1.patch This is the first attempt to fix the issue. Would like to get some reviews and confirm if the direction is right. Thanks. Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.0 Reporter: Jay Kreps Labels: newbie, patch Attachments: KAFKA-1926-p1.patch There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tong Li updated KAFKA-1926: --- Attachment: (was: KAFKA-1926-p1.patch) Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.0 Reporter: Jay Kreps Labels: newbie, patch Attachments: KAFKA-1926.patch There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14313014#comment-14313014 ] Tong Li commented on KAFKA-1926: Jay. awesome comment, really appreciated. I will start with this one. Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Reporter: Jay Kreps Labels: newbie There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312229#comment-14312229 ] Tong Li commented on KAFKA-1926: Jay, if you can provide a bit more direction, I would like to start working on these things. Replace kafka.utils.Utils with o.a.k.common.utils.Utils --- Key: KAFKA-1926 URL: https://issues.apache.org/jira/browse/KAFKA-1926 Project: Kafka Issue Type: Improvement Reporter: Jay Kreps Labels: newbie There is currently a lot of duplication between the Utils class in common and the one in core. Our plan has been to deprecate duplicate code in the server and replace it with the new common code. As such we should evaluate each method in the scala Utils and do one of the following: 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose utility in active use that is not Kafka-specific. If we migrate it we should really think about the API and make sure there is some test coverage. A few things in there are kind of funky and we shouldn't just blindly copy them over. 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold any utilities that really need to make use of Scala features to be convenient. 3. Delete it if it is not used, or has a bad api. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1927) Replace requests in kafka.api with requests in org.apache.kafka.common.requests
[ https://issues.apache.org/jira/browse/KAFKA-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312280#comment-14312280 ] Tong Li commented on KAFKA-1927: Gwen Shapira, indeed. I was doing OpenStack development for quite some time. Recently IBM wants to be involved in Kafka, that is why I am here. I like to give it a shot. Reading the link and the process probably will be slow. How urgent is this patch being needed? If this is very urgent, probably better for some more experienced Kafka developer. Thanks a lot for your quick response though. Replace requests in kafka.api with requests in org.apache.kafka.common.requests --- Key: KAFKA-1927 URL: https://issues.apache.org/jira/browse/KAFKA-1927 Project: Kafka Issue Type: Improvement Reporter: Jay Kreps The common package introduced a better way of defining requests using a new protocol definition DSL and also includes wrapper objects for these. We should switch KafkaApis over to use these request definitions and consider the scala classes deprecated (we probably need to retain some of them for a while for the scala clients). This will be a big improvement because 1. We will have each request now defined in only one place (Protocol.java) 2. We will have built-in support for multi-version requests 3. We will have much better error messages (no more cryptic underflow errors) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1756) never allow the replica fetch size to be less than the max message size
[ https://issues.apache.org/jira/browse/KAFKA-1756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301373#comment-14301373 ] Tong Li commented on KAFKA-1756: I wonder if we really do need two parameters. Can't we just use one? If not, in the program we can always use the one which is the greater of the two. Seems we can solve that issue relatively easy. never allow the replica fetch size to be less than the max message size --- Key: KAFKA-1756 URL: https://issues.apache.org/jira/browse/KAFKA-1756 Project: Kafka Issue Type: Bug Affects Versions: 0.8.1.1, 0.8.2 Reporter: Joe Stein Priority: Blocker Fix For: 0.8.3 There exists a very hazardous scenario where if the max.message.bytes is greather than the replica.fetch.max.bytes the message will never replicate. This will bring the ISR down to 1 (eventually/quickly once replica.lag.max.messages is reached). If during this window the leader itself goes out of the ISR then the new leader will commit the last offset it replicated. This is also bad for sync producers with -1 ack because they will all block (heard affect caused upstream) in this scenario too. The fix here is two fold 1) when setting max.message.bytes using kafka-topics we must check first each and every broker (which will need some thought about how todo this because of the topiccommand zk notification) that max.message.bytes = replica.fetch.max.bytes and if it is NOT then DO NOT create the topic 2) if you change this in server.properties then the broker should not start if max.message.bytes replica.fetch.max.bytes This does beg the question/issue some about centralizing certain/some/all configurations so that inconsistencies do not occur (where broker 1 has max.message.bytes replica.fetch.max.bytes but broker 2 max.message.bytes = replica.fetch.max.bytes because of error in properties). I do not want to conflate this ticket but I think it is worth mentioning/bringing up here as it is a good example where it could make sense. I set this as BLOCKER for 0.8.2-beta because we did so much work to enable consistency vs availability and 0 data loss this corner case should be part of 0.8.2-final Also, I could go one step further (though I would not consider this part as a blocker for 0.8.2 but interested to what other folks think) about a consumer replica fetch size so that if the message max is increased messages will no longer be consumed (since the consumer fetch max would be max.message.bytes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1810) Add IP Filtering / Whitelists-Blacklists
[ https://issues.apache.org/jira/browse/KAFKA-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14299202#comment-14299202 ] Tong Li commented on KAFKA-1810: rather than add specific security measures, can we add some kind of plugin point so that any plugins can be configured to do that type of work. Either it is a IP filter or certificate filter or basic authentication filter we can simply enable these plugins according to our own needs. This way, kafka only provide the plugin point, nothing else, how the plugin gets developed , performs, are not really the concern of the kafka community, we can have a clear separation of concerns. This has been done in many other successful projects, new to kafka, just saying we can do some thing like middle ware (in python term) or servlet filter in java world. Add IP Filtering / Whitelists-Blacklists - Key: KAFKA-1810 URL: https://issues.apache.org/jira/browse/KAFKA-1810 Project: Kafka Issue Type: New Feature Components: core, network, security Reporter: Jeff Holoman Assignee: Jeff Holoman Priority: Minor Fix For: 0.8.3 Attachments: KAFKA-1810.patch, KAFKA-1810_2015-01-15_19:47:14.patch While longer-term goals of security in Kafka are on the roadmap there exists some value for the ability to restrict connection to Kafka brokers based on IP address. This is not intended as a replacement for security but more of a precaution against misconfiguration and to provide some level of control to Kafka administrators about who is reading/writing to their cluster. 1) In some organizations software administration vs o/s systems administration and network administration is disjointed and not well choreographed. Providing software administrators the ability to configure their platform relatively independently (after initial configuration) from Systems administrators is desirable. 2) Configuration and deployment is sometimes error prone and there are situations when test environments could erroneously read/write to production environments 3) An additional precaution against reading sensitive data is typically welcomed in most large enterprise deployments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
