[jira] [Commented] (KAFKA-2176) DefaultPartitioner doesn't perform consistent hashing based on
[ https://issues.apache.org/jira/browse/KAFKA-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541484#comment-14541484 ] Igor Maravić commented on KAFKA-2176: - Partitioning is working as expected with new producer. I'll close the ticket than. DefaultPartitioner doesn't perform consistent hashing based on --- Key: KAFKA-2176 URL: https://issues.apache.org/jira/browse/KAFKA-2176 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1 Reporter: Igor Maravić Labels: easyfix, newbie Fix For: 0.8.1 Attachments: KAFKA-2176.patch While deploying MirrorMakers in production, we configured it to use kafka.producer.DefaultPartitioner. By doing this and since we had the same amount partitions for the topic in local and aggregation cluster, we expect that the messages will be partitioned the same way. This wasn't the case. Messages were properly partitioned with DefaultPartitioner on our local cluster, since the key was of the type String. On the MirrorMaker side, the messages were not properly partitioned. Problem is that the Array[Byte] doesn't implement hashCode function, since it is mutable collection. Fix is to calculate the deep hash code if the key is of Array type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2176) DefaultPartitioner doesn't perform consistent hashing based on
[ https://issues.apache.org/jira/browse/KAFKA-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539320#comment-14539320 ] Gwen Shapira commented on KAFKA-2176: - Do we want to fix this? It is not an issue in the new Producer, which MirrorMaker can use. DefaultPartitioner doesn't perform consistent hashing based on --- Key: KAFKA-2176 URL: https://issues.apache.org/jira/browse/KAFKA-2176 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1 Reporter: Igor Maravić Labels: easyfix, newbie Fix For: 0.8.1 Attachments: KAFKA-2176.patch While deploying MirrorMakers in production, we configured it to use kafka.producer.DefaultPartitioner. By doing this and since we had the same amount partitions for the topic in local and aggregation cluster, we expect that the messages will be partitioned the same way. This wasn't the case. Messages were properly partitioned with DefaultPartitioner on our local cluster, since the key was of the type String. On the MirrorMaker side, the messages were not properly partitioned. Problem is that the Array[Byte] doesn't implement hashCode function, since it is mutable collection. Fix is to calculate the deep hash code if the key is of Array type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2176) DefaultPartitioner doesn't perform consistent hashing based on
[ https://issues.apache.org/jira/browse/KAFKA-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539393#comment-14539393 ] Igor Maravić commented on KAFKA-2176: - I don't see why this wouldn't be fixed. As far as I can tell MirrorMaker is not using new producer yet, so this BUG is still relevant for it. DefaultPartitioner doesn't perform consistent hashing based on --- Key: KAFKA-2176 URL: https://issues.apache.org/jira/browse/KAFKA-2176 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1 Reporter: Igor Maravić Labels: easyfix, newbie Fix For: 0.8.1 Attachments: KAFKA-2176.patch While deploying MirrorMakers in production, we configured it to use kafka.producer.DefaultPartitioner. By doing this and since we had the same amount partitions for the topic in local and aggregation cluster, we expect that the messages will be partitioned the same way. This wasn't the case. Messages were properly partitioned with DefaultPartitioner on our local cluster, since the key was of the type String. On the MirrorMaker side, the messages were not properly partitioned. Problem is that the Array[Byte] doesn't implement hashCode function, since it is mutable collection. Fix is to calculate the deep hash code if the key is of Array type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2176) DefaultPartitioner doesn't perform consistent hashing based on
[ https://issues.apache.org/jira/browse/KAFKA-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539409#comment-14539409 ] Gwen Shapira commented on KAFKA-2176: - 0.8.2 has new.producer parameter that can be set to true. DefaultPartitioner doesn't perform consistent hashing based on --- Key: KAFKA-2176 URL: https://issues.apache.org/jira/browse/KAFKA-2176 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1 Reporter: Igor Maravić Labels: easyfix, newbie Fix For: 0.8.1 Attachments: KAFKA-2176.patch While deploying MirrorMakers in production, we configured it to use kafka.producer.DefaultPartitioner. By doing this and since we had the same amount partitions for the topic in local and aggregation cluster, we expect that the messages will be partitioned the same way. This wasn't the case. Messages were properly partitioned with DefaultPartitioner on our local cluster, since the key was of the type String. On the MirrorMaker side, the messages were not properly partitioned. Problem is that the Array[Byte] doesn't implement hashCode function, since it is mutable collection. Fix is to calculate the deep hash code if the key is of Array type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2176) DefaultPartitioner doesn't perform consistent hashing based on
[ https://issues.apache.org/jira/browse/KAFKA-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539430#comment-14539430 ] Igor Maravić commented on KAFKA-2176: - Didn't know that. How much is this battle tested? We had a lot of issues with new producer, and by looking at the mailing lists a lot of other people are having problems with it. In our case, we just reverted to use the old Kafka Producer for our internal API. But my point is that I still think that the patch is relevant, since the old producer is still used. DefaultPartitioner doesn't perform consistent hashing based on --- Key: KAFKA-2176 URL: https://issues.apache.org/jira/browse/KAFKA-2176 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1 Reporter: Igor Maravić Labels: easyfix, newbie Fix For: 0.8.1 Attachments: KAFKA-2176.patch While deploying MirrorMakers in production, we configured it to use kafka.producer.DefaultPartitioner. By doing this and since we had the same amount partitions for the topic in local and aggregation cluster, we expect that the messages will be partitioned the same way. This wasn't the case. Messages were properly partitioned with DefaultPartitioner on our local cluster, since the key was of the type String. On the MirrorMaker side, the messages were not properly partitioned. Problem is that the Array[Byte] doesn't implement hashCode function, since it is mutable collection. Fix is to calculate the deep hash code if the key is of Array type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2176) DefaultPartitioner doesn't perform consistent hashing based on
[ https://issues.apache.org/jira/browse/KAFKA-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539429#comment-14539429 ] Igor Maravić commented on KAFKA-2176: - Didn't know that. How much is this battle tested? We had a lot of issues with new producer, and by looking at the mailing lists a lot of other people are having problems with it. In our case, we just reverted to use the old Kafka Producer for our internal API. But my point is that I still think that the patch is relevant, since the old producer is still used. DefaultPartitioner doesn't perform consistent hashing based on --- Key: KAFKA-2176 URL: https://issues.apache.org/jira/browse/KAFKA-2176 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1 Reporter: Igor Maravić Labels: easyfix, newbie Fix For: 0.8.1 Attachments: KAFKA-2176.patch While deploying MirrorMakers in production, we configured it to use kafka.producer.DefaultPartitioner. By doing this and since we had the same amount partitions for the topic in local and aggregation cluster, we expect that the messages will be partitioned the same way. This wasn't the case. Messages were properly partitioned with DefaultPartitioner on our local cluster, since the key was of the type String. On the MirrorMaker side, the messages were not properly partitioned. Problem is that the Array[Byte] doesn't implement hashCode function, since it is mutable collection. Fix is to calculate the deep hash code if the key is of Array type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2176) DefaultPartitioner doesn't perform consistent hashing based on
[ https://issues.apache.org/jira/browse/KAFKA-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539451#comment-14539451 ] Gwen Shapira commented on KAFKA-2176: - 1. Regarding battle testing, LinkedIn is the biggest MirrorMaker user AFAIK, so I'll let [~guozhang], your original reviewer reply to that 2. As I said, I'd rather put efforts into improving the new clients and declare the existing as deprecated rather than keep maintaining them. I know the old producer is still used, but the way features are typically deprecated is that when someone runs into this issue, we recommend migrating to the newer code. Anyway, since you already patched the old one, I'll make [~guozhang] make that call. Just giving my 2c and letting you know you can test the new producer (the one we actually put a lot of effort into improving). DefaultPartitioner doesn't perform consistent hashing based on --- Key: KAFKA-2176 URL: https://issues.apache.org/jira/browse/KAFKA-2176 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1 Reporter: Igor Maravić Labels: easyfix, newbie Fix For: 0.8.1 Attachments: KAFKA-2176.patch While deploying MirrorMakers in production, we configured it to use kafka.producer.DefaultPartitioner. By doing this and since we had the same amount partitions for the topic in local and aggregation cluster, we expect that the messages will be partitioned the same way. This wasn't the case. Messages were properly partitioned with DefaultPartitioner on our local cluster, since the key was of the type String. On the MirrorMaker side, the messages were not properly partitioned. Problem is that the Array[Byte] doesn't implement hashCode function, since it is mutable collection. Fix is to calculate the deep hash code if the key is of Array type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2176) DefaultPartitioner doesn't perform consistent hashing based on
[ https://issues.apache.org/jira/browse/KAFKA-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14540784#comment-14540784 ] Jiangjie Qin commented on KAFKA-2176: - [~i_maravic], LinkedIn uses the mirror maker on trunk, so we are already using the new producer. I think KIP-22 probably will solve your issue. We are fixing some issues that have been identified in the new producer as well as adding some things to it. As [~gwenshap] said, I think it is probably better to fix forward on the new producer instead of sticking to the old producer. DefaultPartitioner doesn't perform consistent hashing based on --- Key: KAFKA-2176 URL: https://issues.apache.org/jira/browse/KAFKA-2176 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1 Reporter: Igor Maravić Labels: easyfix, newbie Fix For: 0.8.1 Attachments: KAFKA-2176.patch While deploying MirrorMakers in production, we configured it to use kafka.producer.DefaultPartitioner. By doing this and since we had the same amount partitions for the topic in local and aggregation cluster, we expect that the messages will be partitioned the same way. This wasn't the case. Messages were properly partitioned with DefaultPartitioner on our local cluster, since the key was of the type String. On the MirrorMaker side, the messages were not properly partitioned. Problem is that the Array[Byte] doesn't implement hashCode function, since it is mutable collection. Fix is to calculate the deep hash code if the key is of Array type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2176) DefaultPartitioner doesn't perform consistent hashing based on
[ https://issues.apache.org/jira/browse/KAFKA-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532514#comment-14532514 ] Igor Maravić commented on KAFKA-2176: - Created reviewboard https://reviews.apache.org/r/33939/diff/ against branch origin/trunk DefaultPartitioner doesn't perform consistent hashing based on --- Key: KAFKA-2176 URL: https://issues.apache.org/jira/browse/KAFKA-2176 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1 Reporter: Igor Maravić Labels: easyfix, newbie Fix For: 0.8.1 Attachments: KAFKA-2176.patch While deploying MirrorMakers in production, we configured it to use kafka.producer.DefaultPartitioner. By doing this and since we had the same amount partitions for the topic in local and aggregation cluster, we expect that the messages will be partitioned the same way. This wasn't the case. Messages were properly partitioned with DefaultPartitioner on our local cluster, since the key was of the type String. On the MirrorMaker side, the messages were not properly partitioned. Problem is that the Array[Byte] doesn't implement hashCode function, since it is mutable collection. Fix is to calculate the deep hash code if the key is of Array type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
