[jira] [Commented] (KAFKA-1870) Cannot commit with simpleConsumer on Zookeeper only with Java API
[ https://issues.apache.org/jira/browse/KAFKA-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302679#comment-14302679 ] Joel Koshy commented on KAFKA-1870: --- [~junrao] (at least on trunk) I think the change to kafka.javaapi.OffsetFetchRequest should be reverted since people who are using that original constructor should be explicitly setting the versionId to 0. The issue is that right now you cannot fetch offsets from Kafka using javaapi. Cannot commit with simpleConsumer on Zookeeper only with Java API - Key: KAFKA-1870 URL: https://issues.apache.org/jira/browse/KAFKA-1870 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 0.8.2 Reporter: Thomas Vandevelde Assignee: Jun Rao Priority: Blocker Fix For: 0.8.2 Attachments: OffsetCommitRequest.diff, kafka-1870.patch, kafka-1870_2015-01-16_12:08:05.patch Original Estimate: 1h Remaining Estimate: 1h From Kafka 0.8.2, we need to pass version 0 in the OffsetCommitRequest if we want to commit only on zookeeper. However the Java API doesnt allow to pass the version in parameter. Can you please add the version in the constructor of kafka.javaapi.OffsetCommitRequest ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1870) Cannot commit with simpleConsumer on Zookeeper only with Java API
[ https://issues.apache.org/jira/browse/KAFKA-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281389#comment-14281389 ] Thomas Vandevelde commented on KAFKA-1870: -- Thanks for the very quick fix guys! It was indeed blocking us to move from 0.8.1 to 0.8.2 I can see that only the javaapi version is restricted to use v0, The scala one still has v1 as default. Is it expected? In our case, i suppose we will wait for the new kafkaconsumer to commit to kafka instead of zookeeper. Do you know when 0.8.2 will be released by any chance? Cannot commit with simpleConsumer on Zookeeper only with Java API - Key: KAFKA-1870 URL: https://issues.apache.org/jira/browse/KAFKA-1870 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 0.8.2 Reporter: Thomas Vandevelde Assignee: Jun Rao Priority: Blocker Fix For: 0.8.2 Attachments: OffsetCommitRequest.diff, kafka-1870.patch, kafka-1870_2015-01-16_12:08:05.patch Original Estimate: 1h Remaining Estimate: 1h From Kafka 0.8.2, we need to pass version 0 in the OffsetCommitRequest if we want to commit only on zookeeper. However the Java API doesnt allow to pass the version in parameter. Can you please add the version in the constructor of kafka.javaapi.OffsetCommitRequest ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1870) Cannot commit with simpleConsumer on Zookeeper only with Java API
[ https://issues.apache.org/jira/browse/KAFKA-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281436#comment-14281436 ] Jun Rao commented on KAFKA-1870: Yes, the scala one still defaults to version 1. This is a bit harder to fix since both OffsetCommitRequest and OffsetFetchRequest are used internally and we always want the default to be the latest version. Technically, only the javaapi is our public api. If you are using the scala api, the workaround is to explicitly specify version 0 when constructing OffsetCommitRequest and OffsetFetchRequest. 0.8.2 is being voted now and hopefully we can release it by end of next week. Cannot commit with simpleConsumer on Zookeeper only with Java API - Key: KAFKA-1870 URL: https://issues.apache.org/jira/browse/KAFKA-1870 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 0.8.2 Reporter: Thomas Vandevelde Assignee: Jun Rao Priority: Blocker Fix For: 0.8.2 Attachments: OffsetCommitRequest.diff, kafka-1870.patch, kafka-1870_2015-01-16_12:08:05.patch Original Estimate: 1h Remaining Estimate: 1h From Kafka 0.8.2, we need to pass version 0 in the OffsetCommitRequest if we want to commit only on zookeeper. However the Java API doesnt allow to pass the version in parameter. Can you please add the version in the constructor of kafka.javaapi.OffsetCommitRequest ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1870) Cannot commit with simpleConsumer on Zookeeper only with Java API
[ https://issues.apache.org/jira/browse/KAFKA-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280429#comment-14280429 ] Sriharsha Chintalapani commented on KAFKA-1870: --- [~tomvdv] can you send the patch via review board. Here are the instructions https://cwiki.apache.org/confluence/display/KAFKA/Kafka+patch+review+tool Cannot commit with simpleConsumer on Zookeeper only with Java API - Key: KAFKA-1870 URL: https://issues.apache.org/jira/browse/KAFKA-1870 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 0.8.2 Reporter: Thomas Vandevelde Assignee: Neha Narkhede Priority: Minor Attachments: OffsetCommitRequest.diff Original Estimate: 1h Remaining Estimate: 1h From Kafka 0.8.2, we need to pass version 0 in the OffsetCommitRequest if we want to commit only on zookeeper. However the Java API doesnt allow to pass the version in parameter. Can you please add the version in the constructor of kafka.javaapi.OffsetCommitRequest ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1870) Cannot commit with simpleConsumer on Zookeeper only with Java API
[ https://issues.apache.org/jira/browse/KAFKA-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280517#comment-14280517 ] Guozhang Wang commented on KAFKA-1870: -- Thanks for the findings Thomas. However since the javaapi.OffsetCommitRequest is a public API class, adding the version_id into the construct directly will make backward incompatible changes: people using it without worrying the version id will break their apps when moving on to the new kafka vesion. How about adding another constructor which takes the version id and leave the current one as-is? Guozhang Cannot commit with simpleConsumer on Zookeeper only with Java API - Key: KAFKA-1870 URL: https://issues.apache.org/jira/browse/KAFKA-1870 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 0.8.2 Reporter: Thomas Vandevelde Assignee: Neha Narkhede Priority: Minor Attachments: OffsetCommitRequest.diff Original Estimate: 1h Remaining Estimate: 1h From Kafka 0.8.2, we need to pass version 0 in the OffsetCommitRequest if we want to commit only on zookeeper. However the Java API doesnt allow to pass the version in parameter. Can you please add the version in the constructor of kafka.javaapi.OffsetCommitRequest ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1870) Cannot commit with simpleConsumer on Zookeeper only with Java API
[ https://issues.apache.org/jira/browse/KAFKA-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280619#comment-14280619 ] Joel Koshy commented on KAFKA-1870: --- Committing offsets to Kafka is a little more complicated - i.e., you need to discover the offset manager for your group and then send the request to that broker. https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka I think the SimpleConsumer should only send v0 of OffsetCommitRequest. For people who wish to commit offsets manually to ZooKeeper they will need to do so explicitly (via a BlockingChannel) or until we get KAFKA-1013 in. Cannot commit with simpleConsumer on Zookeeper only with Java API - Key: KAFKA-1870 URL: https://issues.apache.org/jira/browse/KAFKA-1870 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 0.8.2 Reporter: Thomas Vandevelde Priority: Minor Attachments: OffsetCommitRequest.diff Original Estimate: 1h Remaining Estimate: 1h From Kafka 0.8.2, we need to pass version 0 in the OffsetCommitRequest if we want to commit only on zookeeper. However the Java API doesnt allow to pass the version in parameter. Can you please add the version in the constructor of kafka.javaapi.OffsetCommitRequest ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1870) Cannot commit with simpleConsumer on Zookeeper only with Java API
[ https://issues.apache.org/jira/browse/KAFKA-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280632#comment-14280632 ] Thomas Vandevelde commented on KAFKA-1870: -- Do you mean we should then have the default version 0 instead of 1 in OffsetCommitRequest ? Cannot commit with simpleConsumer on Zookeeper only with Java API - Key: KAFKA-1870 URL: https://issues.apache.org/jira/browse/KAFKA-1870 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 0.8.2 Reporter: Thomas Vandevelde Priority: Minor Attachments: OffsetCommitRequest.diff Original Estimate: 1h Remaining Estimate: 1h From Kafka 0.8.2, we need to pass version 0 in the OffsetCommitRequest if we want to commit only on zookeeper. However the Java API doesnt allow to pass the version in parameter. Can you please add the version in the constructor of kafka.javaapi.OffsetCommitRequest ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1870) Cannot commit with simpleConsumer on Zookeeper only with Java API
[ https://issues.apache.org/jira/browse/KAFKA-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280715#comment-14280715 ] Joel Koshy commented on KAFKA-1870: --- Not really - the SimpleConsumer commitOffsets method can explicitly set the version to 0. What do you/others think? Cannot commit with simpleConsumer on Zookeeper only with Java API - Key: KAFKA-1870 URL: https://issues.apache.org/jira/browse/KAFKA-1870 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 0.8.2 Reporter: Thomas Vandevelde Priority: Minor Attachments: OffsetCommitRequest.diff Original Estimate: 1h Remaining Estimate: 1h From Kafka 0.8.2, we need to pass version 0 in the OffsetCommitRequest if we want to commit only on zookeeper. However the Java API doesnt allow to pass the version in parameter. Can you please add the version in the constructor of kafka.javaapi.OffsetCommitRequest ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1870) Cannot commit with simpleConsumer on Zookeeper only with Java API
[ https://issues.apache.org/jira/browse/KAFKA-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280725#comment-14280725 ] Guozhang Wang commented on KAFKA-1870: -- Joel, I think for v0 now the offsets will be still forwarded to ZK, no? Cannot commit with simpleConsumer on Zookeeper only with Java API - Key: KAFKA-1870 URL: https://issues.apache.org/jira/browse/KAFKA-1870 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 0.8.2 Reporter: Thomas Vandevelde Priority: Minor Attachments: OffsetCommitRequest.diff Original Estimate: 1h Remaining Estimate: 1h From Kafka 0.8.2, we need to pass version 0 in the OffsetCommitRequest if we want to commit only on zookeeper. However the Java API doesnt allow to pass the version in parameter. Can you please add the version in the constructor of kafka.javaapi.OffsetCommitRequest ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1870) Cannot commit with simpleConsumer on Zookeeper only with Java API
[ https://issues.apache.org/jira/browse/KAFKA-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280728#comment-14280728 ] Jun Rao commented on KAFKA-1870: Created reviewboard https://reviews.apache.org/r/29977/diff/ against branch origin/0.8.2 Cannot commit with simpleConsumer on Zookeeper only with Java API - Key: KAFKA-1870 URL: https://issues.apache.org/jira/browse/KAFKA-1870 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 0.8.2 Reporter: Thomas Vandevelde Priority: Minor Attachments: OffsetCommitRequest.diff, kafka-1870.patch Original Estimate: 1h Remaining Estimate: 1h From Kafka 0.8.2, we need to pass version 0 in the OffsetCommitRequest if we want to commit only on zookeeper. However the Java API doesnt allow to pass the version in parameter. Can you please add the version in the constructor of kafka.javaapi.OffsetCommitRequest ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1870) Cannot commit with simpleConsumer on Zookeeper only with Java API
[ https://issues.apache.org/jira/browse/KAFKA-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280736#comment-14280736 ] Joel Koshy commented on KAFKA-1870: --- [~guozhang] yes that is the intended effect. i.e., the documentation will be: you cannot commit to Kafka with SimpleConsumer. Cannot commit with simpleConsumer on Zookeeper only with Java API - Key: KAFKA-1870 URL: https://issues.apache.org/jira/browse/KAFKA-1870 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 0.8.2 Reporter: Thomas Vandevelde Assignee: Jun Rao Priority: Minor Attachments: OffsetCommitRequest.diff, kafka-1870.patch Original Estimate: 1h Remaining Estimate: 1h From Kafka 0.8.2, we need to pass version 0 in the OffsetCommitRequest if we want to commit only on zookeeper. However the Java API doesnt allow to pass the version in parameter. Can you please add the version in the constructor of kafka.javaapi.OffsetCommitRequest ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1870) Cannot commit with simpleConsumer on Zookeeper only with Java API
[ https://issues.apache.org/jira/browse/KAFKA-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280742#comment-14280742 ] Joel Koshy commented on KAFKA-1870: --- Actually, sorry - nm. I just misread the API. For some reason I was thinking SimpleConsumer's commitOffsets took the offset array and not the actual request. Cannot commit with simpleConsumer on Zookeeper only with Java API - Key: KAFKA-1870 URL: https://issues.apache.org/jira/browse/KAFKA-1870 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 0.8.2 Reporter: Thomas Vandevelde Assignee: Jun Rao Priority: Minor Attachments: OffsetCommitRequest.diff, kafka-1870.patch Original Estimate: 1h Remaining Estimate: 1h From Kafka 0.8.2, we need to pass version 0 in the OffsetCommitRequest if we want to commit only on zookeeper. However the Java API doesnt allow to pass the version in parameter. Can you please add the version in the constructor of kafka.javaapi.OffsetCommitRequest ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1870) Cannot commit with simpleConsumer on Zookeeper only with Java API
[ https://issues.apache.org/jira/browse/KAFKA-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280756#comment-14280756 ] Jun Rao commented on KAFKA-1870: Updated reviewboard https://reviews.apache.org/r/29977/diff/ against branch origin/0.8.2 Cannot commit with simpleConsumer on Zookeeper only with Java API - Key: KAFKA-1870 URL: https://issues.apache.org/jira/browse/KAFKA-1870 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 0.8.2 Reporter: Thomas Vandevelde Assignee: Jun Rao Priority: Minor Attachments: OffsetCommitRequest.diff, kafka-1870.patch, kafka-1870_2015-01-16_12:08:05.patch Original Estimate: 1h Remaining Estimate: 1h From Kafka 0.8.2, we need to pass version 0 in the OffsetCommitRequest if we want to commit only on zookeeper. However the Java API doesnt allow to pass the version in parameter. Can you please add the version in the constructor of kafka.javaapi.OffsetCommitRequest ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1870) Cannot commit with simpleConsumer on Zookeeper only with Java API
[ https://issues.apache.org/jira/browse/KAFKA-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280768#comment-14280768 ] Jun Rao commented on KAFKA-1870: I think it makes sense to pin the version of OffsetCommitRequest and OffsetFetchRequest to version 0 so that they have the same behavior as in 0.8.1. There are likely people using this api and stopping support it w/o deprecating it first may not be ideal. Attaching a patch. This probably should be an 0.8.2 blocker. Cannot commit with simpleConsumer on Zookeeper only with Java API - Key: KAFKA-1870 URL: https://issues.apache.org/jira/browse/KAFKA-1870 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 0.8.2 Reporter: Thomas Vandevelde Assignee: Jun Rao Priority: Minor Attachments: OffsetCommitRequest.diff, kafka-1870.patch, kafka-1870_2015-01-16_12:08:05.patch Original Estimate: 1h Remaining Estimate: 1h From Kafka 0.8.2, we need to pass version 0 in the OffsetCommitRequest if we want to commit only on zookeeper. However the Java API doesnt allow to pass the version in parameter. Can you please add the version in the constructor of kafka.javaapi.OffsetCommitRequest ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1870) Cannot commit with simpleConsumer on Zookeeper only with Java API
[ https://issues.apache.org/jira/browse/KAFKA-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14280778#comment-14280778 ] Joel Koshy commented on KAFKA-1870: --- This is a pretty severe limitation of the scala javaapi request definitions. i.e., we don't provide explicit constructors for different versions (as we do in the Java request definitions in the clients package). We should really get rid of these and move completely over to the Java request definitions in the next or next++ release - so on that note I think it is reasonable to pin the version and enforce SimpleConsumer commits/fetches only to/from ZK. Cannot commit with simpleConsumer on Zookeeper only with Java API - Key: KAFKA-1870 URL: https://issues.apache.org/jira/browse/KAFKA-1870 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 0.8.2 Reporter: Thomas Vandevelde Assignee: Jun Rao Priority: Minor Attachments: OffsetCommitRequest.diff, kafka-1870.patch, kafka-1870_2015-01-16_12:08:05.patch Original Estimate: 1h Remaining Estimate: 1h From Kafka 0.8.2, we need to pass version 0 in the OffsetCommitRequest if we want to commit only on zookeeper. However the Java API doesnt allow to pass the version in parameter. Can you please add the version in the constructor of kafka.javaapi.OffsetCommitRequest ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
