[ https://issues.apache.org/jira/browse/KAFKA-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
George Smith updated KAFKA-6434: -------------------------------- Description: Our usecase: We are migrating Spark streaming app into Kafka streaming. We want to continue processing from the last processed offsets of the Spark streaming app. Therefore we want to define new consumer group (application id) with given offsets. The new app was not launched yet (we don't want to make side effects of processing into db) -> new consumer group does not exist. I was happy to see the updated Kafka-consumer-groups.sh supports reset-offsets method. Unfortunately it seems it's not working as expected. {code} kafka-consumer-groups.sh --reset-offsets --bootstrap-server localhost:9092 --topic testTopic:0 --group testGROUP --to-offset 10 Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers). TOPIC PARTITION NEW-OFFSET testTopic 0 10 {code} Now I want to check offsets for the group: {code} kafka-consumer-groups.sh --describe --bootstrap-server localhost:9092 --group testGROUP Error: Consumer group 'testGROUP' does not exist. {code} That's strange, isn't it? On the other side when I use kafka-streams-application-reset.sh - the group is obviously created - unfortunately this tool does not support given offsets for partitions (only the beginning is supported) + it does not support secured Kafka connection... was: Our usecase: We are migrating Spark streaming app into Kafka streaming. We want to continue processing from the last processed offsets of the Spark streaming app. Therefore we want to create define new consumer group with given offsets. The new app was not launched yet (we don't want to make side effects of processing into db) -> new consumer group does not exist. I was happy to see the updated Kafka-consumer-groups.sh supports reset-offsets method. Unfortunately it seems it's not working as expected. {code} kafka-consumer-groups.sh --reset-offsets --bootstrap-server localhost:9092 --topic testTopic:0 --group testGROUP --to-offset 10 Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers). TOPIC PARTITION NEW-OFFSET testTopic 0 10 {code} Now I want to check offsets for the group: {code} kafka-consumer-groups.sh --describe --bootstrap-server localhost:9092 --group testGROUP Error: Consumer group 'testGROUP' does not exist. {code} That's strange, isn't it? On the other side when I use kafka-streams-application-reset.sh - the group is obviously created - unfortunately this tool does not support given offsets for partitions (only the beginning is supported) + it does not support secured Kafka connection... > Kafka-consumer-groups.sh reset-offsets does not work properly for not > existing group > ------------------------------------------------------------------------------------ > > Key: KAFKA-6434 > URL: https://issues.apache.org/jira/browse/KAFKA-6434 > Project: Kafka > Issue Type: Bug > Components: tools > Affects Versions: 0.11.0.2 > Reporter: George Smith > > Our usecase: We are migrating Spark streaming app into Kafka streaming. We > want to continue processing from the last processed offsets of the Spark > streaming app. Therefore we want to define new consumer group (application > id) with given offsets. The new app was not launched yet (we don't want to > make side effects of processing into db) -> new consumer group does not exist. > I was happy to see the updated Kafka-consumer-groups.sh supports > reset-offsets method. Unfortunately it seems it's not working as expected. > {code} > kafka-consumer-groups.sh --reset-offsets --bootstrap-server localhost:9092 > --topic testTopic:0 --group testGROUP --to-offset 10 > Note: This will only show information about consumers that use the Java > consumer API (non-ZooKeeper-based consumers). > TOPIC PARTITION NEW-OFFSET > testTopic 0 10 > {code} > Now I want to check offsets for the group: > {code} > kafka-consumer-groups.sh --describe --bootstrap-server localhost:9092 --group > testGROUP > Error: Consumer group 'testGROUP' does not exist. > {code} > That's strange, isn't it? > On the other side when I use kafka-streams-application-reset.sh - the group > is obviously created - unfortunately this tool does not support given offsets > for partitions (only the beginning is supported) + it does not support > secured Kafka connection... -- This message was sent by Atlassian JIRA (v6.4.14#64029)