[jira] [Commented] (KAFKA-969) Need to prevent failure of rebalance when there are no brokers available when consumer comes up
[ https://issues.apache.org/jira/browse/KAFKA-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13705517#comment-13705517 ] Swapnil Ghike commented on KAFKA-969: - I agree with Joel. Registering a consumers' subscription in zookeeper and being open to brokers and topics showing up after the consumer has started is reasonable. It's similar to a consumer waiting to consume data from the tail of a topic when no new data is coming in and partition count could change. Need to prevent failure of rebalance when there are no brokers available when consumer comes up --- Key: KAFKA-969 URL: https://issues.apache.org/jira/browse/KAFKA-969 Project: Kafka Issue Type: Bug Reporter: Sriram Subramanian Assignee: Sriram Subramanian Fix For: 0.8 Attachments: emptybrokeronrebalance-1.patch There are some rare instances when a consumer would be up before bringing up the Kafka brokers. This would usually happen in a test scenario. In such conditions, during rebalance instead of failing the rebalance we just log the error and subscribe to broker changes. When the broker comes back up, we trigger the rebalance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-969) Need to prevent failure of rebalance when there are no brokers available when consumer comes up
[ https://issues.apache.org/jira/browse/KAFKA-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13705954#comment-13705954 ] Guozhang Wang commented on KAFKA-969: - I think this is related to how we would enforce the order of starting servers/clients. In the quickstart Wiki page http://kafka.apache.org/08/quickstart.html We suggest to 1) start servers, 2) start producers and start sending messages, and then 3) start consumers. This is the normal case for production, whereas for dev testing this order would be different, such as 1) start servers with no topics created, 2) start producers and consumers, 3) producer start sending messages; or start 1) and 2) at roughly the same time; or even start 2) before start 1). When such order happens, these issues will arise. Need to prevent failure of rebalance when there are no brokers available when consumer comes up --- Key: KAFKA-969 URL: https://issues.apache.org/jira/browse/KAFKA-969 Project: Kafka Issue Type: Bug Reporter: Sriram Subramanian Assignee: Sriram Subramanian Fix For: 0.8 Attachments: emptybrokeronrebalance-1.patch There are some rare instances when a consumer would be up before bringing up the Kafka brokers. This would usually happen in a test scenario. In such conditions, during rebalance instead of failing the rebalance we just log the error and subscribe to broker changes. When the broker comes back up, we trigger the rebalance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-969) Need to prevent failure of rebalance when there are no brokers available when consumer comes up
[ https://issues.apache.org/jira/browse/KAFKA-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13705971#comment-13705971 ] Sam Meder commented on KAFKA-969: - I feel that Kafka servers and clients should really not have to care about their respective start-up order and should be able to recover from any order. Otherwise certain recovery scenarios become cumbersome and it introduces a lot of extra complexity in the deployment system. Need to prevent failure of rebalance when there are no brokers available when consumer comes up --- Key: KAFKA-969 URL: https://issues.apache.org/jira/browse/KAFKA-969 Project: Kafka Issue Type: Bug Reporter: Sriram Subramanian Assignee: Sriram Subramanian Fix For: 0.8 Attachments: emptybrokeronrebalance-1.patch There are some rare instances when a consumer would be up before bringing up the Kafka brokers. This would usually happen in a test scenario. In such conditions, during rebalance instead of failing the rebalance we just log the error and subscribe to broker changes. When the broker comes back up, we trigger the rebalance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-969) Need to prevent failure of rebalance when there are no brokers available when consumer comes up
[ https://issues.apache.org/jira/browse/KAFKA-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13705340#comment-13705340 ] Joel Koshy commented on KAFKA-969: -- This seems reasonable, but I'm not fully convinced about it. E.g., a test framework should ensure external dependencies are up before attempting to make service calls to those dependencies. That said, it is perhaps also reasonable from a consumer's perspective to expect that returned streams be empty at first, and whenever brokers and topics show up, then events should just show up. I'm +1 on this patch except for the if-else formatting issue. Also, I think this patch alone would be insufficient to meet the above. There are two other issues: - We should register a watcher under the topics path (currently done only if a wildcard is specified) - KAFKA-956 is also related. I need to give that one some thought. Need to prevent failure of rebalance when there are no brokers available when consumer comes up --- Key: KAFKA-969 URL: https://issues.apache.org/jira/browse/KAFKA-969 Project: Kafka Issue Type: Bug Reporter: Sriram Subramanian Assignee: Sriram Subramanian Attachments: emptybrokeronrebalance-1.patch There are some rare instances when a consumer would be up before bringing up the Kafka brokers. This would usually happen in a test scenario. In such conditions, during rebalance instead of failing the rebalance we just log the error and subscribe to broker changes. When the broker comes back up, we trigger the rebalance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-969) Need to prevent failure of rebalance when there are no brokers available when consumer comes up
[ https://issues.apache.org/jira/browse/KAFKA-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13705461#comment-13705461 ] Sriram Subramanian commented on KAFKA-969: -- The other issues you mention are separate from this. You should file JIRAs for those. Need to prevent failure of rebalance when there are no brokers available when consumer comes up --- Key: KAFKA-969 URL: https://issues.apache.org/jira/browse/KAFKA-969 Project: Kafka Issue Type: Bug Reporter: Sriram Subramanian Assignee: Sriram Subramanian Attachments: emptybrokeronrebalance-1.patch There are some rare instances when a consumer would be up before bringing up the Kafka brokers. This would usually happen in a test scenario. In such conditions, during rebalance instead of failing the rebalance we just log the error and subscribe to broker changes. When the broker comes back up, we trigger the rebalance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-969) Need to prevent failure of rebalance when there are no brokers available when consumer comes up
[ https://issues.apache.org/jira/browse/KAFKA-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13705484#comment-13705484 ] Guozhang Wang commented on KAFKA-969: - I think KAFKA-956 is orthogonal to this JIRA. It is about broker metadata not able to propagate to the brokers yet. Need to prevent failure of rebalance when there are no brokers available when consumer comes up --- Key: KAFKA-969 URL: https://issues.apache.org/jira/browse/KAFKA-969 Project: Kafka Issue Type: Bug Reporter: Sriram Subramanian Assignee: Sriram Subramanian Attachments: emptybrokeronrebalance-1.patch There are some rare instances when a consumer would be up before bringing up the Kafka brokers. This would usually happen in a test scenario. In such conditions, during rebalance instead of failing the rebalance we just log the error and subscribe to broker changes. When the broker comes back up, we trigger the rebalance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-969) Need to prevent failure of rebalance when there are no brokers available when consumer comes up
[ https://issues.apache.org/jira/browse/KAFKA-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13705490#comment-13705490 ] Joel Koshy commented on KAFKA-969: -- As I already said, I'm +1 on this patch for what it intends to address. Those two issues I mentioned are orthogonal. By above in my comment I was referring to the possible expectation from consumers: .. from a consumer's perspective to expect that returned streams be empty at first, and whenever brokers and topics show up, then events should just show up. - not the failed to rebalance issue. Need to prevent failure of rebalance when there are no brokers available when consumer comes up --- Key: KAFKA-969 URL: https://issues.apache.org/jira/browse/KAFKA-969 Project: Kafka Issue Type: Bug Reporter: Sriram Subramanian Assignee: Sriram Subramanian Fix For: 0.8 Attachments: emptybrokeronrebalance-1.patch There are some rare instances when a consumer would be up before bringing up the Kafka brokers. This would usually happen in a test scenario. In such conditions, during rebalance instead of failing the rebalance we just log the error and subscribe to broker changes. When the broker comes back up, we trigger the rebalance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
