Hi there, Is anyone able to confirm what is the expected behaviour of using the Kafka output in case of an unlikely multi-broker outage?
I have been doing some tests against my pipeline (heka TcpInput -> KafkaOutput -> Broker) all running on the same machine and I noticed the following behavior: Test 1 - Broker is dead when heka starts: Error making runner for TestKafkaOutput: Initialization failed for 'TestKafkaOutput': kafka: client has run out of available brokers to talk to (Is your cluster reachable?) Test 2 - All brokers (1 in this case) go offline (healthy shutdown) during message streaming 2015/09/25 23:12:12 Plugin 'TestKafkaOutput' error: kafka error 'kafka: client has run out of available brokers to talk to (Is your cluster reachable?)' for message '<38>2015-09-25T22:39:36 localhost localprg[1234]: seq: 0000670524, runid: 1443184763, ' 2015/09/25 23:12:13 Plugin 'TestKafkaOutput' error: kafka error 'kafka: client has run out of available brokers to talk to (Is your cluster reachable?)' for message '<38>2015-09-25T22:39:36 localhost localprg[1234]: seq: 0000670525, runid: 1443184763, ' 2015/09/25 23:12:14 Plugin 'TestKafkaOutput' error: kafka error 'kafka: client has run out of available brokers to talk to (Is your cluster reachable?)' for message '<38>2015-09-25T22:39:36 localhost localprg[1234]: seq: 0000670526, runid: 1443184763 ' 2015/09/25 23:12:14 Plugin 'TestKafkaOutput' error: kafka error 'circuit breaker is open' for message '<38>2015-09-25T22:39:36 localhost localprg[1234]: seq: 0000670527, runid: 1443184763 I was surprised by this one. After reading the documentation I would assume that required_acks default WaitForLocal would prevent this from happening? Relevant TOML section is: [TestKafkaOutput] type = "KafkaOutput" message_matcher = "Type == 'TcpInput'" topic = "test_topic" addrs = ["localhost:9092"] encoder = "PayloadEncoder" I thank you in advance _______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
