All, Just to clarify (as my previous message didn't mention this):
During test 2 messages get lost in transit once the broker is shutdown and despite the very low volume of messages 5 messages per second over 25 seconds. Cheers On Fri, Sep 25, 2015 at 11:49 PM, Andre <[email protected]> wrote: > Hi there, > > Is anyone able to confirm what is the expected behaviour of using the > Kafka output in case of an unlikely multi-broker outage? > > I have been doing some tests against my pipeline (heka TcpInput -> > KafkaOutput -> Broker) all running on the same machine and I noticed > the following behavior: > > Test 1 - Broker is dead when heka starts: > > Error making runner for TestKafkaOutput: Initialization failed for > 'TestKafkaOutput': kafka: client has run out of available brokers to > talk to (Is your cluster reachable?) > > > Test 2 - All brokers (1 in this case) go offline (healthy shutdown) > during message streaming > > > 2015/09/25 23:12:12 Plugin 'TestKafkaOutput' error: kafka error > 'kafka: client has run out of available brokers to talk to (Is your > cluster reachable?)' for message '<38>2015-09-25T22:39:36 localhost > localprg[1234]: seq: 0000670524, runid: 1443184763, > ' > 2015/09/25 23:12:13 Plugin 'TestKafkaOutput' error: kafka error > 'kafka: client has run out of available brokers to talk to (Is your > cluster reachable?)' for message '<38>2015-09-25T22:39:36 localhost > localprg[1234]: seq: 0000670525, runid: 1443184763, > ' > 2015/09/25 23:12:14 Plugin 'TestKafkaOutput' error: kafka error > 'kafka: client has run out of available brokers to talk to (Is your > cluster reachable?)' for message '<38>2015-09-25T22:39:36 localhost > localprg[1234]: seq: 0000670526, runid: 1443184763 > ' > 2015/09/25 23:12:14 Plugin 'TestKafkaOutput' error: kafka error > 'circuit breaker is open' for message '<38>2015-09-25T22:39:36 > localhost localprg[1234]: seq: 0000670527, runid: 1443184763 > > > I was surprised by this one. After reading the documentation I would > assume that required_acks default WaitForLocal would prevent this from > happening? > > Relevant TOML section is: > > [TestKafkaOutput] > type = "KafkaOutput" > message_matcher = "Type == 'TcpInput'" > topic = "test_topic" > addrs = ["localhost:9092"] > encoder = "PayloadEncoder" > > > > > I thank you in advance _______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
