Hi, I need some help figuring out what's going on.
I'm running Kafka 0.8.2.1 and Samza 0.9.0 on YARN. All the topics have replication factor of 2. I'm bouncing the Kafka broker using SIGTERM (with controlled.shutdown.enable=true). I see the Samza job log this message and then hang (does not exit nor does it make any progress). 2015-04-28 14:28:25 KafkaSystemProducer [ERROR] task[Partition 2] ssp[kafka,my-topic,2] offset[9129395] Unable to send message from TaskName-Partition 1 to system kafka The Kafka consumer (Druid Real-Time node) on the other side then barfs on the message: Exception in thread "chief-svc-perf" kafka.message.InvalidMessageException: Message is corrupt (stored crc = 1792882425, computed crc = 3898271689) at kafka.message.Message.ensureValid(Message.scala:166) at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:101) at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33) at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66) at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58) at io.druid.firehose.kafka.KafkaEightFirehoseFactory$1.hasMore(KafkaEightFirehoseFactory.java:106) at io.druid.segment.realtime.RealtimeManager$FireChief.run(RealtimeManager.java:234) My questions are: 1) What is the right way to bounce a Kafka broker? 2) Is this a bug in Samza that the job hangs after producer request fails? Has anyone seen this? Thanks, Roger
