Roger, could you paste the full log from Samza container? If you can figure out which Kafka broker the message was sent to, it would be helpful if we get the log from the broker as well.
On Tue, Apr 28, 2015 at 3:31 PM, Roger Hoover <roger.hoo...@gmail.com> wrote: > Hi, > > I need some help figuring out what's going on. > > I'm running Kafka 0.8.2.1 and Samza 0.9.0 on YARN. All the topics have > replication factor of 2. > > I'm bouncing the Kafka broker using SIGTERM (with > controlled.shutdown.enable=true). I see the Samza job log this message and > then hang (does not exit nor does it make any progress). > > 2015-04-28 14:28:25 KafkaSystemProducer [ERROR] task[Partition 2] > ssp[kafka,my-topic,2] offset[9129395] Unable to send message from > TaskName-Partition 1 to system kafka > > The Kafka consumer (Druid Real-Time node) on the other side then barfs on > the message: > > Exception in thread "chief-svc-perf" kafka.message.InvalidMessageException: > Message is corrupt (stored crc = 1792882425, computed crc = 3898271689) > at kafka.message.Message.ensureValid(Message.scala:166) > at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:101) > at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33) > at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66) > at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58) > at > > io.druid.firehose.kafka.KafkaEightFirehoseFactory$1.hasMore(KafkaEightFirehoseFactory.java:106) > at > > io.druid.segment.realtime.RealtimeManager$FireChief.run(RealtimeManager.java:234) > > My questions are: > 1) What is the right way to bounce a Kafka broker? > 2) Is this a bug in Samza that the job hangs after producer request fails? > Has anyone seen this? > > Thanks, > > Roger >
