D'oh. Bad config on our part. Something we thought we had fixed long ago, but it crept back in.
Make sure fetch sizes are big enough! Philp On Oct 31, 2013, at 7:18 PM, "Philip O'Toole" <[email protected]> wrote: > We suddenly started seeing these messages from one our consumers tonight. > What do they mean? Our high-level consumers -- most of them -- are not moving. > > What has happened? > > Philip > > 2013-11-01 01:58:09,033 [ERROR] [FetcherRunnable.error] error in > FetcherRunnable for pro > cessed:6-6: fetched offset = 403315202733: consumed offset = 403315202733 > kafka.common.InvalidMessageSizeException: invalid message size: 1129552 only > received by > tes: 1048572 at 403315202733( possible causes (1) a single message larger > than the fetch > size; (2) log corruption ) > at > kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet > .scala:120) > at > kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scal > a:160) > at > kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scal > a:100) > at > kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59) > at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51) > at > kafka.message.ByteBufferMessageSet.shallowValidBytes(ByteBufferMessageSet.sca > la:64) > at > kafka.message.ByteBufferMessageSet.validBytes(ByteBufferMessageSet.scala:59) > at > kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:57) > at > kafka.consumer.FetcherRunnable$$anonfun$run$6.apply(FetcherRunnable.scala:79) > at > kafka.consumer.FetcherRunnable$$anonfun$run$6.apply(FetcherRunnable.scala:65) > at > scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59 > ) > at scala.collection.immutable.List.foreach(List.scala:45) > at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:65) > 2013-11-01 01:58:09,033 [DEBUG] [PartitionTopicInfo.debug] reset consume > offset of proce > ssed:6-7: fetched offset = 404145188795: consumed offset = 404143227662 to > 404143227662 > 2013-11-01 01:58:09,034 [ERROR] [FetcherRunnable.error] error in > FetcherRunnable > kafka.common.InvalidMessageSizeException: invalid message size: 1129552 only > received by > tes: 1048572 at 403315202733( possible causes (1) a single message larger > than the fetch > size; (2) log corruption ) > at > kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet > .scala:120) > at > kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scal > a:160) > at > kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scal > a:100) > at > kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59) > at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51) > at > kafka.message.ByteBufferMessageSet.shallowValidBytes(ByteBufferMessageSet.sca > la:64) > at > kafka.message.ByteBufferMessageSet.validBytes(ByteBufferMessageSet.scala:59) > at > kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:57) > at > kafka.consumer.FetcherRunnable$$anonfun$run$6.apply(FetcherRunnable.scala:79) > at > kafka.consumer.FetcherRunnable$$anonfun$run$6.apply(FetcherRunnable.scala:65) > at > scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59 > ) > at scala.collection.immutable.List.foreach(List.scala:45) > at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:65)
