Finally, was compression enabled when you hit this exception? If so, which compression algorithm was enabled?
On Thu, Jun 15, 2017 at 5:04 PM, Apurva Mehta <apu...@confluent.io> wrote: > Frank: it would be even better if you could share the key and value which > was causing this problem. Maybe share it on the JIRA: > https://issues.apache.org/jira/browse/KAFKA-5456 ? > > Thanks, > Apurva > > On Thu, Jun 15, 2017 at 4:07 PM, Apurva Mehta <apu...@confluent.io> wrote: > >> Hi Frank, >> >> What is is the value of `batch.size` in your producer? What is the size >> of the key and value you are trying to write? >> >> Thanks, >> Apurva >> >> On Thu, Jun 15, 2017 at 2:28 AM, Frank Lyaruu <flya...@gmail.com> wrote: >> >>> Hey people, I see an error I haven't seen before. It is on a lowlevel-API >>> based streams application. I've started it once, then it ran fine, then >>> did >>> a graceful shutdown and since then I always see this error on startup. >>> >>> I'm using yesterday's trunk. >>> >>> It seems that the MemoryRecordsBuilder overflows somehow, is there >>> something I need to configure? >>> >>> java.lang.NullPointerException >>> >>> at org.apache.kafka.common.utils.Utils.notNull(Utils.java:243) >>> at >>> org.apache.kafka.clients.producer.internals.RecordAccumulato >>> r.append(RecordAccumulator.java:219) >>> at >>> org.apache.kafka.clients.producer.KafkaProducer.doSend(Kafka >>> Producer.java:650) >>> at >>> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaPr >>> oducer.java:604) >>> at >>> org.apache.kafka.streams.processor.internals.RecordCollector >>> Impl.send(RecordCollectorImpl.java:97) >>> at >>> org.apache.kafka.streams.state.internals.StoreChangeLogger.l >>> ogChange(StoreChangeLogger.java:59) >>> at >>> org.apache.kafka.streams.state.internals.ChangeLoggingKeyVal >>> ueBytesStore.put(ChangeLoggingKeyValueBytesStore.java:58) >>> at >>> org.apache.kafka.streams.state.internals.ChangeLoggingKeyVal >>> ueStore.put(ChangeLoggingKeyValueStore.java:73) >>> at >>> org.apache.kafka.streams.state.internals.MeteredKeyValueStor >>> e$2.run(MeteredKeyValueStore.java:66) >>> at >>> org.apache.kafka.streams.processor.internals.StreamsMetricsI >>> mpl.measureLatencyNs(StreamsMetricsImpl.java:187) >>> at >>> org.apache.kafka.streams.state.internals.MeteredKeyValueStor >>> e.put(MeteredKeyValueStore.java:149) >>> at >>> com.dexels.kafka.streams.remotejoin.StoreProcessor.process(S >>> toreProcessor.java:47) >>> at >>> com.dexels.kafka.streams.remotejoin.StoreProcessor.process(S >>> toreProcessor.java:1) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode$1 >>> .run(ProcessorNode.java:47) >>> at >>> org.apache.kafka.streams.processor.internals.StreamsMetricsI >>> mpl.measureLatencyNs(StreamsMetricsImpl.java:187) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode.p >>> rocess(ProcessorNode.java:133) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorContex >>> tImpl.forward(ProcessorContextImpl.java:82) >>> at >>> com.dexels.kafka.streams.remotejoin.ranged.OneToManyGroupedP >>> rocessor.emitMessage(OneToManyGroupedProcessor.java:95) >>> at >>> com.dexels.kafka.streams.remotejoin.ranged.OneToManyGroupedP >>> rocessor.process(OneToManyGroupedProcessor.java:80) >>> at >>> com.dexels.kafka.streams.remotejoin.ranged.OneToManyGroupedP >>> rocessor.process(OneToManyGroupedProcessor.java:1) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode$1 >>> .run(ProcessorNode.java:47) >>> at >>> org.apache.kafka.streams.processor.internals.StreamsMetricsI >>> mpl.measureLatencyNs(StreamsMetricsImpl.java:187) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode.p >>> rocess(ProcessorNode.java:133) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorContex >>> tImpl.forward(ProcessorContextImpl.java:82) >>> at >>> com.dexels.kafka.streams.remotejoin.StoreProcessor.process(S >>> toreProcessor.java:48) >>> at >>> com.dexels.kafka.streams.remotejoin.StoreProcessor.process(S >>> toreProcessor.java:1) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode$1 >>> .run(ProcessorNode.java:47) >>> at >>> org.apache.kafka.streams.processor.internals.StreamsMetricsI >>> mpl.measureLatencyNs(StreamsMetricsImpl.java:187) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode.p >>> rocess(ProcessorNode.java:133) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorContex >>> tImpl.forward(ProcessorContextImpl.java:82) >>> at >>> com.dexels.kafka.streams.remotejoin.ranged.OneToManyGroupedP >>> rocessor.emitMessage(OneToManyGroupedProcessor.java:95) >>> at >>> com.dexels.kafka.streams.remotejoin.ranged.OneToManyGroupedP >>> rocessor.process(OneToManyGroupedProcessor.java:80) >>> at >>> com.dexels.kafka.streams.remotejoin.ranged.OneToManyGroupedP >>> rocessor.process(OneToManyGroupedProcessor.java:1) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode$1 >>> .run(ProcessorNode.java:47) >>> at >>> org.apache.kafka.streams.processor.internals.StreamsMetricsI >>> mpl.measureLatencyNs(StreamsMetricsImpl.java:187) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode.p >>> rocess(ProcessorNode.java:133) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorContex >>> tImpl.forward(ProcessorContextImpl.java:82) >>> at >>> com.dexels.kafka.streams.remotejoin.StoreProcessor.process(S >>> toreProcessor.java:48) >>> at >>> com.dexels.kafka.streams.remotejoin.StoreProcessor.process(S >>> toreProcessor.java:1) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode$1 >>> .run(ProcessorNode.java:47) >>> at >>> org.apache.kafka.streams.processor.internals.StreamsMetricsI >>> mpl.measureLatencyNs(StreamsMetricsImpl.java:187) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode.p >>> rocess(ProcessorNode.java:133) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorContex >>> tImpl.forward(ProcessorContextImpl.java:82) >>> at >>> com.dexels.kafka.streams.remotejoin.ranged.OneToManyGroupedP >>> rocessor.emitMessage(OneToManyGroupedProcessor.java:95) >>> at >>> com.dexels.kafka.streams.remotejoin.ranged.OneToManyGroupedP >>> rocessor.process(OneToManyGroupedProcessor.java:80) >>> at >>> com.dexels.kafka.streams.remotejoin.ranged.OneToManyGroupedP >>> rocessor.process(OneToManyGroupedProcessor.java:1) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode$1 >>> .run(ProcessorNode.java:47) >>> at >>> org.apache.kafka.streams.processor.internals.StreamsMetricsI >>> mpl.measureLatencyNs(StreamsMetricsImpl.java:187) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode.p >>> rocess(ProcessorNode.java:133) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorContex >>> tImpl.forward(ProcessorContextImpl.java:82) >>> at >>> com.dexels.kafka.streams.remotejoin.StoreProcessor.process(S >>> toreProcessor.java:48) >>> at >>> com.dexels.kafka.streams.remotejoin.StoreProcessor.process(S >>> toreProcessor.java:1) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode$1 >>> .run(ProcessorNode.java:47) >>> at >>> org.apache.kafka.streams.processor.internals.StreamsMetricsI >>> mpl.measureLatencyNs(StreamsMetricsImpl.java:187) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode.p >>> rocess(ProcessorNode.java:133) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorContex >>> tImpl.forward(ProcessorContextImpl.java:82) >>> at >>> com.dexels.kafka.streams.remotejoin.OneToOneProcessor.proces >>> s(OneToOneProcessor.java:64) >>> at >>> com.dexels.kafka.streams.remotejoin.OneToOneProcessor.proces >>> s(OneToOneProcessor.java:1) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode$1 >>> .run(ProcessorNode.java:47) >>> at >>> org.apache.kafka.streams.processor.internals.StreamsMetricsI >>> mpl.measureLatencyNs(StreamsMetricsImpl.java:187) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode.p >>> rocess(ProcessorNode.java:133) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorContex >>> tImpl.forward(ProcessorContextImpl.java:82) >>> at >>> com.dexels.kafka.streams.remotejoin.StoreProcessor.process(S >>> toreProcessor.java:48) >>> at >>> com.dexels.kafka.streams.remotejoin.StoreProcessor.process(S >>> toreProcessor.java:1) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode$1 >>> .run(ProcessorNode.java:47) >>> at >>> org.apache.kafka.streams.processor.internals.StreamsMetricsI >>> mpl.measureLatencyNs(StreamsMetricsImpl.java:187) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode.p >>> rocess(ProcessorNode.java:133) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorContex >>> tImpl.forward(ProcessorContextImpl.java:82) >>> at >>> com.dexels.kafka.streams.remotejoin.ranged.OneToManyGroupedP >>> rocessor.emitMessage(OneToManyGroupedProcessor.java:95) >>> at >>> com.dexels.kafka.streams.remotejoin.ranged.OneToManyGroupedP >>> rocessor.process(OneToManyGroupedProcessor.java:80) >>> at >>> com.dexels.kafka.streams.remotejoin.ranged.OneToManyGroupedP >>> rocessor.process(OneToManyGroupedProcessor.java:1) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode$1 >>> .run(ProcessorNode.java:47) >>> at >>> org.apache.kafka.streams.processor.internals.StreamsMetricsI >>> mpl.measureLatencyNs(StreamsMetricsImpl.java:187) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode.p >>> rocess(ProcessorNode.java:133) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorContex >>> tImpl.forward(ProcessorContextImpl.java:82) >>> at >>> com.dexels.kafka.streams.remotejoin.StoreProcessor.process(S >>> toreProcessor.java:48) >>> at >>> com.dexels.kafka.streams.remotejoin.StoreProcessor.process(S >>> toreProcessor.java:1) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode$1 >>> .run(ProcessorNode.java:47) >>> at >>> org.apache.kafka.streams.processor.internals.StreamsMetricsI >>> mpl.measureLatencyNs(StreamsMetricsImpl.java:187) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode.p >>> rocess(ProcessorNode.java:133) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorContex >>> tImpl.forward(ProcessorContextImpl.java:82) >>> at >>> com.dexels.kafka.streams.remotejoin.ranged.OneToManyGroupedP >>> rocessor.emitMessage(OneToManyGroupedProcessor.java:95) >>> at >>> com.dexels.kafka.streams.remotejoin.ranged.OneToManyGroupedP >>> rocessor.process(OneToManyGroupedProcessor.java:80) >>> at >>> com.dexels.kafka.streams.remotejoin.ranged.OneToManyGroupedP >>> rocessor.process(OneToManyGroupedProcessor.java:1) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode$1 >>> .run(ProcessorNode.java:47) >>> at >>> org.apache.kafka.streams.processor.internals.StreamsMetricsI >>> mpl.measureLatencyNs(StreamsMetricsImpl.java:187) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode.p >>> rocess(ProcessorNode.java:133) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorContex >>> tImpl.forward(ProcessorContextImpl.java:82) >>> at >>> com.dexels.kafka.streams.remotejoin.StoreProcessor.process(S >>> toreProcessor.java:48) >>> at >>> com.dexels.kafka.streams.remotejoin.StoreProcessor.process(S >>> toreProcessor.java:1) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode$1 >>> .run(ProcessorNode.java:47) >>> at >>> org.apache.kafka.streams.processor.internals.StreamsMetricsI >>> mpl.measureLatencyNs(StreamsMetricsImpl.java:187) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode.p >>> rocess(ProcessorNode.java:133) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorContex >>> tImpl.forward(ProcessorContextImpl.java:82) >>> at >>> com.dexels.kafka.streams.remotejoin.ranged.OneToManyGroupedP >>> rocessor.emitMessage(OneToManyGroupedProcessor.java:95) >>> at >>> com.dexels.kafka.streams.remotejoin.ranged.OneToManyGroupedP >>> rocessor.process(OneToManyGroupedProcessor.java:80) >>> at >>> com.dexels.kafka.streams.remotejoin.ranged.OneToManyGroupedP >>> rocessor.process(OneToManyGroupedProcessor.java:1) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode$1 >>> .run(ProcessorNode.java:47) >>> at >>> org.apache.kafka.streams.processor.internals.StreamsMetricsI >>> mpl.measureLatencyNs(StreamsMetricsImpl.java:187) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode.p >>> rocess(ProcessorNode.java:133) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorContex >>> tImpl.forward(ProcessorContextImpl.java:82) >>> at >>> com.dexels.kafka.streams.remotejoin.StoreProcessor.process(S >>> toreProcessor.java:48) >>> at >>> com.dexels.kafka.streams.remotejoin.StoreProcessor.process(S >>> toreProcessor.java:1) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode$1 >>> .run(ProcessorNode.java:47) >>> at >>> org.apache.kafka.streams.processor.internals.StreamsMetricsI >>> mpl.measureLatencyNs(StreamsMetricsImpl.java:187) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode.p >>> rocess(ProcessorNode.java:133) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorContex >>> tImpl.forward(ProcessorContextImpl.java:82) >>> at >>> com.dexels.kafka.streams.remotejoin.XmlTransformerProcessor. >>> process(XmlTransformerProcessor.java:51) >>> at >>> com.dexels.kafka.streams.remotejoin.XmlTransformerProcessor. >>> process(XmlTransformerProcessor.java:1) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode$1 >>> .run(ProcessorNode.java:47) >>> at >>> org.apache.kafka.streams.processor.internals.StreamsMetricsI >>> mpl.measureLatencyNs(StreamsMetricsImpl.java:187) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorNode.p >>> rocess(ProcessorNode.java:133) >>> at >>> org.apache.kafka.streams.processor.internals.ProcessorContex >>> tImpl.forward(ProcessorContextImpl.java:82) >>> at >>> org.apache.kafka.streams.processor.internals.SourceNode.proc >>> ess(SourceNode.java:80) >>> at >>> org.apache.kafka.streams.processor.internals.StreamTask.proc >>> ess(StreamTask.java:189) >>> at >>> org.apache.kafka.streams.processor.internals.StreamThread.pr >>> ocessAndPunctuate(StreamThread.java:677) >>> at >>> org.apache.kafka.streams.processor.internals.StreamThread.ru >>> nLoop(StreamThread.java:555) >>> at >>> org.apache.kafka.streams.processor.internals.StreamThread.ru >>> n(StreamThread.java:525) >>> >> >> >
