Hi Team,
I am using kafka streams in my application and I am running in cloud with 5
kafka instances. I am getting below exception in kafka machines and after some
point my application stop working because of
org.apache.kafka.common.errors.TimeoutException. Please help me on this as we
couldn’t move further.
server.properties(kafka performance configuration):
num.network.threads=8
socket.receive.buffer.bytes=-1
socket.send.buffer.bytes=-1
socket.request.max.bytes=2147483647
Kafka Exception:
[2019-06-29 21:17:57,649] INFO [Log
partition=location-analytics-live-v2-visit-processing-5,
dir=/mnt/data/kafka/kafka-logs] Incrementing log start offset to 263555323
(kafka.log.Log)
[2019-06-29 21:18:18,992] WARN [SocketServer brokerId=0] Unexpected error from
/172.21.4.208; closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size
= -1145372416)
at
org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:102)
at
org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:381)
at
org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:342)
at
org.apache.kafka.common.network.Selector.attemptRead(Selector.java:609)
at
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:541)
at org.apache.kafka.common.network.Selector.poll(Selector.java:467)
at kafka.network.Processor.poll(SocketServer.scala:689)
at kafka.network.Processor.run(SocketServer.scala:594)
at java.lang.Thread.run(Thread.java:748)
[2019-06-29 21:18:18,993] WARN [SocketServer brokerId=0] Unexpected error from
/172.21.4.208; closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size
= -1145372671)
at
org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:102)
at
org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:381)
at
org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:342)
at
org.apache.kafka.common.network.Selector.attemptRead(Selector.java:609)
at
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:541)
at org.apache.kafka.common.network.Selector.poll(Selector.java:467)
at kafka.network.Processor.poll(SocketServer.scala:689)
at kafka.network.Processor.run(SocketServer.scala:594)
at java.lang.Thread.run(Thread.java:748)
[2019-06-29 21:18:57,649] INFO [Log
partition=location-analytics-live-v2-visit-processing-5,
dir=/mnt/data/kafka/kafka-logs] Deleting segment 256880856 (kafka.log.Log)
[2019-06-29 21:18:57,676] INFO Deleted log
/mnt/data/kafka/kafka-logs/location-analytics-live-v2-visit-processing-5/00000000000256880856.log.deleted.
(kafka.log.LogSegment)
[2019-06-29 21:18:57,676] INFO Deleted offset index
/mnt/data/kafka/kafka-logs/location-analytics-live-v2-visit-processing-5/00000000000256880856.index.deleted.
(kafka.log.LogSegment)
[2019-06-29 21:18:57,676] INFO Deleted time index
/mnt/data/kafka/kafka-logs/location-analytics-live-v2-visit-processing-5/00000000000256880856.timeindex.deleted.
(kafka.log.LogSegment)
[2019-06-29 21:19:19,034] ERROR Closing socket for
172.21.46.208:9092-172.21.4.208:36798-2446 because of error
(kafka.network.Processor)
org.apache.kafka.common.errors.InvalidRequestException: Error parsing request
header. Our best guess of the apiKey is: 1032
[2019-06-29 21:19:19,034] ERROR Closing socket for
172.21.46.208:9092-172.21.4.208:36798-2446 because of error
(kafka.network.Processor)
org.apache.kafka.common.errors.InvalidRequestException: Error parsing request
header. Our best guess of the apiKey is: 1032
Caused by: java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:506)
at java.nio.HeapByteBuffer.getShort(HeapByteBuffer.java:310)
at
org.apache.kafka.common.requests.RequestHeader.parse(RequestHeader.java:118)
at
kafka.network.Processor.$anonfun$processCompletedReceives$1(SocketServer.scala:703)
at
kafka.network.Processor.$anonfun$processCompletedReceives$1$adapted(SocketServer.scala:699)
at scala.collection.Iterator.foreach(Iterator.scala:937)
at scala.collection.Iterator.foreach$(Iterator.scala:937)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
at scala.collection.IterableLike.foreach(IterableLike.scala:70)
at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at
kafka.network.Processor.processCompletedReceives(SocketServer.scala:699)
at kafka.network.Processor.run(SocketServer.scala:595)
at java.lang.Thread.run(Thread.java:748)
[2019-06-29 21:19:19,039] ERROR Exception while processing request from
172.21.46.208:9092-172.21.4.208:36798-2446 (kafka.network.Processor)
org.apache.kafka.common.errors.InvalidRequestException: Error parsing request
header. Our best guess of the apiKey is: 1032
Caused by: java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:506)
at java.nio.HeapByteBuffer.getShort(HeapByteBuffer.java:310)
at
org.apache.kafka.common.requests.RequestHeader.parse(RequestHeader.java:118)
at
kafka.network.Processor.$anonfun$processCompletedReceives$1(SocketServer.scala:703)
at
kafka.network.Processor.$anonfun$processCompletedReceives$1$adapted(SocketServer.scala:699)
at scala.collection.Iterator.foreach(Iterator.scala:937)
at scala.collection.Iterator.foreach$(Iterator.scala:937)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
at scala.collection.IterableLike.foreach(IterableLike.scala:70)
at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at
kafka.network.Processor.processCompletedReceives(SocketServer.scala:699)
at kafka.network.Processor.run(SocketServer.scala:595)
at java.lang.Thread.run(Thread.java:748)
org.apache.kafka.common.errors.InvalidRequestException: Error parsing request
header. Our best guess of the apiKey is: -32767
Caused by: org.apache.kafka.common.protocol.types.SchemaException: Error
reading field 'client_id': Error reading string of length 27759, only 73 bytes
available
at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:77)
at
org.apache.kafka.common.requests.RequestHeader.parse(RequestHeader.java:121)
at
kafka.network.Processor.$anonfun$processCompletedReceives$1(SocketServer.scala:703)
at
kafka.network.Processor.$anonfun$processCompletedReceives$1$adapted(SocketServer.scala:699)
at scala.collection.Iterator.foreach(Iterator.scala:937)
at scala.collection.Iterator.foreach$(Iterator.scala:937)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
at scala.collection.IterableLike.foreach(IterableLike.scala:70)
at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at
kafka.network.Processor.processCompletedReceives(SocketServer.scala:699)
at kafka.network.Processor.run(SocketServer.scala:595)
at java.lang.Thread.run(Thread.java:748)
[2019-06-29 21:19:43,050] ERROR Exception while processing request from
172.21.46.208:9092-172.21.4.208:38368-2446 (kafka.network.Processor)
org.apache.kafka.common.errors.InvalidRequestException: Error parsing request
header. Our best guess of the apiKey is: -32767
Caused by: org.apache.kafka.common.protocol.types.SchemaException: Error
reading field 'client_id': Error reading string of length 27759, only 73 bytes
available
at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:77)
at
org.apache.kafka.common.requests.RequestHeader.parse(RequestHeader.java:121)
at
kafka.network.Processor.$anonfun$processCompletedReceives$1(SocketServer.scala:703)
at
kafka.network.Processor.$anonfun$processCompletedReceives$1$adapted(SocketServer.scala:699)
at scala.collection.Iterator.foreach(Iterator.scala:937)
at scala.collection.Iterator.foreach$(Iterator.scala:937)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
at scala.collection.IterableLike.foreach(IterableLike.scala:70)
at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at
kafka.network.Processor.processCompletedReceives(SocketServer.scala:699)
at kafka.network.Processor.run(SocketServer.scala:595)
at java.lang.Thread.run(Thread.java:748)
[2019-06-29 21:19:43,229] ERROR Closing socket for
172.21.46.208:9092-172.21.4.208:38370-2447 because of error
(kafka.network.Processor)
java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkIndex(Buffer.java:546)
at java.nio.HeapByteBuffer.getShort(HeapByteBuffer.java:314)
at
org.apache.kafka.common.requests.RequestHeader.parse(RequestHeader.java:126)
at
kafka.network.Processor.$anonfun$processCompletedReceives$1(SocketServer.scala:703)
at
kafka.network.Processor.$anonfun$processCompletedReceives$1$adapted(SocketServer.scala:699)
at scala.collection.Iterator.foreach(Iterator.scala:937)
at scala.collection.Iterator.foreach$(Iterator.scala:937)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
at scala.collection.IterableLike.foreach(IterableLike.scala:70)
at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at
kafka.network.Processor.processCompletedReceives(SocketServer.scala:699)
at kafka.network.Processor.run(SocketServer.scala:595)
at java.lang.Thread.run(Thread.java:748)
[2019-06-29 21:19:43,443] ERROR Exception while processing request from
172.21.46.208:9092-172.21.4.208:38370-2447 (kafka.network.Processor)
java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkIndex(Buffer.java:546)
at java.nio.HeapByteBuffer.getShort(HeapByteBuffer.java:314)
at
org.apache.kafka.common.requests.RequestHeader.parse(RequestHeader.java:126)
at
kafka.network.Processor.$anonfun$processCompletedReceives$1(SocketServer.scala:703)
at
kafka.network.Processor.$anonfun$processCompletedReceives$1$adapted(SocketServer.scala:699)
at scala.collection.Iterator.foreach(Iterator.scala:937)
at scala.collection.Iterator.foreach$(Iterator.scala:937)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
at scala.collection.IterableLike.foreach(IterableLike.scala:70)
at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at
kafka.network.Processor.processCompletedReceives(SocketServer.scala:699)
at kafka.network.Processor.run(SocketServer.scala:595)
at java.lang.Thread.run(Thread.java:748)
Thanks,
Vigneswaran
