iBlackeyes created KAFKA-12494:
----------------------------------
Summary: Broker raise InternalError after disk sector medium error
without marking dir to offline
Key: KAFKA-12494
URL: https://issues.apache.org/jira/browse/KAFKA-12494
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 2.7.0, 2.5.1, 2.6.0, 2.4.0, 1.1.2
Environment: Kafka Version: 1.1.0
Jdk Version: jdk1.8
Reporter: iBlackeyes
In my produce env, we encounter a case that kafka broker only raise errors like
`_*2021-02-16 23:24:24,965 | ERROR | [data-plane-kafka-request-handler-19] |
[ReplicaManager broker=7] Error processing append operation on partition
xxxxxxx-0 | kafka.server.ReplicaManager (Logging.scala:76)*_
_*java.lang.InternalError: a fault occurred in a recent unsafe memory access
operation in compiled Java code*_`
when broker append to a error disk sector and doesn't mark the dir on this
disk to offline.
This result in many partitions which assign replicas on this disk in
under-replicated state .
Here is the logs:
*os messages log:*
{code:java}
Feb 16 23:24:24 hd-node109 kernel: blk_update_request: critical medium error,
dev sds, sector 2308010408
Feb 16 23:24:24 hd-node109 kernel: sd 14:1:0:18: [sds] FAILED Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
Feb 16 23:24:24 hd-node109 kernel: sd 14:1:0:18: [sds] Sense Key : Medium Error
[current]
Feb 16 23:24:24 hd-node109 kernel: sd 14:1:0:18: [sds] Add. Sense: Unrecovered
read error
Feb 16 23:24:24 hd-node109 kernel: sd 14:1:0:18: [sds] CDB: Read(10) 28 00 89
91 71 a8 00 00 08 00
Feb 16 23:24:24 hd-node109 kernel: blk_update_request: critical medium error,
dev sds, sector 2308010408
Feb 16 23:24:24 hd-node109 kernel: sd 14:1:0:18: [sds] FAILED Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
Feb 16 23:24:24 hd-node109 kernel: sd 14:1:0:18: [sds] Sense Key : Medium Error
[current]
Feb 16 23:24:24 hd-node109 kernel: sd 14:1:0:18: [sds] Add. Sense: Unrecovered
read error
Feb 16 23:24:24 hd-node109 kernel: sd 14:1:0:18: [sds] CDB: Read(10) 28 00 89
91 71 a8 00 00 08 00
Feb 16 23:24:24 hd-node109 kernel: blk_update_request: critical medium error,
dev sds, sector 2308010408
Feb 16 23:24:24 hd-node109 kernel: sd 14:1:0:18: [sds] FAILED Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
Feb 16 23:24:24 hd-node109 kernel: sd 14:1:0:18: [sds] Sense Key : Medium Error
[current]
Feb 16 23:24:24 hd-node109 kernel: sd 14:1:0:18: [sds] Add. Sense: Unrecovered
read error
Feb 16 23:24:24 hd-node109 kernel: sd 14:1:0:18: [sds] CDB: Read(10) 28 00 89
91 71 a8 00 00 08 00
Feb 16 23:24:24 hd-node109 kernel: blk_update_request: critical medium error,
dev sds, sector 2308010408
Feb 16 23:24:24 hd-node109 kernel: sd 14:1:0:18: [sds] FAILED Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE
Feb 16 23:24:24 hd-node109 kernel: sd 14:1:0:18: [sds] Sense Key : Medium Error
[current]
Feb 16 23:24:24 hd-node109 kernel: sd 14:1:0:18: [sds] Add. Sense: Unrecovered
read error
Feb 16 23:24:24 hd-node109 kernel: sd 14:1:0:18: [sds] CDB: Read(10) 28 00 89
91 71 a8 00 00 08 00
Feb 16 23:24:24 hd-node109 kernel: blk_update_request: critical medium error,
dev sds, sector 2308010408{code}
*broker server.log:*
{code:java}
2021-02-16 23:24:24,965 | ERROR | [data-plane-kafka-request-handler-19] |
[ReplicaManager broker=7] Error processing append operation on xxxxxxxxx-0 |
kafka.server.ReplicaManager (Logging.scala:76) 2021-02-16 23:24:24,965 | ERROR
| [data-plane-kafka-request-handler-19] | [ReplicaManager broker=7] Error
processing append operation on xxxxxxxxx-0 | kafka.server.ReplicaManager
(Logging.scala:76) java.lang.InternalError: a fault occurred in a recent unsafe
memory access operation in compiled Java code at
java.util.zip.Inflater.<init>(Inflater.java:102) at
java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:77) at
org.apache.kafka.common.record.CompressionType$2.wrapForInput(CompressionType.java:69)
at
org.apache.kafka.common.record.DefaultRecordBatch.compressedIterator(DefaultRecordBatch.java:265)
at
org.apache.kafka.common.record.DefaultRecordBatch.iterator(DefaultRecordBatch.java:332)
at
scala.collection.convert.Wrappers$JIterableWrapper.iterator(Wrappers.scala:54)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at
scala.collection.AbstractIterable.foreach(Iterable.scala:54) at
kafka.log.LogValidator$$anonfun$validateMessagesAndAssignOffsetsCompressed$1.apply(LogValidator.scala:267)
at
kafka.log.LogValidator$$anonfun$validateMessagesAndAssignOffsetsCompressed$1.apply(LogValidator.scala:259)
at scala.collection.Iterator$class.foreach(Iterator.scala:891) at
scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at
scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at
scala.collection.AbstractIterable.foreach(Iterable.scala:54) at
kafka.log.LogValidator$.validateMessagesAndAssignOffsetsCompressed(LogValidator.scala:259)
at
kafka.log.LogValidator$.validateMessagesAndAssignOffsets(LogValidator.scala:70)
at kafka.log.Log$$anonfun$append$2.liftedTree1$1(Log.scala:672) at
kafka.log.Log$$anonfun$append$2.apply(Log.scala:671) at
kafka.log.Log$$anonfun$append$2.apply(Log.scala:653) at
kafka.log.Log.maybeHandleIOException(Log.scala:1711) at
kafka.log.Log.append(Log.scala:653) at
kafka.log.Log.appendAsLeader(Log.scala:623) at
kafka.cluster.Partition$$anonfun$13.apply(Partition.scala:609) at
kafka.cluster.Partition$$anonfun$13.apply(Partition.scala:597) at
kafka.utils.CoreUtils$.inLock(CoreUtils.scala:250) at
kafka.utils.CoreUtils$.inReadLock(CoreUtils.scala:256) at
kafka.cluster.Partition.appendRecordsToLeader(Partition.scala:596) at
kafka.server.ReplicaManager$$anonfun$appendToLocalLog$2.apply(ReplicaManager.scala:739)
at
kafka.server.ReplicaManager$$anonfun$appendToLocalLog$2.apply(ReplicaManager.scala:723)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) at
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236) at
scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) at
scala.collection.mutable.HashMap.foreach(HashMap.scala:130) at
scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at
scala.collection.AbstractTraversable.map(Traversable.scala:104) at
kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:723) at
kafka.server.ReplicaManager.appendRecords(ReplicaManager.scala:464) at
kafka.server.KafkaApis.handleProduceRequest(KafkaApis.scala:471) at
kafka.server.KafkaApis.handle(KafkaApis.scala:104) at
kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:69) at
java.lang.Thread.run(Thread.java:748)2021-02-16 23:24:24,999 | ERROR |
[data-plane-kafka-request-handler-19] | [ReplicaManager broker=7] Error
processing append operation on partition xxxxxxx-0 |
kafka.server.ReplicaManager (Logging.scala:76) java.lang.InternalError: a fault
occurred in a recent unsafe memory access operation in compiled Java code at
java.util.zip.Inflater.<init>(Inflater.java:102) at
java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:77) at
org.apache.kafka.common.record.CompressionType$2.wrapForInput(CompressionType.java:69)
at
org.apache.kafka.common.record.DefaultRecordBatch.compressedIterator(DefaultRecordBatch.java:265)
at
org.apache.kafka.common.record.DefaultRecordBatch.iterator(DefaultRecordBatch.java:332)
at
scala.collection.convert.Wrappers$JIterableWrapper.iterator(Wrappers.scala:54)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at
scala.collection.AbstractIterable.foreach(Iterable.scala:54) at
kafka.log.LogValidator$$anonfun$validateMessagesAndAssignOffsetsCompressed$1.apply(LogValidator.scala:267)
at
kafka.log.LogValidator$$anonfun$validateMessagesAndAssignOffsetsCompressed$1.apply(LogValidator.scala:259)
at scala.collection.Iterator$class.foreach(Iterator.scala:891) at
scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at
scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at
scala.collection.AbstractIterable.foreach(Iterable.scala:54) at
kafka.log.LogValidator$.validateMessagesAndAssignOffsetsCompressed(LogValidator.scala:259)
at
kafka.log.LogValidator$.validateMessagesAndAssignOffsets(LogValidator.scala:70)
at kafka.log.Log$$anonfun$append$2.liftedTree1$1(Log.scala:672) at
kafka.log.Log$$anonfun$append$2.apply(Log.scala:671) at
kafka.log.Log$$anonfun$append$2.apply(Log.scala:653) at
kafka.log.Log.maybeHandleIOException(Log.scala:1711) at
kafka.log.Log.append(Log.scala:653) at
kafka.log.Log.appendAsLeader(Log.scala:623) at
kafka.cluster.Partition$$anonfun$13.apply(Partition.scala:609) at
kafka.cluster.Partition$$anonfun$13.apply(Partition.scala:597) at
kafka.utils.CoreUtils$.inLock(CoreUtils.scala:250) at
kafka.utils.CoreUtils$.inReadLock(CoreUtils.scala:256) at
kafka.cluster.Partition.appendRecordsToLeader(Partition.scala:596) at
kafka.server.ReplicaManager$$anonfun$appendToLocalLog$2.apply(ReplicaManager.scala:739)
at
kafka.server.ReplicaManager$$anonfun$appendToLocalLog$2.apply(ReplicaManager.scala:723)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) at
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236) at
scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) at
scala.collection.mutable.HashMap.foreach(HashMap.scala:130) at
scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at
scala.collection.AbstractTraversable.map(Traversable.scala:104) at
kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:723) at
kafka.server.ReplicaManager.appendRecords(ReplicaManager.scala:464) at
kafka.server.KafkaApis.handleProduceRequest(KafkaApis.scala:471) at
kafka.server.KafkaApis.handle(KafkaApis.scala:104) at
kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:69) at
java.lang.Thread.run(Thread.java:748)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
