[ https://issues.apache.org/jira/browse/KAFKA-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16580861#comment-16580861 ]
Alastair Munro commented on KAFKA-7282: --------------------------------------- I think this might be related to glusterfs; the underlying storage system. We use kubernetes (actually openshift) for the kafka cluster, and we have multiple instances of them. I see this issue on glusterfs based systems, but not on nfs based systems. The thing about glusterfs is there are 3 replicas for the storage, and each request could be hitting different replicas, which may not be insync (yet). We see a similar issue with tar on glusterfs, when writing an archive: {code:java} tar: ./jobs/Platform-smoke-tests/builds/86/htmlreports/Last_20Gatling_20Report/segmenterbootstrapanddeleteerror-1530180334104/js/theme.js: file changed as we read it tar: ./jobs/Platform-smoke-tests/builds/86/htmlreports/Last_20Gatling_20Report/segmenterbootstrapanddeleteerror-1530180334104/js/jquery.min.js: file changed as we read it tar: ./jobs/Platform-smoke-tests/builds/86/htmlreports/Last_20Gatling_20Report/segmenterbootstrapanddeleteerror-1530180334104/js/assertions.json: file changed as we read it tar: ./jobs/Platform-smoke-tests/builds/86/htmlreports/Last_20Gatling_20Report/segmenterbootstrapanddeleteerror-1530180334104/js/moment.min.js: file changed as we read it tar: ./jobs/Platform-smoke-tests/builds/86/htmlreports/Last_20Gatling_20Report/segmenterbootstrapanddeleteerror-1530180334104/js/menu.js: file changed as we read it tar: ./jobs/Platform-smoke-tests/builds/86/htmlreports/Last_20Gatling_20Report/segmenterscheduledmscommand-1530180208724/req_list-the-schedu-72891.html: file changed as we read it tar: ./jobs/Platform-smoke-tests/builds/86/htmlreports/Last_20Gatling_20Report/segmenterscheduledmscommand-1530180208724/style/bootstrap.min.css: file changed as we read it tar: ./jobs/Platform-smoke-tests/builds/86/htmlreports/Last_20Gatling_20Report/segmenterscheduledmscommand-1530180208724/style/cible.png: file changed as we read it tar: ./jobs/Platform-smoke-tests/builds/86/htmlreports/Last_20Gatling_20Report/segmenterscheduledmscommand-1530180208724/style/logo-gatling.jpg: file changed as we read it tar: ./jobs/Platform-smoke-tests/builds/86/htmlreports/Last_20Gatling_20Report/segmenterscheduledmscommand-1530180208724/style/stat-fond.png: file changed as we read it{code} > Failed to read `log header` from file channel > --------------------------------------------- > > Key: KAFKA-7282 > URL: https://issues.apache.org/jira/browse/KAFKA-7282 > Project: Kafka > Issue Type: Bug > Affects Versions: 0.11.0.2, 1.1.1, 2.0.0 > Environment: Linux > Reporter: Alastair Munro > Priority: Major > > Full stack trace: > {code:java} > [2018-08-13 11:22:01,635] ERROR [ReplicaManager broker=2] Error processing > fetch operation on partition segmenter-evt-v1-14, offset 96745 > (kafka.server.ReplicaManager) > org.apache.kafka.common.KafkaException: java.io.EOFException: Failed to read > `log header` from file channel `sun.nio.ch.FileChannelImpl@6e6d8ddd`. > Expected to read 17 bytes, but reached end of file after reading 0 bytes. > Started read from position 25935. > at > org.apache.kafka.common.record.RecordBatchIterator.makeNext(RecordBatchIterator.java:40) > at > org.apache.kafka.common.record.RecordBatchIterator.makeNext(RecordBatchIterator.java:24) > at > org.apache.kafka.common.utils.AbstractIterator.maybeComputeNext(AbstractIterator.java:79) > at > org.apache.kafka.common.utils.AbstractIterator.hasNext(AbstractIterator.java:45) > at > org.apache.kafka.common.record.FileRecords.searchForOffsetWithSize(FileRecords.java:286) > at kafka.log.LogSegment.translateOffset(LogSegment.scala:254) > at kafka.log.LogSegment.read(LogSegment.scala:277) > at kafka.log.Log$$anonfun$read$2.apply(Log.scala:1159) > at kafka.log.Log$$anonfun$read$2.apply(Log.scala:1114) > at kafka.log.Log.maybeHandleIOException(Log.scala:1837) > at kafka.log.Log.read(Log.scala:1114) > at > kafka.server.ReplicaManager.kafka$server$ReplicaManager$$read$1(ReplicaManager.scala:912) > at > kafka.server.ReplicaManager$$anonfun$readFromLocalLog$1.apply(ReplicaManager.scala:974) > at > kafka.server.ReplicaManager$$anonfun$readFromLocalLog$1.apply(ReplicaManager.scala:973) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) > at kafka.server.ReplicaManager.readFromLocalLog(ReplicaManager.scala:973) > at kafka.server.ReplicaManager.readFromLog$1(ReplicaManager.scala:802) > at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:815) > at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:678) > at kafka.server.KafkaApis.handle(KafkaApis.scala:107) > at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:69) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)