Hi folks,
We have been experiencing node failures with the error "Failed to write
buffer." recently. Any ideas or optimizations not to get the error and node
failure?
Thanks...
[2019-08-22T01:20:55,916][ERROR][wal-write-worker%null-#221][] Critical
system error detected. Will be handled accordingly to configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
[type=CRITICAL_ERROR, err=class
o.a.i.i.processors.cache.persistence.StorageException: Failed to write
buffer.]]
org.apache.ignite.internal.processors.cache.persistence.StorageException:
Failed to write buffer.
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$WALWriter.writeBuffer(FileWriteAheadLogManager.java:3484)
[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$WALWriter.body(FileWriteAheadLogManager.java:3301)
[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
[ignite-core-2.7.5.jar:2.7.5]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: java.nio.channels.ClosedChannelException
at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110)
~[?:1.8.0_201]
at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:253)
~[?:1.8.0_201]
at
org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIO.position(RandomAccessFileIO.java:48)
~[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.persistence.file.FileIODecorator.position(FileIODecorator.java:41)
~[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.persistence.file.AbstractFileIO.writeFully(AbstractFileIO.java:111)
~[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$WALWriter.writeBuffer(FileWriteAheadLogManager.java:3477)
~[ignite-core-2.7.5.jar:2.7.5]
... 3 more
[2019-08-22T01:20:55,921][WARN
][wal-write-worker%null-#221][FailureProcessor] No deadlocked threads
detected.
[2019-08-22T01:20:56,347][WARN
][wal-write-worker%null-#221][FailureProcessor] Thread dump at 2019/08/22
01:20:56 UTC
*Ignite version*: 2.7.5
*Cluster size*: 16
*Client size*: 22
*Cluster OS version*: Centos 7
*Cluster Kernel version*: 4.4.185-1.el7.elrepo.x86_64
*Java version* :
java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)
Current disk sizes;
Screen_Shot_2019-08-22_at_12.png
<http://apache-ignite-users.70518.x6.nabble.com/file/t2515/Screen_Shot_2019-08-22_at_12.png>
Ignite and gc logs;
ignite-9.zip
<http://apache-ignite-users.70518.x6.nabble.com/file/t2515/ignite-9.zip>
Ignite configuration file;
default-config.xml
<http://apache-ignite-users.70518.x6.nabble.com/file/t2515/default-config.xml>
-----
İbrahim Halil Altun
Senior Software Engineer @ Segmentify
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
