[
https://issues.apache.org/jira/browse/RATIS-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sammi Chen updated RATIS-2507:
------------------------------
Description:
When RaftLog end log index is smaller than last snapshot index during
RaftServer startup, ratis applied a new entity to raft log before it throws out
{code:java}
2026-04-19 17:28:34,546 ERROR
[om43-server-thread1]-org.apache.ratis.server.raftlog.RaftLog:
om43@group-A6BF76F23EF4-SegmentedRaftLog: Failed to append (t:11, i:1251979),
METADATAENTRY(c:1251977)
java.lang.IllegalStateException: gap between entries 1251979 and 975728
at org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:60)
at
org.apache.ratis.server.raftlog.segmented.LogSegment.append(LogSegment.java:320)
at
org.apache.ratis.server.raftlog.segmented.LogSegment.appendToOpenSegment(LogSegment.java:307)
at
org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogCache.appendEntry(SegmentedRaftLogCache.java:556)
at
org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.appendEntryImpl(SegmentedRaftLog.java:414)
at
org.apache.ratis.server.raftlog.RaftLogBase.lambda$appendEntry$10(RaftLogBase.java:330)
at
org.apache.ratis.server.raftlog.RaftLogSequentialOps$Runner.runSequentially(RaftLogSequentialOps.java:78)
at
org.apache.ratis.server.raftlog.RaftLogBase.appendEntry(RaftLogBase.java:330)
at
org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.appendImpl(SegmentedRaftLog.java:456)
at
org.apache.ratis.server.raftlog.RaftLogBase.lambda$append$11(RaftLogBase.java:337)
at
org.apache.ratis.server.raftlog.RaftLogSequentialOps$Runner.runSequentially(RaftLogSequentialOps.java:69)
at org.apache.ratis.server.raftlog.RaftLogBase.append(RaftLogBase.java:337)
at
org.apache.ratis.server.impl.RaftServerImpl.appendEntriesAsync(RaftServerImpl.java:1532)
at
org.apache.ratis.server.impl.RaftServerImpl.appendEntriesAsync(RaftServerImpl.java:1396)
at
org.apache.ratis.server.impl.RaftServerProxy.lambda$null$25(RaftServerProxy.java:637)
at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:117)
at
org.apache.ratis.server.impl.RaftServerImpl.lambda$executeSubmitServerRequestAsync$11(RaftServerImpl.java:825)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
{code}
Ratis should not apply the new entity if it's index has gap with last index in
raft log.
Refer to https://issues.apache.org/jira/browse/HDDS-15103 to get more info.
was:
When RaftLog end log index is smaller than last snapshot index during
RaftServer startup, ratis applied a new entity to raft log before it throws out
{code:java}
2026-04-19 17:28:34,546 ERROR
[om43-server-thread1]-org.apache.ratis.server.raftlog.RaftLog:
om43@group-A6BF76F23EF4-SegmentedRaftLog: Failed to append (t:11, i:1251979),
METADATAENTRY(c:1251977)
java.lang.IllegalStateException: gap between entries 1251979 and 975728
at org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:60)
at
org.apache.ratis.server.raftlog.segmented.LogSegment.append(LogSegment.java:320)
at
org.apache.ratis.server.raftlog.segmented.LogSegment.appendToOpenSegment(LogSegment.java:307)
at
org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogCache.appendEntry(SegmentedRaftLogCache.java:556)
at
org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.appendEntryImpl(SegmentedRaftLog.java:414)
at
org.apache.ratis.server.raftlog.RaftLogBase.lambda$appendEntry$10(RaftLogBase.java:330)
at
org.apache.ratis.server.raftlog.RaftLogSequentialOps$Runner.runSequentially(RaftLogSequentialOps.java:78)
at
org.apache.ratis.server.raftlog.RaftLogBase.appendEntry(RaftLogBase.java:330)
at
org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.appendImpl(SegmentedRaftLog.java:456)
at
org.apache.ratis.server.raftlog.RaftLogBase.lambda$append$11(RaftLogBase.java:337)
at
org.apache.ratis.server.raftlog.RaftLogSequentialOps$Runner.runSequentially(RaftLogSequentialOps.java:69)
at org.apache.ratis.server.raftlog.RaftLogBase.append(RaftLogBase.java:337)
at
org.apache.ratis.server.impl.RaftServerImpl.appendEntriesAsync(RaftServerImpl.java:1532)
at
org.apache.ratis.server.impl.RaftServerImpl.appendEntriesAsync(RaftServerImpl.java:1396)
at
org.apache.ratis.server.impl.RaftServerProxy.lambda$null$25(RaftServerProxy.java:637)
at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:117)
at
org.apache.ratis.server.impl.RaftServerImpl.lambda$executeSubmitServerRequestAsync$11(RaftServerImpl.java:825)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
{code}
A real case, is an OM failed to install snapshot and terminate itself. The on
disk state machine state(rocksdb) and raft log files are from different source,
state machine data from leader, raft log files are local, and snapshot index
get from state machine is bigger than raft log end index, which further cause
another issue.
Refer to https://issues.apache.org/jira/browse/HDDS-15103 to get more info.
> Fix java.lang.IllegalStateException "gap between entries"
> ---------------------------------------------------------
>
> Key: RATIS-2507
> URL: https://issues.apache.org/jira/browse/RATIS-2507
> Project: Ratis
> Issue Type: Bug
> Reporter: Sammi Chen
> Assignee: Sammi Chen
> Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> When RaftLog end log index is smaller than last snapshot index during
> RaftServer startup, ratis applied a new entity to raft log before it throws
> out
> {code:java}
> 2026-04-19 17:28:34,546 ERROR
> [om43-server-thread1]-org.apache.ratis.server.raftlog.RaftLog:
> om43@group-A6BF76F23EF4-SegmentedRaftLog: Failed to append (t:11, i:1251979),
> METADATAENTRY(c:1251977)
> java.lang.IllegalStateException: gap between entries 1251979 and 975728
> at org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:60)
> at
> org.apache.ratis.server.raftlog.segmented.LogSegment.append(LogSegment.java:320)
> at
> org.apache.ratis.server.raftlog.segmented.LogSegment.appendToOpenSegment(LogSegment.java:307)
> at
> org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogCache.appendEntry(SegmentedRaftLogCache.java:556)
> at
> org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.appendEntryImpl(SegmentedRaftLog.java:414)
> at
> org.apache.ratis.server.raftlog.RaftLogBase.lambda$appendEntry$10(RaftLogBase.java:330)
> at
> org.apache.ratis.server.raftlog.RaftLogSequentialOps$Runner.runSequentially(RaftLogSequentialOps.java:78)
> at
> org.apache.ratis.server.raftlog.RaftLogBase.appendEntry(RaftLogBase.java:330)
> at
> org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.appendImpl(SegmentedRaftLog.java:456)
> at
> org.apache.ratis.server.raftlog.RaftLogBase.lambda$append$11(RaftLogBase.java:337)
> at
> org.apache.ratis.server.raftlog.RaftLogSequentialOps$Runner.runSequentially(RaftLogSequentialOps.java:69)
> at org.apache.ratis.server.raftlog.RaftLogBase.append(RaftLogBase.java:337)
> at
> org.apache.ratis.server.impl.RaftServerImpl.appendEntriesAsync(RaftServerImpl.java:1532)
> at
> org.apache.ratis.server.impl.RaftServerImpl.appendEntriesAsync(RaftServerImpl.java:1396)
> at
> org.apache.ratis.server.impl.RaftServerProxy.lambda$null$25(RaftServerProxy.java:637)
> at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:117)
> at
> org.apache.ratis.server.impl.RaftServerImpl.lambda$executeSubmitServerRequestAsync$11(RaftServerImpl.java:825)
> at
> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:840)
> {code}
> Ratis should not apply the new entity if it's index has gap with last index
> in raft log.
> Refer to https://issues.apache.org/jira/browse/HDDS-15103 to get more info.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
