[
https://issues.apache.org/jira/browse/HDDS-15068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18111239#comment-18111239
]
Wei-Chiu Chuang commented on HDDS-15068:
----------------------------------------
HDDS-16196 will bump Ratis to 3.3.0 and should eliminate this problem.
> OM start failed with exception: java.lang.IllegalStateException: gap between
> entry term
> ---------------------------------------------------------------------------------------
>
> Key: HDDS-15068
> URL: https://issues.apache.org/jira/browse/HDDS-15068
> Project: Apache Ozone
> Issue Type: Task
> Components: Ozone Manager
> Affects Versions: 2.1.0
> Reporter: Devesh Kumar Singh
> Assignee: Sammi Chen
> Priority: Major
> Attachments: om-ratis-follower-logs.zip
>
>
> While creating bulk keys through freon, hit the below stack trace error.
> Freon was running smooth with all OMs up, but after 1 hour of freon running,
> one of the follower OM went down and I found exactly the same stack trace, so
> I thought of running the command :
> {code:java}
> ozone debug ratislogparser om --segment-path=log_inprogress_890723
> {code}
> and after all parsing, it failed with same below error.
> {code:java}
> java.util.concurrent.CompletionException: java.lang.IllegalStateException:
> gap between entry term: 11
> index: 975728
> metadataEntry {
> commitIndex: 975726
> }
> and entry term: 11
> index: 1251979
> metadataEntry {
> commitIndex: 1251977
> }
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
> at
> java.base/java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1498)
> at
> java.base/java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1219)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> at
> org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:191)
> at
> org.apache.ratis.util.ConcurrentUtils.lambda$null$4(ConcurrentUtils.java:180)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:840)
> Caused by: java.lang.IllegalStateException: gap between entry term: 11
> index: 975728
> metadataEntry {
> commitIndex: 975726
> }
> and entry term: 11
> index: 1251979
> metadataEntry {
> commitIndex: 1251977
> }
> at org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:60)
> at
> org.apache.ratis.server.raftlog.segmented.LogSegment.readSegmentFile(LogSegment.java:133)
> at
> org.apache.ratis.server.raftlog.segmented.LogSegment.loadSegment(LogSegment.java:164)
> at
> org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogCache.loadSegment(SegmentedRaftLogCache.java:381)
> at
> org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.loadLogSegments(SegmentedRaftLog.java:241)
> at
> org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.openImpl(SegmentedRaftLog.java:214)
> at
> org.apache.ratis.server.raftlog.RaftLogBase.open(RaftLogBase.java:251)
> at
> org.apache.ratis.server.impl.ServerState.initRaftLog(ServerState.java:239)
> at
> org.apache.ratis.server.impl.ServerState.initRaftLog(ServerState.java:220)
> at
> org.apache.ratis.server.impl.ServerState.lambda$new$5(ServerState.java:161)
> at org.apache.ratis.util.MemoizedSupplier.get(MemoizedSupplier.java:62)
> at
> org.apache.ratis.server.impl.ServerState.initialize(ServerState.java:177)
> at
> org.apache.ratis.server.impl.RaftServerImpl.start(RaftServerImpl.java:338)
> at
> org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:188)
> {code}
> {code:java}
> [root@<cluster>cluster> current]# ls -larht
> total 65M
> -rw-r--r-- 1 hdfs hdfs 54 Apr 19 16:19 raft-meta
> -rw-r--r-- 1 hdfs hdfs 186 Apr 19 16:19 raft-meta.conf
> drwxr-xr-x 2 hdfs hdfs 4.0K Apr 19 17:28 .
> -rw-r--r-- 1 hdfs hdfs 64M Apr 19 17:28 log_inprogress_890723
> drwxr-xr-x 4 hdfs hdfs 4.0K Apr 20 05:08 ..
> [root@<cluster> current]#
> [root@<cluster> sm]# pwd
> /var/lib/hadoop-ozone/om/ratis/5341b4ef-bcd6-3183-aa83-a6bf76f23ef4/sm
> [root@<cluster> sm]# ls -larht
> total 8.0K
> drwxr-xr-x 2 hdfs hdfs 4.0K Apr 17 06:57 .
> drwxr-xr-x 4 hdfs hdfs 4.0K Apr 20 05:08 ..
> [root@<cluster> sm]#
> {code}
> OM follower is failing to start. Logs for raft folder and OM follower logs
> are attached.
> cc: [~erose] [~Sammi]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
