Devesh Kumar Singh created HDDS-15068:
-----------------------------------------

             Summary: OM start failed with exception: 
java.lang.IllegalStateException: gap between entry term
                 Key: HDDS-15068
                 URL: https://issues.apache.org/jira/browse/HDDS-15068
             Project: Apache Ozone
          Issue Type: Task
          Components: Ozone Manager
    Affects Versions: 2.1.0
            Reporter: Devesh Kumar Singh


While creating bulk keys through freon, hit the below stack trace error. Freon 
was running smooth with all OMs up, but after 1 hour of freon running, one of 
the follower OM went down and I found exactly the same stack trace, so I 
thought of running the command :

{code:java}
ozone debug ratislogparser om --segment-path=log_inprogress_890723
{code}


and after all parsing, it failed with same below error.


{code:java}
java.util.concurrent.CompletionException: java.lang.IllegalStateException: gap 
between entry term: 11
index: 975728
metadataEntry {
  commitIndex: 975726
}
 and entry term: 11
index: 1251979
metadataEntry {
  commitIndex: 1251977
}

        at 
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
        at 
java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
        at 
java.base/java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1498)
        at 
java.base/java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1219)
        at 
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at 
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
        at 
org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:191)
        at 
org.apache.ratis.util.ConcurrentUtils.lambda$null$4(ConcurrentUtils.java:180)
        at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.IllegalStateException: gap between entry term: 11
index: 975728
metadataEntry {
  commitIndex: 975726
}
 and entry term: 11
index: 1251979
metadataEntry {
  commitIndex: 1251977
}

        at org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:60)
        at 
org.apache.ratis.server.raftlog.segmented.LogSegment.readSegmentFile(LogSegment.java:133)
        at 
org.apache.ratis.server.raftlog.segmented.LogSegment.loadSegment(LogSegment.java:164)
        at 
org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogCache.loadSegment(SegmentedRaftLogCache.java:381)
        at 
org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.loadLogSegments(SegmentedRaftLog.java:241)
        at 
org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.openImpl(SegmentedRaftLog.java:214)
        at 
org.apache.ratis.server.raftlog.RaftLogBase.open(RaftLogBase.java:251)
        at 
org.apache.ratis.server.impl.ServerState.initRaftLog(ServerState.java:239)
        at 
org.apache.ratis.server.impl.ServerState.initRaftLog(ServerState.java:220)
        at 
org.apache.ratis.server.impl.ServerState.lambda$new$5(ServerState.java:161)
        at org.apache.ratis.util.MemoizedSupplier.get(MemoizedSupplier.java:62)
        at 
org.apache.ratis.server.impl.ServerState.initialize(ServerState.java:177)
        at 
org.apache.ratis.server.impl.RaftServerImpl.start(RaftServerImpl.java:338)
        at 
org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:188)
{code}


{code:java}
[root@<cluster>cluster> current]# ls -larht
total 65M
-rw-r--r-- 1 hdfs hdfs   54 Apr 19 16:19 raft-meta
-rw-r--r-- 1 hdfs hdfs  186 Apr 19 16:19 raft-meta.conf
drwxr-xr-x 2 hdfs hdfs 4.0K Apr 19 17:28 .
-rw-r--r-- 1 hdfs hdfs  64M Apr 19 17:28 log_inprogress_890723
drwxr-xr-x 4 hdfs hdfs 4.0K Apr 20 05:08 ..
[root@<cluster> current]#

[root@<cluster> sm]# pwd
/var/lib/hadoop-ozone/om/ratis/5341b4ef-bcd6-3183-aa83-a6bf76f23ef4/sm
[root@<cluster> sm]# ls -larht
total 8.0K
drwxr-xr-x 2 hdfs hdfs 4.0K Apr 17 06:57 .
drwxr-xr-x 4 hdfs hdfs 4.0K Apr 20 05:08 ..
[root@<cluster> sm]#
{code}


OM follower is failing to start. Logs for raft folder and OM follower logs are 
attached. 

cc: [~erose] [~Sammi] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to