[
https://issues.apache.org/jira/browse/HDDS-8880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17733638#comment-17733638
]
Attila Doroszlai commented on HDDS-8880:
----------------------------------------
OM is waiting for Ratis:
{code}
"IPC Server handler 15 on default port 15004"
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
at
java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at
java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisServer.submitRequestToRatis(OzoneManagerRatisServer.java:288)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisServer.submitRequest(OzoneManagerRatisServer.java:250)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestToRatis(OzoneManagerProtocolServerSideTranslatorPB.java:214)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:199)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB$$Lambda$1338/1195943629.apply(Unknown
Source)
at
org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:147)
{code}
OM1 is waiting for OM3 to install snapshot, append log or similar:
{code}
"omNode-1@group-523986131536->omNode-3-GrpcLogAppender-LogAppenderDaemon"
java.lang.Thread.State: TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at
org.apache.ratis.grpc.server.GrpcLogAppender.appendLog(GrpcLogAppender.java:286)
at
org.apache.ratis.grpc.server.GrpcLogAppender.run(GrpcLogAppender.java:168)
at
org.apache.ratis.server.leader.LogAppenderDaemon.run(LogAppenderDaemon.java:78)
{code}
Comparing OM2 and OM3, the latter lacks key threads.
{code:title=omNode-2 threads}
"omNode-2-server-thread4"
"omNode-2-server-thread3"
"omNode-2@group-523986131536-FollowerState"
"omNode-2@group-523986131536-StateMachineUpdater"
"omNode-2@group-523986131536-SegmentedRaftLogWorker"
"omNode-2-groupManagement"
{code}
{code:title=omNode-3 threads}
"omNode-3-groupManagement"
{code}
Log is full of {{Failed appendEntries}} and {{Failed InstallSnapshot}}, but I
don't see any exceptions from OM3.
Last log from OM3:
{code}
2023-06-16 17:16:42,313 [main] INFO om.OzoneManager
(OzoneManager.java:initializeRatisServer(2106)) - OzoneManager Ratis server
initialized at port 15015
2023-06-16 17:16:42,314 [main] INFO om.OzoneManager
(OzoneManager.java:getRpcServer(1166)) - Creating RPC Server
2023-06-16 17:16:42,313 [omNode-3-groupManagement] INFO
server.RaftServer$Division (RaftServerImpl.java:<init>(198)) - omNode-3: new
RaftServerImpl for
group-523986131536:[omNode-3|rpc:localhost:15015|priority:0|startupRole:FOLLOWER,
omNode-1|rpc:localhost:15007|priority:0|startupRole:FOLLOWER,
omNode-2|rpc:localhost:15011|priority:0|startupRole:FOLLOWER] with
OzoneManagerStateMachine:uninitialized
2023-06-16 17:16:42,316 [omNode-3-groupManagement] INFO
server.RaftServer$Division (ServerState.java:<init>(120)) -
omNode-3@group-523986131536: ConfigurationManager, init=-1:
peers:[omNode-3|rpc:localhost:15015|priority:0|startupRole:FOLLOWER,
omNode-1|rpc:localhost:15007|priority:0|startupRole:FOLLOWER,
omNode-2|rpc:localhost:15011|priority:0|startupRole:FOLLOWER]|listeners:[],
old=null, confs=<EMPTY_MAP>
...
2023-06-16 17:16:42,330 [omNode-3-groupManagement] INFO
server.RaftServerConfigKeys (ConfUtils.java:logGet(46)) -
raft.server.threadpool.client.cached = true (default)
2023-06-16 17:16:42,330 [omNode-3-groupManagement] INFO
server.RaftServerConfigKeys (ConfUtils.java:logGet(46)) -
raft.server.threadpool.client.size = 0 (default)
{code}
CC [~szetszwo]
> Intermittent fork timeout in TestOMRatisSnapshots
> -------------------------------------------------
>
> Key: HDDS-8880
> URL: https://issues.apache.org/jira/browse/HDDS-8880
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Attila Doroszlai
> Priority: Critical
> Attachments: it-om.zip
>
>
> Not sure if this has the same root cause as HDDS-8876 or not, so filing
> separately.
> Surefire fork is killed due to timeout for TestOMRatisSnapshot.
> Main thread is waiting for response from OM at various points (see thread
> dumps linked). Example:
> {code}
> "main"
> java.lang.Thread.State: WAITING
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at
> org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(AsyncGet.java:65)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1572)
> ...
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:304)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.updateKey(OzoneManagerProtocolClientSideTranslatorPB.java:802)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.commitKey(OzoneManagerProtocolClientSideTranslatorPB.java:760)
> at
> org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool.commitKey(BlockOutputStreamEntryPool.java:341)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.close(KeyOutputStream.java:557)
> at
> org.apache.hadoop.ozone.client.io.OzoneOutputStream.close(OzoneOutputStream.java:86)
> at
> org.apache.hadoop.ozone.om.TestOzoneManagerHA.createKey(TestOzoneManagerHA.java:231)
> at
> org.apache.hadoop.ozone.om.TestOMRatisSnapshots.writeKeys(TestOMRatisSnapshots.java:1016)
> at
> org.apache.hadoop.ozone.om.TestOMRatisSnapshots.testInstallSnapshot(TestOMRatisSnapshots.java:205)
> {code}
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/05/30/22802/it-om/2023-05-30T07-30-54_006-jvmRun1.dump
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/06/16/23399/it-om/2023-06-16T10-42-07_017-jvmRun1.dump
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/06/16/23408/it-om/2023-06-16T12-31-16_382-jvmRun1.dump
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/06/16/23417/it-om/2023-06-16T13-41-55_057-jvmRun1.dump
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/06/16/23421/it-om/2023-06-16T14-54-13_823-jvmRun1.dump
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/06/16/23425/it-om/2023-06-16T16-03-34_143-jvmRun1.dump
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
