[ 
https://issues.apache.org/jira/browse/HDDS-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18075753#comment-18075753
 ] 

Priyesh K commented on HDDS-14969:
----------------------------------

[~adoroszlai] TestOzoneManagerHAWithStoppedNodes failures are because repeated 
leader election happening after {{testListVolumes test which}} slow test 
execution and random tests get timedout.

Three configurations interact to cause this issue
 # Aggressive log purging ({{{}LOG_PURGE_GAP = 50{}}}, {{{}SNAPSHOT_THRESHOLD = 
50{}}})
A Ratis snapshot is taken every 50 transactions, and log entries older than 
{{snapshotIndex 50}} are permanently deleted. Creating 100 volumes generates 
100+ Raft log entries, which pushes the cluster well past the purge boundary. 
After the purge, entries in the early part of the log are gone.
 # {{installSnapshotEnabled = false}}
Ratis's built-in gRPC snapshot transfer is disabled in OM. When the leader's 
LogAppender detects that a lagging follower needs log entries that have been 
purged, it cannot send them via {{{}AppendEntries{}}}. Instead it triggers 
{{{}notifyInstallSnapshotFromLeader(){}}}, which starts an async HTTP snapshot 
download. While the download is in progress, the LogAppender sends no 
heartbeats to that follower.

 # 1-second election timeout ({{{}RATIS_RPC_TIMEOUT = 1000ms{}}})
Without heartbeats, the follower's election timer fires every ~1 second. It 
repeatedly attempts {{{}PRE_VOTE{}}}, gets rejected instantly (valid leader 
exists), and resets its timer. This loop continues for the entire duration of 
the snapshot download, which scales with the size of the RocksDB state that is 
with how many volumes were created.

I figured out this problem when i changed total volume created to 10 and then 
compared logs with the existing one. It can be resolved by changing 
{{LOG_PURGE_GAP to higher value. Updated PR with these change.}}

> Bump Ratis to 3.2.2
> -------------------
>
>                 Key: HDDS-14969
>                 URL: https://issues.apache.org/jira/browse/HDDS-14969
>             Project: Apache Ozone
>          Issue Type: Task
>          Components: build
>            Reporter: Attila Doroszlai
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: 
> org.apache.hadoop.ozone.om.TestOzoneManagerHAWithStoppedNodes-output.txt, 
> org.apache.hadoop.ozone.om.TestOzoneManagerHAWithStoppedNodes.txt
>
>
> Bump Ratis and its dependencies to:
> {code}
>     <!-- versions included in ratis-thirdparty, update in sync -->
>     <ratis-thirdparty.grpc.version>1.77.1</ratis-thirdparty.grpc.version>
>     
> <ratis-thirdparty.netty.version>4.1.130.Final</ratis-thirdparty.netty.version>
>     
> <ratis-thirdparty.protobuf.version>3.25.8</ratis-thirdparty.protobuf.version>
>     <ratis.thirdparty.version>1.0.11</ratis.thirdparty.version>
>     <ratis.version>3.2.2</ratis.version>
> {code}
> Fix failure in TestOzoneManagerHAWithStoppedNodes:
> {code}
> Tests run: 13, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 229.9 s <<< 
> FAILURE! -- in org.apache.hadoop.ozone.om.TestOzoneManagerHAWithStoppedNodes
> org.apache.hadoop.ozone.om.TestOzoneManagerHAWithStoppedNodes.testRetryCacheWithDownedOM
>  -- Time elapsed: 18.21 s <<< ERROR!
> org.apache.ratis.protocol.exceptions.TransferLeadershipException: 
> omNode-3@group-523986131536: Failed to transfer leadership to omNode-1 (the 
> current leader is omNode-3): TIMED_OUT(10s)
>       at 
> org.apache.ratis.server.impl.TransferLeadership$PendingRequest.complete(TransferLeadership.java:152)
>       at 
> org.apache.ratis.server.impl.TransferLeadership.lambda$complete$14(TransferLeadership.java:345)
>       at java.base/java.util.Optional.ifPresent(Optional.java:178)
>       at 
> org.apache.ratis.server.impl.TransferLeadership.complete(TransferLeadership.java:345)
>       at 
> org.apache.ratis.server.impl.TransferLeadership.lambda$start$9(TransferLeadership.java:310)
>       at 
> org.apache.ratis.util.TimeoutTimer.lambda$onTimeout$2(TimeoutTimer.java:101)
>       at org.apache.ratis.util.LogUtils.runAndLog(LogUtils.java:38)
>       at org.apache.ratis.util.LogUtils$1.run(LogUtils.java:78)
>       at org.apache.ratis.util.TimeoutTimer$Task.run(TimeoutTimer.java:55)
>       at java.base/java.util.TimerThread.mainLoop(Timer.java:566)
>       at java.base/java.util.TimerThread.run(Timer.java:516)
> org.apache.hadoop.ozone.om.TestOzoneManagerHAWithStoppedNodes.testIncrementalWaitTimeWithSameNodeFailover
>  -- Time elapsed: 8.006 s <<< ERROR!
> java.io.IOException: Could not determine or connect to OM Leader.
>       at 
> org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransport.submitRequest(Hadoop3OmTransport.java:119)
>       at 
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:341)
>       at 
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.createVolume(OzoneManagerProtocolClientSideTranslatorPB.java:363)
>       at 
> org.apache.hadoop.ozone.client.rpc.RpcClient.createVolume(RpcClient.java:466)
>       at 
> org.apache.hadoop.ozone.client.ObjectStore.createVolume(ObjectStore.java:122)
>       at 
> org.apache.hadoop.ozone.om.TestOzoneManagerHA.createKeyTest(TestOzoneManagerHA.java:387)
>       at 
> org.apache.hadoop.ozone.om.TestOzoneManagerHAWithStoppedNodes.testIncrementalWaitTimeWithSameNodeFailover(TestOzoneManagerHAWithStoppedNodes.java:487)
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to