[ https://issues.apache.org/jira/browse/RATIS-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Duong updated RATIS-2129: ------------------------- Summary: Low replication performance because of lock contention on RaftLog (was: Low replication performance because GrpcLogAppender is often blocked by RaftLog's readLock) > Low replication performance because of lock contention on RaftLog > ----------------------------------------------------------------- > > Key: RATIS-2129 > URL: https://issues.apache.org/jira/browse/RATIS-2129 > Project: Ratis > Issue Type: Bug > Components: server > Affects Versions: 3.1.0 > Reporter: Duong > Assignee: Tsz-wo Sze > Priority: Blocker > Labels: Performance, performance > Attachments: Screenshot 2024-07-22 at 4.40.07 PM-1.png, Screenshot > 2024-07-22 at 4.40.07 PM.png, dn_echo_leader_profile.html, > image-2024-07-22-15-25-46-155.png, ratis_ratfLog_lock_contention.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > Today, the GrpcLogAppender thread makes a lot of calls that need RaftLog's > readLock. In an active environment, RaftLog is always busy appending > transactions from clients, thus writeLock is frequently busy. This makes the > replication performance slow. > See the [^dn_echo_leader_profile.html], or in the picture below, the purple > is the time taken to acquire readLock from RaftLog. > # !image-2024-07-22-15-25-46-155.png|width=854,height=425! > So far, I'm not sure if this is a regression from a recent change in > 3.1.0/3.0.0, or if it's been always the case. > A few early considerations: > # The rate of calling RaftLog per GrpcLogAppender seems to be too high. > Instead of calling RaftLog multiple, maybe the log appended can call once to > obtain all the required information? > # Can RaftLog expose those data without requiring a read lock? -- This message was sent by Atlassian Jira (v8.20.10#820010)