[
https://issues.apache.org/jira/browse/HDFS-5064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13732990#comment-13732990
]
Aaron T. Myers commented on HDFS-5064:
--------------------------------------
bq. I think most of writers on SBN are datanodes. If this is true, separating
FSN and BlockManager locking will help. Last time I checked, we wanted a
facility to enforce lock hierarchy before attempting to do this.
Yes, this would be great. I took a look into this and concluded that it would
be a pretty complex change as it currently stands. I also think that it won't
necessarily completely address the issue, since there are several threads which
periodically wake up in the NN and try to get the FSNS write lock, and if this
occurs while the FSNS read lock is held for a long time, we'll be right back in
this situation.
bq. Or we could resort to a SBN-scpecific solution, since it probably only
needs to block EditLogTailer and perhaps prevent concurrent checkpointing.
This is what I was thinking. I chatted about this offline with Todd Lipcon, and
we came up with the following idea:
* Add new methods {{FSNS#readLock(boolean longRead)}} and
{{FSNS#readUnlock(boolean longRead)}}. When these are called, if the argument
longRead is true, an AtomicInteger will be incremented or decremented to
indicate that the read lock may be held by this thread for a long time.
* We change {{FSNS#writeLock()}} to check if the aforementioned counter is
greater than zero, which would indicate that there's a long reader currently
holding the lock. In this case, instead of immediately trying to get the write
lock, the thread will {{wait(...)}} in a loop until the counter returns to zero.
* We will change the handful of places in the SBN where we know the read lock
may be held for a long time to indicate that these are long reads. The edit log
tailer and the checkpointing thread are the two places that come to mind, as
Kihwal mentioned.
* All other operations will still take the (short) read lock as normal.
This should allow short reads to proceed concurrently with the reads which we
have now identified in the code as potentially taking a long time.
What do you think of this proposal, Kihwal?
> Standby checkpoints should not block concurrent readers
> -------------------------------------------------------
>
> Key: HDFS-5064
> URL: https://issues.apache.org/jira/browse/HDFS-5064
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, namenode
> Affects Versions: 2.1.1-beta
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
>
> We've observed an issue which causes fetches of the {{/jmx}} page of the NN
> to take a long time to load when the standby is in the process of creating a
> checkpoint.
> Even though both creating the checkpoint and gathering the statistics for
> {{/jmx}} take only the FSNS read lock, the issue is that since the FSNS uses
> a _fair_ RW lock, a single writer attempting to get the lock will block all
> threads attempting to get only the read lock for the duration of the
> checkpoint. This will cause {{/jmx}}, and really any thread only attempting
> to get the read lock, to block for the duration of the checkpoint, even
> though they should be able to proceed concurrently with the checkpointing
> thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
