Thanks for initiating this discussion here. I am +1 to the general approach proposed. With DN getting denser, this is necessary more than ever.
On Tue, Feb 4, 2020 at 10:33 AM Stephen O'Donnell <sodonn...@cloudera.com.invalid> wrote: > I would like to reopen an old topic, which is to introduce a Read Write > lock to the datanode. > > In the current trunk, a RentrentLock is used, so it is always exclusive. > However there are many code paths in the DN where an exclusive lock is not > necessary and a read lock would suffice. > > We know the ReentrantReadWrite lock scales fine, as it is used extensively > in the namenode, so the performance of the lock should not be a concern. > > My proposal in https://issues.apache.org/jira/browse/HDFS-15150 is to > start > small on this, and simply replace the Reentrant lock with a > ReentrantReadWrite lock, and then make all lock acquisitions take the write > lock. That would keep the locking exactly as it is now, and hopefully > result in a patch that is relatively easy to review. > > If we can agree on a patch for that, we can create followup Jiras to switch > various code paths to use the readlock over time. > > I have a patch available on HDFS-15150, so I would appreciate any thoughts > on the patch and this idea in general. > > Thanks, > > Stephen. >