Thanks Mingxian for the great discussion. We have a really good turnout yesterday. Nearly 30 participants joined the call.
The recording is available here: https://cloudera.zoom .us/rec/share/RC5mIj9Wf_M34soGcXvfHZP-5YS3uZez0zBRN5NVVSCpKTiWPueBHMS90bFAT7eZ.c5ZLNd7MTjnJGpsW Passcode: P6skX#W+ Please find the notes below. Mingxian will translate the slides and share publicly later. 9/9 Mandarin online sync Weichiu, Xiaoqiao, Aiphago, brahma, huhaiyang, Leon Gao, Lisheng Sun, WangHongbing, zy jordan, Chen Liang, xuzq, starliu, WA, le, wangyongli, feilongh, fengpk, jiayu xu, Tang Campbell, PhiloHE, Sibyl, Jinglun Aiphago (limingxiang) discussed DataNode lock optimization. - HDFS-15000 <https://issues.apache.org/jira/browse/HDFS-15000> Partition locks by block pool. Problem: multiple block pool may share the same disk, interfere each other. - HDFS-15382 <https://issues.apache.org/jira/browse/HDFS-15382> Partition locks even further by disk granularity.: # of locks = # of block pools * (1 + # of disks). LockManager updates locks dynamically. Benchmark: the improvement is even more with more stress. Prior to the change, the latency can go as high as 6000ms. After the change, the latency reduced to less than 100ms. Future work: The implementation was based on 2.7. Need to forward port to newer releases for the community. Load balance between multiple disks. The slides will be shared and translated later. On Tue, Sep 8, 2020 at 7:08 AM Wei-Chiu Chuang <weic...@apache.org> wrote: > We are happy to have 李铭翔 leading the discussion on DataNode拆锁优化 (DataNode > locking optimization) this week. > > > Time/Date: > > 9/9 10PM (US west coast PST) > > 9/10 1PM (Beijing, China CST) > > > Join Zoom Meeting > > https://cloudera.zoom.us/j/880548968 > > >
