Re: Discussion about NameNode Fine-grained locking
Apr 26, 2024 at 3:50 PM Hui Fei >>> wrote: >>> >> > >>> >> > Thanks for interest and advice on this. >>> >> > >>> >> > Just would like to share some info here >>> >> > >>> >> > ZanderXu leads this feature and he has spent a lot of time on it. >>> He is >>> >> the main developer in stage 1. Yuanboliu and Kokonguyen191 also took >>> some >>> >> tasks. Other developers (slfan1989 haiyang1987 huangzhaobo99 >>> RocMarshal >>> >> kokonguyen191) helped review PRs. (Forgive me if I missed someone) >>> >> > >>> >> > Actually haiyang1987, Yuanboliu and Kokonguyen191 are also very >>> >> familiar with this feature. We discussed many details offline. >>> >> > >>> >> > Welcome to more people interested in joining the development and >>> review >>> >> of the stage 2 and 3. >>> >> > >>> >> > >>> >> > Zengqiang XU 于2024年4月26日周五 14:56写道： >>> >> >> >>> >> >> Thanks Shilun for your response: >>> >> >> >>> >> >> 1. This is a big and very useful feature, so it really needs more >>> >> >> developers to get on board. >>> >> >> 2. This fine grained lock has been implemented based on internal >>> >> branches >>> >> >> and has gained benefits by many companies, such as: Meituan, >>> Kuaishou, >>> >> >> Bytedance, etc. But it has not been contributed to the community >>> due >>> >> to >>> >> >> various reasons, such as there is a big difference between the >>> version >>> >> of >>> >> >> the internal branch and the community trunk branch, the internal >>> >> branch may >>> >> >> ignore some functions to make FGL clear, and the contribution >>> needs a >>> >> lot >>> >> >> of work and will take many times. It means that this solution has >>> >> already >>> >> >> been practiced in their prod environment. We have also practiced >>> it in >>> >> our >>> >> >> prod environment and gained benefits, and we are also willing to >>> spend >>> >> a >>> >> >> lot of time contributing to the community. >>> >> >> 3. Regarding the benchmark testing, we don't need to pay more >>> >> attention to >>> >> >> whether the performance is improved by 5 times, 10 times or 20 >>> times, >>> >> >> because there are too many factors that affect it. >>> >> >> 4. As I described above, this solution is already being practiced >>> by >>> >> many >>> >> >> companies. Right now, we just need to think about how to implement >>> it >>> >> with >>> >> >> high quality and more comprehensively. >>> >> >> 5. I firmly believe that all problems can be solved as long as the >>> >> overall >>> >> >> solution is right. >>> >> >> 6. I can spend a lot of time leading the promotion of this entire >>> >> feature >>> >> >> and I hope more people can join us in promoting it. >>> >> >> 7. You are always welcome to raise your concerns. >>> >> >> >>> >> >> >>> >> >> Thanks Shilun again, I hope you can help review designs and PRs. >>> Thanks >>> >> >> >>> >> >> On Fri, 26 Apr 2024 at 08:00, slfan1989 >>> wrote: >>> >> >> >>> >> >> > Thank you for your hard work! This is a very meaningful >>> improvement, >>> >> and >>> >> >> > from the design document, we can see a significant increase in >>> HDFS >>> >> >> > read/write throughput. >>> >> >> > >>> >> >> > I am happy to see the progress made on HDFS-17384. >>> >> >> > >>> >> >> > However, I still have some concerns, which roughly involve the >>> >> following >>> >> >> > aspects: >>> >> >> > >>> >> >> > 1. While ZanderXu and Hui Fei have de
Re: Discussion about NameNode Fine-grained locking
s reasons, such as there is a big difference between the >> version >> >> of >> >> >> the internal branch and the community trunk branch, the internal >> >> branch may >> >> >> ignore some functions to make FGL clear, and the contribution needs >> a >> >> lot >> >> >> of work and will take many times. It means that this solution has >> >> already >> >> >> been practiced in their prod environment. We have also practiced it >> in >> >> our >> >> >> prod environment and gained benefits, and we are also willing to >> spend >> >> a >> >> >> lot of time contributing to the community. >> >> >> 3. Regarding the benchmark testing, we don't need to pay more >> >> attention to >> >> >> whether the performance is improved by 5 times, 10 times or 20 >> times, >> >> >> because there are too many factors that affect it. >> >> >> 4. As I described above, this solution is already being practiced >> by >> >> many >> >> >> companies. Right now, we just need to think about how to implement >> it >> >> with >> >> >> high quality and more comprehensively. >> >> >> 5. I firmly believe that all problems can be solved as long as the >> >> overall >> >> >> solution is right. >> >> >> 6. I can spend a lot of time leading the promotion of this entire >> >> feature >> >> >> and I hope more people can join us in promoting it. >> >> >> 7. You are always welcome to raise your concerns. >> >> >> >> >> >> >> >> >> Thanks Shilun again, I hope you can help review designs and PRs. >> Thanks >> >> >> >> >> >> On Fri, 26 Apr 2024 at 08:00, slfan1989 >> wrote: >> >> >> >> >> >> > Thank you for your hard work! This is a very meaningful >> improvement, >> >> and >> >> >> > from the design document, we can see a significant increase in >> HDFS >> >> >> > read/write throughput. >> >> >> > >> >> >> > I am happy to see the progress made on HDFS-17384. >> >> >> > >> >> >> > However, I still have some concerns, which roughly involve the >> >> following >> >> >> > aspects: >> >> >> > >> >> >> > 1. While ZanderXu and Hui Fei have deep expertise in HDFS and are >> >> familiar >> >> >> > with related development details, we still need more community >> >> member to >> >> >> > review the code to ensure that the relevant upgrades meet >> >> expectations. >> >> >> > >> >> >> > 2. We need more details on benchmarks to ensure that test results >> >> can be >> >> >> > reproduced and to allow more community member to participate in >> the >> >> testing >> >> >> > process. >> >> >> > >> >> >> > Looking forward to everything going smoothly in the future. >> >> >> > >> >> >> > Best Regards, >> >> >> > - Shilun Fan. >> >> >> > >> >> >> > On Wed, Apr 24, 2024 at 3:51 PM Xiaoqiao He < >> hexiaoq...@apache.org> >> >> wrote: >> >> >> > >> >> >> >> cc private@h.a.o. >> >> >> >> >> >> >> >> On Wed, Apr 24, 2024 at 3:35 PM ZanderXu >> >> wrote: >> >> >> >> > >> >> >> >> > Here are some summaries about the first phase: >> >> >> >> > 1. There are no big changes in this phase >> >> >> >> > 2. This phase just uses FS lock and BM lock to replace the >> >> original >> >> >> >> global >> >> >> >> > lock >> >> >> >> > 3. It's useful to improve the performance, since some >> operations >> >> just >> >> >> >> need >> >> >> >> > to hold FS lock or BM lock instead of the global lock >> >> >> >> > 4. This feature is turned off by default, you can enable it by >> >> setting >> >> >> >> > dfs.namenode.lock.model.provider.class to >> >> >> >> > >> >> org.apache.hadoop.hdfs.server.namenode.fgl.FineGrainedFSNamesystemLock >> >> >> >> > 5. This phase is very import for the ongoing development of the >> >> entire >> >> >> >> FGL >> >> >> >> > >> >> >> >> > Here I would like to express my special thanks to >> @kokonguyen191 >> >> and >> >> >> >> > @yuanboliu for their contributions. And you are also welcome >> to >> >> join us >> >> >> >> > and complete it together. >> >> >> >> > >> >> >> >> > >> >> >> >> > On Wed, 24 Apr 2024 at 14:54, ZanderXu >> >> wrote: >> >> >> >> > >> >> >> >> > > Hi everyone >> >> >> >> > > >> >> >> >> > > All subtasks of the first phase of the FGL have been >> completed >> >> and I >> >> >> >> plan >> >> >> >> > > to merge them into the trunk and start the second phase based >> >> on the >> >> >> >> trunk. >> >> >> >> > > >> >> >> >> > > Here is the PR that used to merge the first phases into >> trunk: >> >> >> >> > > https://github.com/apache/hadoop/pull/6762 >> >> >> >> > > Here is the ticket: >> >> https://issues.apache.org/jira/browse/HDFS-17384 >> >> >> >> > > >> >> >> >> > > I hope you can help to review this PR when you are available >> >> and give >> >> >> >> some >> >> >> >> > > ideas. >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > HDFS-17385 <https://issues.apache.org/jira/browse/HDFS-17385 >> > >> >> is >> >> >> >> used for >> >> >> >> > > the second phase and I have created some subtasks to describe >> >> >> >> solutions for >> >> >> >> > > some problems, such as: snapshot, getListing, quota. >> >> >> >> > > You are welcome to join us to complete it together. >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > -- Forwarded message - >> >> >> >> > > From: Zengqiang XU >> >> >> >> > > Date: Fri, 2 Feb 2024 at 11:07 >> >> >> >> > > Subject: Discussion about NameNode Fine-grained locking >> >> >> >> > > To: >> >> >> >> > > Cc: Zengqiang XU >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > Hi everyone >> >> >> >> > > >> >> >> >> > > I have started a discussion about NameNode Fine-grained >> Locking >> >> to >> >> >> >> improve >> >> >> >> > > performance of write operations in NameNode. >> >> >> >> > > >> >> >> >> > > I started this discussion again for serval main reasons: >> >> >> >> > > 1. We have implemented it and gained nearly 7x performance >> >> >> >> improvement in >> >> >> >> > > our prod environment >> >> >> >> > > 2. Many other companies made similar improvements based on >> their >> >> >> >> internal >> >> >> >> > > branch. >> >> >> >> > > 3. This topic has been discussed for a long time, but still >> >> without >> >> >> >> any >> >> >> >> > > results. >> >> >> >> > > >> >> >> >> > > I hope we can push this important improvement in the >> community >> >> so >> >> >> >> that all >> >> >> >> > > end-users can enjoy this significant improvement. >> >> >> >> > > >> >> >> >> > > I'd really appreciate you can join in and work with me to >> push >> >> this >> >> >> >> > > feature forward. >> >> >> >> > > >> >> >> >> > > Thanks very much. >> >> >> >> > > >> >> >> >> > > Ticket: HDFS-17366 < >> >> https://issues.apache.org/jira/browse/HDFS-17366> >> >> >> >> > > Design: NameNode Fine-grained locking based on directory tree >> >> >> >> > > < >> >> >> >> >> >> >> https://docs.google.com/document/d/1X499gHxT0WSU1fj8uo4RuF3GqKxWkWXznXx4tspTBLY/edit?usp=sharing >> >> >> >> > >> >> >> >> > > >> >> >> >> >> >> >> >> >> >> - >> >> >> >> To unsubscribe, e-mail: private-unsubscr...@hadoop.apache.org >> >> >> >> For additional commands, e-mail: private-h...@hadoop.apache.org >> >> >> >> >> >> >> >> >> >> >> >> - >> >> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org >> >> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org >> >> >> >> >> >
Re: Discussion about NameNode Fine-grained locking
; and I hope more people can join us in promoting it. > >> >> 7. You are always welcome to raise your concerns. > >> >> > >> >> > >> >> Thanks Shilun again, I hope you can help review designs and PRs. > Thanks > >> >> > >> >> On Fri, 26 Apr 2024 at 08:00, slfan1989 > wrote: > >> >> > >> >> > Thank you for your hard work! This is a very meaningful > improvement, > >> and > >> >> > from the design document, we can see a significant increase in HDFS > >> >> > read/write throughput. > >> >> > > >> >> > I am happy to see the progress made on HDFS-17384. > >> >> > > >> >> > However, I still have some concerns, which roughly involve the > >> following > >> >> > aspects: > >> >> > > >> >> > 1. While ZanderXu and Hui Fei have deep expertise in HDFS and are > >> familiar > >> >> > with related development details, we still need more community > >> member to > >> >> > review the code to ensure that the relevant upgrades meet > >> expectations. > >> >> > > >> >> > 2. We need more details on benchmarks to ensure that test results > >> can be > >> >> > reproduced and to allow more community member to participate in the > >> testing > >> >> > process. > >> >> > > >> >> > Looking forward to everything going smoothly in the future. > >> >> > > >> >> > Best Regards, > >> >> > - Shilun Fan. > >> >> > > >> >> > On Wed, Apr 24, 2024 at 3:51 PM Xiaoqiao He > > >> wrote: > >> >> > > >> >> >> cc private@h.a.o. > >> >> >> > >> >> >> On Wed, Apr 24, 2024 at 3:35 PM ZanderXu > >> wrote: > >> >> >> > > >> >> >> > Here are some summaries about the first phase: > >> >> >> > 1. There are no big changes in this phase > >> >> >> > 2. This phase just uses FS lock and BM lock to replace the > >> original > >> >> >> global > >> >> >> > lock > >> >> >> > 3. It's useful to improve the performance, since some operations > >> just > >> >> >> need > >> >> >> > to hold FS lock or BM lock instead of the global lock > >> >> >> > 4. This feature is turned off by default, you can enable it by > >> setting > >> >> >> > dfs.namenode.lock.model.provider.class to > >> >> >> > > >> org.apache.hadoop.hdfs.server.namenode.fgl.FineGrainedFSNamesystemLock > >> >> >> > 5. This phase is very import for the ongoing development of the > >> entire > >> >> >> FGL > >> >> >> > > >> >> >> > Here I would like to express my special thanks to @kokonguyen191 > >> and > >> >> >> > @yuanboliu for their contributions. And you are also welcome to > >> join us > >> >> >> > and complete it together. > >> >> >> > > >> >> >> > > >> >> >> > On Wed, 24 Apr 2024 at 14:54, ZanderXu > >> wrote: > >> >> >> > > >> >> >> > > Hi everyone > >> >> >> > > > >> >> >> > > All subtasks of the first phase of the FGL have been completed > >> and I > >> >> >> plan > >> >> >> > > to merge them into the trunk and start the second phase based > >> on the > >> >> >> trunk. > >> >> >> > > > >> >> >> > > Here is the PR that used to merge the first phases into trunk: > >> >> >> > > https://github.com/apache/hadoop/pull/6762 > >> >> >> > > Here is the ticket: > >> https://issues.apache.org/jira/browse/HDFS-17384 > >> >> >> > > > >> >> >> > > I hope you can help to review this PR when you are available > >> and give > >> >> >> some > >> >> >> > > ideas. > >> >> >> > > > >> >> >> > > > >> >> >> > > HDFS-17385 <https://issues.apache.org/jira/browse/HDFS-17385> > >> is > >> >> >> used for > >> >> >> > > the second phase and I have created some subtasks to describe > >> >> >> solutions for > >> >> >> > > some problems, such as: snapshot, getListing, quota. > >> >> >> > > You are welcome to join us to complete it together. > >> >> >> > > > >> >> >> > > > >> >> >> > > -- Forwarded message - > >> >> >> > > From: Zengqiang XU > >> >> >> > > Date: Fri, 2 Feb 2024 at 11:07 > >> >> >> > > Subject: Discussion about NameNode Fine-grained locking > >> >> >> > > To: > >> >> >> > > Cc: Zengqiang XU > >> >> >> > > > >> >> >> > > > >> >> >> > > Hi everyone > >> >> >> > > > >> >> >> > > I have started a discussion about NameNode Fine-grained > Locking > >> to > >> >> >> improve > >> >> >> > > performance of write operations in NameNode. > >> >> >> > > > >> >> >> > > I started this discussion again for serval main reasons: > >> >> >> > > 1. We have implemented it and gained nearly 7x performance > >> >> >> improvement in > >> >> >> > > our prod environment > >> >> >> > > 2. Many other companies made similar improvements based on > their > >> >> >> internal > >> >> >> > > branch. > >> >> >> > > 3. This topic has been discussed for a long time, but still > >> without > >> >> >> any > >> >> >> > > results. > >> >> >> > > > >> >> >> > > I hope we can push this important improvement in the community > >> so > >> >> >> that all > >> >> >> > > end-users can enjoy this significant improvement. > >> >> >> > > > >> >> >> > > I'd really appreciate you can join in and work with me to push > >> this > >> >> >> > > feature forward. > >> >> >> > > > >> >> >> > > Thanks very much. > >> >> >> > > > >> >> >> > > Ticket: HDFS-17366 < > >> https://issues.apache.org/jira/browse/HDFS-17366> > >> >> >> > > Design: NameNode Fine-grained locking based on directory tree > >> >> >> > > < > >> >> >> > >> > https://docs.google.com/document/d/1X499gHxT0WSU1fj8uo4RuF3GqKxWkWXznXx4tspTBLY/edit?usp=sharing > >> >> >> > > >> >> >> > > > >> >> >> > >> >> >> > >> - > >> >> >> To unsubscribe, e-mail: private-unsubscr...@hadoop.apache.org > >> >> >> For additional commands, e-mail: private-h...@hadoop.apache.org > >> >> >> > >> >> >> > >> > >> - > >> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > >> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >> > >> >
Re: Discussion about NameNode Fine-grained locking
; >> >> >> >> >> On Wed, Apr 24, 2024 at 3:35 PM ZanderXu >> wrote: >> >> >> > >> >> >> > Here are some summaries about the first phase: >> >> >> > 1. There are no big changes in this phase >> >> >> > 2. This phase just uses FS lock and BM lock to replace the >> original >> >> >> global >> >> >> > lock >> >> >> > 3. It's useful to improve the performance, since some operations >> just >> >> >> need >> >> >> > to hold FS lock or BM lock instead of the global lock >> >> >> > 4. This feature is turned off by default, you can enable it by >> setting >> >> >> > dfs.namenode.lock.model.provider.class to >> >> >> > >> org.apache.hadoop.hdfs.server.namenode.fgl.FineGrainedFSNamesystemLock >> >> >> > 5. This phase is very import for the ongoing development of the >> entire >> >> >> FGL >> >> >> > >> >> >> > Here I would like to express my special thanks to @kokonguyen191 >> and >> >> >> > @yuanboliu for their contributions. And you are also welcome to >> join us >> >> >> > and complete it together. >> >> >> > >> >> >> > >> >> >> > On Wed, 24 Apr 2024 at 14:54, ZanderXu >> wrote: >> >> >> > >> >> >> > > Hi everyone >> >> >> > > >> >> >> > > All subtasks of the first phase of the FGL have been completed >> and I >> >> >> plan >> >> >> > > to merge them into the trunk and start the second phase based >> on the >> >> >> trunk. >> >> >> > > >> >> >> > > Here is the PR that used to merge the first phases into trunk: >> >> >> > > https://github.com/apache/hadoop/pull/6762 >> >> >> > > Here is the ticket: >> https://issues.apache.org/jira/browse/HDFS-17384 >> >> >> > > >> >> >> > > I hope you can help to review this PR when you are available >> and give >> >> >> some >> >> >> > > ideas. >> >> >> > > >> >> >> > > >> >> >> > > HDFS-17385 <https://issues.apache.org/jira/browse/HDFS-17385> >> is >> >> >> used for >> >> >> > > the second phase and I have created some subtasks to describe >> >> >> solutions for >> >> >> > > some problems, such as: snapshot, getListing, quota. >> >> >> > > You are welcome to join us to complete it together. >> >> >> > > >> >> >> > > >> >> >> > > -- Forwarded message - >> >> >> > > From: Zengqiang XU >> >> >> > > Date: Fri, 2 Feb 2024 at 11:07 >> >> >> > > Subject: Discussion about NameNode Fine-grained locking >> >> >> > > To: >> >> >> > > Cc: Zengqiang XU >> >> >> > > >> >> >> > > >> >> >> > > Hi everyone >> >> >> > > >> >> >> > > I have started a discussion about NameNode Fine-grained Locking >> to >> >> >> improve >> >> >> > > performance of write operations in NameNode. >> >> >> > > >> >> >> > > I started this discussion again for serval main reasons: >> >> >> > > 1. We have implemented it and gained nearly 7x performance >> >> >> improvement in >> >> >> > > our prod environment >> >> >> > > 2. Many other companies made similar improvements based on their >> >> >> internal >> >> >> > > branch. >> >> >> > > 3. This topic has been discussed for a long time, but still >> without >> >> >> any >> >> >> > > results. >> >> >> > > >> >> >> > > I hope we can push this important improvement in the community >> so >> >> >> that all >> >> >> > > end-users can enjoy this significant improvement. >> >> >> > > >> >> >> > > I'd really appreciate you can join in and work with me to push >> this >> >> >> > > feature forward. >> >> >> > > >> >> >> > > Thanks very much. >> >> >> > > >> >> >> > > Ticket: HDFS-17366 < >> https://issues.apache.org/jira/browse/HDFS-17366> >> >> >> > > Design: NameNode Fine-grained locking based on directory tree >> >> >> > > < >> >> >> >> https://docs.google.com/document/d/1X499gHxT0WSU1fj8uo4RuF3GqKxWkWXznXx4tspTBLY/edit?usp=sharing >> >> >> > >> >> >> > > >> >> >> >> >> >> >> - >> >> >> To unsubscribe, e-mail: private-unsubscr...@hadoop.apache.org >> >> >> For additional commands, e-mail: private-h...@hadoop.apache.org >> >> >> >> >> >> >> >> - >> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org >> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org >> >>
Re: Discussion about NameNode Fine-grained locking
> to hold FS lock or BM lock instead of the global lock > >> >> > 4. This feature is turned off by default, you can enable it by > setting > >> >> > dfs.namenode.lock.model.provider.class to > >> >> > > org.apache.hadoop.hdfs.server.namenode.fgl.FineGrainedFSNamesystemLock > >> >> > 5. This phase is very import for the ongoing development of the > entire > >> >> FGL > >> >> > > >> >> > Here I would like to express my special thanks to @kokonguyen191 > and > >> >> > @yuanboliu for their contributions. And you are also welcome to > join us > >> >> > and complete it together. > >> >> > > >> >> > > >> >> > On Wed, 24 Apr 2024 at 14:54, ZanderXu > wrote: > >> >> > > >> >> > > Hi everyone > >> >> > > > >> >> > > All subtasks of the first phase of the FGL have been completed > and I > >> >> plan > >> >> > > to merge them into the trunk and start the second phase based on > the > >> >> trunk. > >> >> > > > >> >> > > Here is the PR that used to merge the first phases into trunk: > >> >> > > https://github.com/apache/hadoop/pull/6762 > >> >> > > Here is the ticket: > https://issues.apache.org/jira/browse/HDFS-17384 > >> >> > > > >> >> > > I hope you can help to review this PR when you are available and > give > >> >> some > >> >> > > ideas. > >> >> > > > >> >> > > > >> >> > > HDFS-17385 <https://issues.apache.org/jira/browse/HDFS-17385> is > >> >> used for > >> >> > > the second phase and I have created some subtasks to describe > >> >> solutions for > >> >> > > some problems, such as: snapshot, getListing, quota. > >> >> > > You are welcome to join us to complete it together. > >> >> > > > >> >> > > > >> >> > > -- Forwarded message - > >> >> > > From: Zengqiang XU > >> >> > > Date: Fri, 2 Feb 2024 at 11:07 > >> >> > > Subject: Discussion about NameNode Fine-grained locking > >> >> > > To: > >> >> > > Cc: Zengqiang XU > >> >> > > > >> >> > > > >> >> > > Hi everyone > >> >> > > > >> >> > > I have started a discussion about NameNode Fine-grained Locking > to > >> >> improve > >> >> > > performance of write operations in NameNode. > >> >> > > > >> >> > > I started this discussion again for serval main reasons: > >> >> > > 1. We have implemented it and gained nearly 7x performance > >> >> improvement in > >> >> > > our prod environment > >> >> > > 2. Many other companies made similar improvements based on their > >> >> internal > >> >> > > branch. > >> >> > > 3. This topic has been discussed for a long time, but still > without > >> >> any > >> >> > > results. > >> >> > > > >> >> > > I hope we can push this important improvement in the community so > >> >> that all > >> >> > > end-users can enjoy this significant improvement. > >> >> > > > >> >> > > I'd really appreciate you can join in and work with me to push > this > >> >> > > feature forward. > >> >> > > > >> >> > > Thanks very much. > >> >> > > > >> >> > > Ticket: HDFS-17366 < > https://issues.apache.org/jira/browse/HDFS-17366> > >> >> > > Design: NameNode Fine-grained locking based on directory tree > >> >> > > < > >> >> > https://docs.google.com/document/d/1X499gHxT0WSU1fj8uo4RuF3GqKxWkWXznXx4tspTBLY/edit?usp=sharing > >> >> > > >> >> > > > >> >> > >> >> - > >> >> To unsubscribe, e-mail: private-unsubscr...@hadoop.apache.org > >> >> For additional commands, e-mail: private-h...@hadoop.apache.org > >> >> > >> >> > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >
Re: Discussion about NameNode Fine-grained locking
>> >> On Wed, Apr 24, 2024 at 3:35 PM ZanderXu wrote: >> >> > >> >> > Here are some summaries about the first phase: >> >> > 1. There are no big changes in this phase >> >> > 2. This phase just uses FS lock and BM lock to replace the original >> >> global >> >> > lock >> >> > 3. It's useful to improve the performance, since some operations just >> >> need >> >> > to hold FS lock or BM lock instead of the global lock >> >> > 4. This feature is turned off by default, you can enable it by setting >> >> > dfs.namenode.lock.model.provider.class to >> >> > org.apache.hadoop.hdfs.server.namenode.fgl.FineGrainedFSNamesystemLock >> >> > 5. This phase is very import for the ongoing development of the entire >> >> FGL >> >> > >> >> > Here I would like to express my special thanks to @kokonguyen191 and >> >> > @yuanboliu for their contributions. And you are also welcome to join us >> >> > and complete it together. >> >> > >> >> > >> >> > On Wed, 24 Apr 2024 at 14:54, ZanderXu wrote: >> >> > >> >> > > Hi everyone >> >> > > >> >> > > All subtasks of the first phase of the FGL have been completed and I >> >> plan >> >> > > to merge them into the trunk and start the second phase based on the >> >> trunk. >> >> > > >> >> > > Here is the PR that used to merge the first phases into trunk: >> >> > > https://github.com/apache/hadoop/pull/6762 >> >> > > Here is the ticket: https://issues.apache.org/jira/browse/HDFS-17384 >> >> > > >> >> > > I hope you can help to review this PR when you are available and give >> >> some >> >> > > ideas. >> >> > > >> >> > > >> >> > > HDFS-17385 <https://issues.apache.org/jira/browse/HDFS-17385> is >> >> used for >> >> > > the second phase and I have created some subtasks to describe >> >> solutions for >> >> > > some problems, such as: snapshot, getListing, quota. >> >> > > You are welcome to join us to complete it together. >> >> > > >> >> > > >> >> > > -- Forwarded message - >> >> > > From: Zengqiang XU >> >> > > Date: Fri, 2 Feb 2024 at 11:07 >> >> > > Subject: Discussion about NameNode Fine-grained locking >> >> > > To: >> >> > > Cc: Zengqiang XU >> >> > > >> >> > > >> >> > > Hi everyone >> >> > > >> >> > > I have started a discussion about NameNode Fine-grained Locking to >> >> improve >> >> > > performance of write operations in NameNode. >> >> > > >> >> > > I started this discussion again for serval main reasons: >> >> > > 1. We have implemented it and gained nearly 7x performance >> >> improvement in >> >> > > our prod environment >> >> > > 2. Many other companies made similar improvements based on their >> >> internal >> >> > > branch. >> >> > > 3. This topic has been discussed for a long time, but still without >> >> any >> >> > > results. >> >> > > >> >> > > I hope we can push this important improvement in the community so >> >> that all >> >> > > end-users can enjoy this significant improvement. >> >> > > >> >> > > I'd really appreciate you can join in and work with me to push this >> >> > > feature forward. >> >> > > >> >> > > Thanks very much. >> >> > > >> >> > > Ticket: HDFS-17366 <https://issues.apache.org/jira/browse/HDFS-17366> >> >> > > Design: NameNode Fine-grained locking based on directory tree >> >> > > < >> >> https://docs.google.com/document/d/1X499gHxT0WSU1fj8uo4RuF3GqKxWkWXznXx4tspTBLY/edit?usp=sharing >> >> > >> >> > > >> >> >> >> - >> >> To unsubscribe, e-mail: private-unsubscr...@hadoop.apache.org >> >> For additional commands, e-mail: private-h...@hadoop.apache.org >> >> >> >> - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: Discussion about NameNode Fine-grained locking
nd phase based on the > >> trunk. > >> > > > >> > > Here is the PR that used to merge the first phases into trunk: > >> > > https://github.com/apache/hadoop/pull/6762 > >> > > Here is the ticket: > https://issues.apache.org/jira/browse/HDFS-17384 > >> > > > >> > > I hope you can help to review this PR when you are available and > give > >> some > >> > > ideas. > >> > > > >> > > > >> > > HDFS-17385 <https://issues.apache.org/jira/browse/HDFS-17385> is > >> used for > >> > > the second phase and I have created some subtasks to describe > >> solutions for > >> > > some problems, such as: snapshot, getListing, quota. > >> > > You are welcome to join us to complete it together. > >> > > > >> > > > >> > > -- Forwarded message - > >> > > From: Zengqiang XU > >> > > Date: Fri, 2 Feb 2024 at 11:07 > >> > > Subject: Discussion about NameNode Fine-grained locking > >> > > To: > >> > > Cc: Zengqiang XU > >> > > > >> > > > >> > > Hi everyone > >> > > > >> > > I have started a discussion about NameNode Fine-grained Locking to > >> improve > >> > > performance of write operations in NameNode. > >> > > > >> > > I started this discussion again for serval main reasons: > >> > > 1. We have implemented it and gained nearly 7x performance > >> improvement in > >> > > our prod environment > >> > > 2. Many other companies made similar improvements based on their > >> internal > >> > > branch. > >> > > 3. This topic has been discussed for a long time, but still without > >> any > >> > > results. > >> > > > >> > > I hope we can push this important improvement in the community so > >> that all > >> > > end-users can enjoy this significant improvement. > >> > > > >> > > I'd really appreciate you can join in and work with me to push this > >> > > feature forward. > >> > > > >> > > Thanks very much. > >> > > > >> > > Ticket: HDFS-17366 < > https://issues.apache.org/jira/browse/HDFS-17366> > >> > > Design: NameNode Fine-grained locking based on directory tree > >> > > < > >> > https://docs.google.com/document/d/1X499gHxT0WSU1fj8uo4RuF3GqKxWkWXznXx4tspTBLY/edit?usp=sharing > >> > > >> > > > >> > >> - > >> To unsubscribe, e-mail: private-unsubscr...@hadoop.apache.org > >> For additional commands, e-mail: private-h...@hadoop.apache.org > >> > >> >
Re: Discussion about NameNode Fine-grained locking
We have created a "hdfs-fgl" channel in slack that you can join to discuss FGL efficiently. On Fri, 26 Apr 2024 at 14:56, Zengqiang XU wrote: > Thanks Shilun for your response: > > 1. This is a big and very useful feature, so it really needs more > developers to get on board. > 2. This fine grained lock has been implemented based on internal branches > and has gained benefits by many companies, such as: Meituan, Kuaishou, > Bytedance, etc. But it has not been contributed to the community due to > various reasons, such as there is a big difference between the version of > the internal branch and the community trunk branch, the internal branch may > ignore some functions to make FGL clear, and the contribution needs a lot > of work and will take many times. It means that this solution has already > been practiced in their prod environment. We have also practiced it in our > prod environment and gained benefits, and we are also willing to spend a > lot of time contributing to the community. > 3. Regarding the benchmark testing, we don't need to pay more attention to > whether the performance is improved by 5 times, 10 times or 20 times, > because there are too many factors that affect it. > 4. As I described above, this solution is already being practiced by many > companies. Right now, we just need to think about how to implement it with > high quality and more comprehensively. > 5. I firmly believe that all problems can be solved as long as the overall > solution is right. > 6. I can spend a lot of time leading the promotion of this entire feature > and I hope more people can join us in promoting it. > 7. You are always welcome to raise your concerns. > > > Thanks Shilun again, I hope you can help review designs and PRs. Thanks > > On Fri, 26 Apr 2024 at 08:00, slfan1989 wrote: > >> Thank you for your hard work! This is a very meaningful improvement, and >> from the design document, we can see a significant increase in HDFS >> read/write throughput. >> >> I am happy to see the progress made on HDFS-17384. >> >> However, I still have some concerns, which roughly involve the following >> aspects: >> >> 1. While ZanderXu and Hui Fei have deep expertise in HDFS and are >> familiar with related development details, we still need more community >> member to review the code to ensure that the relevant upgrades meet >> expectations. >> >> 2. We need more details on benchmarks to ensure that test results can be >> reproduced and to allow more community member to participate in the testing >> process. >> >> Looking forward to everything going smoothly in the future. >> >> Best Regards, >> - Shilun Fan. >> >> On Wed, Apr 24, 2024 at 3:51 PM Xiaoqiao He >> wrote: >> >>> cc private@h.a.o. >>> >>> On Wed, Apr 24, 2024 at 3:35 PM ZanderXu wrote: >>> > >>> > Here are some summaries about the first phase: >>> > 1. There are no big changes in this phase >>> > 2. This phase just uses FS lock and BM lock to replace the original >>> global >>> > lock >>> > 3. It's useful to improve the performance, since some operations just >>> need >>> > to hold FS lock or BM lock instead of the global lock >>> > 4. This feature is turned off by default, you can enable it by setting >>> > dfs.namenode.lock.model.provider.class to >>> > org.apache.hadoop.hdfs.server.namenode.fgl.FineGrainedFSNamesystemLock >>> > 5. This phase is very import for the ongoing development of the entire >>> FGL >>> > >>> > Here I would like to express my special thanks to @kokonguyen191 and >>> > @yuanboliu for their contributions. And you are also welcome to join >>> us >>> > and complete it together. >>> > >>> > >>> > On Wed, 24 Apr 2024 at 14:54, ZanderXu wrote: >>> > >>> > > Hi everyone >>> > > >>> > > All subtasks of the first phase of the FGL have been completed and I >>> plan >>> > > to merge them into the trunk and start the second phase based on the >>> trunk. >>> > > >>> > > Here is the PR that used to merge the first phases into trunk: >>> > > https://github.com/apache/hadoop/pull/6762 >>> > > Here is the ticket: https://issues.apache.org/jira/browse/HDFS-17384 >>> > > >>> > > I hope you can help to review this PR when you are available and >>> give some >>> > > ideas. >>> > > >>> > >
Re: Discussion about NameNode Fine-grained locking
Thanks Shilun for your response: 1. This is a big and very useful feature, so it really needs more developers to get on board. 2. This fine grained lock has been implemented based on internal branches and has gained benefits by many companies, such as: Meituan, Kuaishou, Bytedance, etc. But it has not been contributed to the community due to various reasons, such as there is a big difference between the version of the internal branch and the community trunk branch, the internal branch may ignore some functions to make FGL clear, and the contribution needs a lot of work and will take many times. It means that this solution has already been practiced in their prod environment. We have also practiced it in our prod environment and gained benefits, and we are also willing to spend a lot of time contributing to the community. 3. Regarding the benchmark testing, we don't need to pay more attention to whether the performance is improved by 5 times, 10 times or 20 times, because there are too many factors that affect it. 4. As I described above, this solution is already being practiced by many companies. Right now, we just need to think about how to implement it with high quality and more comprehensively. 5. I firmly believe that all problems can be solved as long as the overall solution is right. 6. I can spend a lot of time leading the promotion of this entire feature and I hope more people can join us in promoting it. 7. You are always welcome to raise your concerns. Thanks Shilun again, I hope you can help review designs and PRs. Thanks On Fri, 26 Apr 2024 at 08:00, slfan1989 wrote: > Thank you for your hard work! This is a very meaningful improvement, and > from the design document, we can see a significant increase in HDFS > read/write throughput. > > I am happy to see the progress made on HDFS-17384. > > However, I still have some concerns, which roughly involve the following > aspects: > > 1. While ZanderXu and Hui Fei have deep expertise in HDFS and are familiar > with related development details, we still need more community member to > review the code to ensure that the relevant upgrades meet expectations. > > 2. We need more details on benchmarks to ensure that test results can be > reproduced and to allow more community member to participate in the testing > process. > > Looking forward to everything going smoothly in the future. > > Best Regards, > - Shilun Fan. > > On Wed, Apr 24, 2024 at 3:51 PM Xiaoqiao He wrote: > >> cc private@h.a.o. >> >> On Wed, Apr 24, 2024 at 3:35 PM ZanderXu wrote: >> > >> > Here are some summaries about the first phase: >> > 1. There are no big changes in this phase >> > 2. This phase just uses FS lock and BM lock to replace the original >> global >> > lock >> > 3. It's useful to improve the performance, since some operations just >> need >> > to hold FS lock or BM lock instead of the global lock >> > 4. This feature is turned off by default, you can enable it by setting >> > dfs.namenode.lock.model.provider.class to >> > org.apache.hadoop.hdfs.server.namenode.fgl.FineGrainedFSNamesystemLock >> > 5. This phase is very import for the ongoing development of the entire >> FGL >> > >> > Here I would like to express my special thanks to @kokonguyen191 and >> > @yuanboliu for their contributions. And you are also welcome to join us >> > and complete it together. >> > >> > >> > On Wed, 24 Apr 2024 at 14:54, ZanderXu wrote: >> > >> > > Hi everyone >> > > >> > > All subtasks of the first phase of the FGL have been completed and I >> plan >> > > to merge them into the trunk and start the second phase based on the >> trunk. >> > > >> > > Here is the PR that used to merge the first phases into trunk: >> > > https://github.com/apache/hadoop/pull/6762 >> > > Here is the ticket: https://issues.apache.org/jira/browse/HDFS-17384 >> > > >> > > I hope you can help to review this PR when you are available and give >> some >> > > ideas. >> > > >> > > >> > > HDFS-17385 <https://issues.apache.org/jira/browse/HDFS-17385> is >> used for >> > > the second phase and I have created some subtasks to describe >> solutions for >> > > some problems, such as: snapshot, getListing, quota. >> > > You are welcome to join us to complete it together. >> > > >> > > >> > > -- Forwarded message - >> > > From: Zengqiang XU >> > > Date: Fri, 2 Feb 2024 at 11:07 >> > > Subject: Discussion about NameNode
Re: Discussion about NameNode Fine-grained locking
Thank you for your hard work! This is a very meaningful improvement, and from the design document, we can see a significant increase in HDFS read/write throughput. I am happy to see the progress made on HDFS-17384. However, I still have some concerns, which roughly involve the following aspects: 1. While ZanderXu and Hui Fei have deep expertise in HDFS and are familiar with related development details, we still need more community member to review the code to ensure that the relevant upgrades meet expectations. 2. We need more details on benchmarks to ensure that test results can be reproduced and to allow more community member to participate in the testing process. Looking forward to everything going smoothly in the future. Best Regards, - Shilun Fan. On Wed, Apr 24, 2024 at 3:51 PM Xiaoqiao He wrote: > cc private@h.a.o. > > On Wed, Apr 24, 2024 at 3:35 PM ZanderXu wrote: > > > > Here are some summaries about the first phase: > > 1. There are no big changes in this phase > > 2. This phase just uses FS lock and BM lock to replace the original > global > > lock > > 3. It's useful to improve the performance, since some operations just > need > > to hold FS lock or BM lock instead of the global lock > > 4. This feature is turned off by default, you can enable it by setting > > dfs.namenode.lock.model.provider.class to > > org.apache.hadoop.hdfs.server.namenode.fgl.FineGrainedFSNamesystemLock > > 5. This phase is very import for the ongoing development of the entire > FGL > > > > Here I would like to express my special thanks to @kokonguyen191 and > > @yuanboliu for their contributions. And you are also welcome to join us > > and complete it together. > > > > > > On Wed, 24 Apr 2024 at 14:54, ZanderXu wrote: > > > > > Hi everyone > > > > > > All subtasks of the first phase of the FGL have been completed and I > plan > > > to merge them into the trunk and start the second phase based on the > trunk. > > > > > > Here is the PR that used to merge the first phases into trunk: > > > https://github.com/apache/hadoop/pull/6762 > > > Here is the ticket: https://issues.apache.org/jira/browse/HDFS-17384 > > > > > > I hope you can help to review this PR when you are available and give > some > > > ideas. > > > > > > > > > HDFS-17385 <https://issues.apache.org/jira/browse/HDFS-17385> is used > for > > > the second phase and I have created some subtasks to describe > solutions for > > > some problems, such as: snapshot, getListing, quota. > > > You are welcome to join us to complete it together. > > > > > > > > > -- Forwarded message - > > > From: Zengqiang XU > > > Date: Fri, 2 Feb 2024 at 11:07 > > > Subject: Discussion about NameNode Fine-grained locking > > > To: > > > Cc: Zengqiang XU > > > > > > > > > Hi everyone > > > > > > I have started a discussion about NameNode Fine-grained Locking to > improve > > > performance of write operations in NameNode. > > > > > > I started this discussion again for serval main reasons: > > > 1. We have implemented it and gained nearly 7x performance improvement > in > > > our prod environment > > > 2. Many other companies made similar improvements based on their > internal > > > branch. > > > 3. This topic has been discussed for a long time, but still without any > > > results. > > > > > > I hope we can push this important improvement in the community so that > all > > > end-users can enjoy this significant improvement. > > > > > > I'd really appreciate you can join in and work with me to push this > > > feature forward. > > > > > > Thanks very much. > > > > > > Ticket: HDFS-17366 <https://issues.apache.org/jira/browse/HDFS-17366> > > > Design: NameNode Fine-grained locking based on directory tree > > > < > https://docs.google.com/document/d/1X499gHxT0WSU1fj8uo4RuF3GqKxWkWXznXx4tspTBLY/edit?usp=sharing > > > > > > > - > To unsubscribe, e-mail: private-unsubscr...@hadoop.apache.org > For additional commands, e-mail: private-h...@hadoop.apache.org > >
Re: Discussion about NameNode Fine-grained locking
cc private@h.a.o. On Wed, Apr 24, 2024 at 3:35 PM ZanderXu wrote: > > Here are some summaries about the first phase: > 1. There are no big changes in this phase > 2. This phase just uses FS lock and BM lock to replace the original global > lock > 3. It's useful to improve the performance, since some operations just need > to hold FS lock or BM lock instead of the global lock > 4. This feature is turned off by default, you can enable it by setting > dfs.namenode.lock.model.provider.class to > org.apache.hadoop.hdfs.server.namenode.fgl.FineGrainedFSNamesystemLock > 5. This phase is very import for the ongoing development of the entire FGL > > Here I would like to express my special thanks to @kokonguyen191 and > @yuanboliu for their contributions. And you are also welcome to join us > and complete it together. > > > On Wed, 24 Apr 2024 at 14:54, ZanderXu wrote: > > > Hi everyone > > > > All subtasks of the first phase of the FGL have been completed and I plan > > to merge them into the trunk and start the second phase based on the trunk. > > > > Here is the PR that used to merge the first phases into trunk: > > https://github.com/apache/hadoop/pull/6762 > > Here is the ticket: https://issues.apache.org/jira/browse/HDFS-17384 > > > > I hope you can help to review this PR when you are available and give some > > ideas. > > > > > > HDFS-17385 <https://issues.apache.org/jira/browse/HDFS-17385> is used for > > the second phase and I have created some subtasks to describe solutions for > > some problems, such as: snapshot, getListing, quota. > > You are welcome to join us to complete it together. > > > > > > -- Forwarded message - > > From: Zengqiang XU > > Date: Fri, 2 Feb 2024 at 11:07 > > Subject: Discussion about NameNode Fine-grained locking > > To: > > Cc: Zengqiang XU > > > > > > Hi everyone > > > > I have started a discussion about NameNode Fine-grained Locking to improve > > performance of write operations in NameNode. > > > > I started this discussion again for serval main reasons: > > 1. We have implemented it and gained nearly 7x performance improvement in > > our prod environment > > 2. Many other companies made similar improvements based on their internal > > branch. > > 3. This topic has been discussed for a long time, but still without any > > results. > > > > I hope we can push this important improvement in the community so that all > > end-users can enjoy this significant improvement. > > > > I'd really appreciate you can join in and work with me to push this > > feature forward. > > > > Thanks very much. > > > > Ticket: HDFS-17366 <https://issues.apache.org/jira/browse/HDFS-17366> > > Design: NameNode Fine-grained locking based on directory tree > > <https://docs.google.com/document/d/1X499gHxT0WSU1fj8uo4RuF3GqKxWkWXznXx4tspTBLY/edit?usp=sharing> > > - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: Discussion about NameNode Fine-grained locking
Here are some summaries about the first phase: 1. There are no big changes in this phase 2. This phase just uses FS lock and BM lock to replace the original global lock 3. It's useful to improve the performance, since some operations just need to hold FS lock or BM lock instead of the global lock 4. This feature is turned off by default, you can enable it by setting dfs.namenode.lock.model.provider.class to org.apache.hadoop.hdfs.server.namenode.fgl.FineGrainedFSNamesystemLock 5. This phase is very import for the ongoing development of the entire FGL Here I would like to express my special thanks to @kokonguyen191 and @yuanboliu for their contributions. And you are also welcome to join us and complete it together. On Wed, 24 Apr 2024 at 14:54, ZanderXu wrote: > Hi everyone > > All subtasks of the first phase of the FGL have been completed and I plan > to merge them into the trunk and start the second phase based on the trunk. > > Here is the PR that used to merge the first phases into trunk: > https://github.com/apache/hadoop/pull/6762 > Here is the ticket: https://issues.apache.org/jira/browse/HDFS-17384 > > I hope you can help to review this PR when you are available and give some > ideas. > > > HDFS-17385 <https://issues.apache.org/jira/browse/HDFS-17385> is used for > the second phase and I have created some subtasks to describe solutions for > some problems, such as: snapshot, getListing, quota. > You are welcome to join us to complete it together. > > > -- Forwarded message - > From: Zengqiang XU > Date: Fri, 2 Feb 2024 at 11:07 > Subject: Discussion about NameNode Fine-grained locking > To: > Cc: Zengqiang XU > > > Hi everyone > > I have started a discussion about NameNode Fine-grained Locking to improve > performance of write operations in NameNode. > > I started this discussion again for serval main reasons: > 1. We have implemented it and gained nearly 7x performance improvement in > our prod environment > 2. Many other companies made similar improvements based on their internal > branch. > 3. This topic has been discussed for a long time, but still without any > results. > > I hope we can push this important improvement in the community so that all > end-users can enjoy this significant improvement. > > I'd really appreciate you can join in and work with me to push this > feature forward. > > Thanks very much. > > Ticket: HDFS-17366 <https://issues.apache.org/jira/browse/HDFS-17366> > Design: NameNode Fine-grained locking based on directory tree > <https://docs.google.com/document/d/1X499gHxT0WSU1fj8uo4RuF3GqKxWkWXznXx4tspTBLY/edit?usp=sharing> >
Fwd: Discussion about NameNode Fine-grained locking
Hi everyone All subtasks of the first phase of the FGL have been completed and I plan to merge them into the trunk and start the second phase based on the trunk. Here is the PR that used to merge the first phases into trunk: https://github.com/apache/hadoop/pull/6762 Here is the ticket: https://issues.apache.org/jira/browse/HDFS-17384 I hope you can help to review this PR when you are available and give some ideas. HDFS-17385 <https://issues.apache.org/jira/browse/HDFS-17385> is used for the second phase and I have created some subtasks to describe solutions for some problems, such as: snapshot, getListing, quota. You are welcome to join us to complete it together. -- Forwarded message - From: Zengqiang XU Date: Fri, 2 Feb 2024 at 11:07 Subject: Discussion about NameNode Fine-grained locking To: Cc: Zengqiang XU Hi everyone I have started a discussion about NameNode Fine-grained Locking to improve performance of write operations in NameNode. I started this discussion again for serval main reasons: 1. We have implemented it and gained nearly 7x performance improvement in our prod environment 2. Many other companies made similar improvements based on their internal branch. 3. This topic has been discussed for a long time, but still without any results. I hope we can push this important improvement in the community so that all end-users can enjoy this significant improvement. I'd really appreciate you can join in and work with me to push this feature forward. Thanks very much. Ticket: HDFS-17366 <https://issues.apache.org/jira/browse/HDFS-17366> Design: NameNode Fine-grained locking based on directory tree <https://docs.google.com/document/d/1X499gHxT0WSU1fj8uo4RuF3GqKxWkWXznXx4tspTBLY/edit?usp=sharing>
Re: Discussion about NameNode Fine-grained locking
t; > > > >> `108K QPS`? If true, would you mind sharing STW time cost? > > > > >> c. Is this deployed in your internal cluster now? If true, any > > > > >> performance > > > > >> benefit differences compare to the > > > > >> benchmark? > > > > >> d. This is one huge feature IMO, If discussion passes, suggest > > > creating > > > > a > > > > >> single branch to develop and follow-up > > > > >> works. > > > > >> > > > > >> Thanks again for this meaningful proposal. > > > > >> > > > > >> Best Regards, > > > > >> - He Xiaoqiao > > > > >> > > > > >> > > > > >> On Tue, Feb 20, 2024 at 5:38 PM Yuanbo Liu > > > > > wrote: > > > > >> > > > > >> > Nice to see this feature brought up. We've implemented this > > feature > > > > >> > internally and gained significant performance improvement. I'll > be > > > > glad > > > > >> to > > > > >> > work on some jiras if necessary. > > > > >> > > > > > >> > > > > > >> > On Tue, Feb 20, 2024 at 4:41 PM ZanderXu > > > wrote: > > > > >> > > > > > >> > > Thank you everyone for reviewing this ticket. > > > > >> > > > > > > >> > > I think if there are no problems with the goal and the overall > > > > >> solution, > > > > >> > we > > > > >> > > are ready to push this ticket forward and I will create some > > > > detailed > > > > >> > > sub-tasks for this ticket. > > > > >> > > > > > > >> > > I will split this project into three milestones to make this > > > project > > > > >> > > cleaner for review and merge. > > > > >> > > Milestone 1: Replacing the current global lock with two locks, > > > > global > > > > >> FS > > > > >> > > lock and global BM lock. End-user can choose which locking > mode > > to > > > > use > > > > >> > > through configuration. > > > > >> > > Milestone 2: Replacing the global FS write lock with directory > > > > >> tree-based > > > > >> > > fine-grained lock. > > > > >> > > Milestone 3: Replacing the global BM lock with directory > > > tree-based > > > > >> > > fine-grained lock. > > > > >> > > > > > > >> > > Each milestone can be merged into the trunk branch in time, > > which > > > > has > > > > >> > > multiple benefits: > > > > >> > > 1. Phased performance improvements can be quickly used by > > everyone > > > > >> > > 2. All developers can better understand the implementation > ideas > > > of > > > > >> the > > > > >> > > fine-grained locking mechanism as soon as possible > > > > >> > > 3. Each milestone is developed based on the latest trunk > branch > > to > > > > >> reduce > > > > >> > > conflicts > > > > >> > > > > > > >> > > If you have any concerns, please feel free to discuss them > > > together. > > > > >> > > I hope you can join us to push this project forward together, > > > > thanks. > > > > >> > > > > > > >> > > > > > > >> > > On Mon, 5 Feb 2024 at 11:33, haiyang hu < > haiyang87...@gmail.com > > > > > > > >> wrote: > > > > >> > > > > > > >> > > > Thank you for raising the issue of this long-standing > > > bottleneck, > > > > >> this > > > > >> > > > will be a very important improvement! > > > > >> > > > > > > > >> > > > Hopefully can participate and push forward together. > > > > >> > > > > > > > >> > > > Best Regards~ > > > > >> > > > > > > > >> > > > Brahma Reddy Battula 于2024年2月3日周六 > > 00
Re: Discussion about NameNode Fine-grained locking
ady to push this ticket forward and I will create some > > > detailed > > > >> > > sub-tasks for this ticket. > > > >> > > > > > >> > > I will split this project into three milestones to make this > > project > > > >> > > cleaner for review and merge. > > > >> > > Milestone 1: Replacing the current global lock with two locks, > > > global > > > >> FS > > > >> > > lock and global BM lock. End-user can choose which locking mode > to > > > use > > > >> > > through configuration. > > > >> > > Milestone 2: Replacing the global FS write lock with directory > > > >> tree-based > > > >> > > fine-grained lock. > > > >> > > Milestone 3: Replacing the global BM lock with directory > > tree-based > > > >> > > fine-grained lock. > > > >> > > > > > >> > > Each milestone can be merged into the trunk branch in time, > which > > > has > > > >> > > multiple benefits: > > > >> > > 1. Phased performance improvements can be quickly used by > everyone > > > >> > > 2. All developers can better understand the implementation ideas > > of > > > >> the > > > >> > > fine-grained locking mechanism as soon as possible > > > >> > > 3. Each milestone is developed based on the latest trunk branch > to > > > >> reduce > > > >> > > conflicts > > > >> > > > > > >> > > If you have any concerns, please feel free to discuss them > > together. > > > >> > > I hope you can join us to push this project forward together, > > > thanks. > > > >> > > > > > >> > > > > > >> > > On Mon, 5 Feb 2024 at 11:33, haiyang hu > > > > >> wrote: > > > >> > > > > > >> > > > Thank you for raising the issue of this long-standing > > bottleneck, > > > >> this > > > >> > > > will be a very important improvement! > > > >> > > > > > > >> > > > Hopefully can participate and push forward together. > > > >> > > > > > > >> > > > Best Regards~ > > > >> > > > > > > >> > > > Brahma Reddy Battula 于2024年2月3日周六 > 00:40写道： > > > >> > > > > > > >> > > >> Thanks for bringing this and considering all the history > around > > > >> this. > > > >> > > >> One of the outstanding bottleneck(global lock) from a long > > time. > > > >> > > >> > > > >> > > >> Hopefully we can push forward this time. > > > >> > > >> > > > >> > > >> > > > >> > > >> On Fri, Feb 2, 2024 at 12:23 PM Hui Fei < > feihui.u...@gmail.com > > > > > > >> > wrote: > > > >> > > >> > > > >> > > >> > Thanks for driving this. It's very meaningful. The > > performance > > > >> > > >> improvement > > > >> > > >> > looks very good. > > > >> > > >> > > > > >> > > >> > Many users are facing the write performance issue. As far > as > > I > > > >> know, > > > >> > > >> some > > > >> > > >> > companies already implemented the similar idea on their > > > internal > > > >> > > >> branches. > > > >> > > >> > But the internal branch is very different from the > community > > > >> one. So > > > >> > > >> it's > > > >> > > >> > very hard to be in sync with the community. If this > > improvement > > > >> can > > > >> > be > > > >> > > >> > involved in the community, that would be great to both > > end-user > > > >> and > > > >> > > the > > > >> > > >> > community. > > > >> > > >> > > > > >> > > >> > It is very worth doing. > > > >> > > >> > > > > >> > > >> > Zengqiang XU 于2024年2月2日周五 11:07写道： > > > >> > > >> > > > > >> > > >> > > Hi everyone > > > >> > > >> > > > > > >> > > >> > > I have started a discussion about NameNode Fine-grained > > > >> Locking to > > > >> > > >> > improve > > > >> > > >> > > performance of write operations in NameNode. > > > >> > > >> > > > > > >> > > >> > > I started this discussion again for serval main reasons: > > > >> > > >> > > 1. We have implemented it and gained nearly 7x > performance > > > >> > > >> improvement in > > > >> > > >> > > our prod environment > > > >> > > >> > > 2. Many other companies made similar improvements based > on > > > >> their > > > >> > > >> internal > > > >> > > >> > > branch. > > > >> > > >> > > 3. This topic has been discussed for a long time, but > still > > > >> > without > > > >> > > >> any > > > >> > > >> > > results. > > > >> > > >> > > > > > >> > > >> > > I hope we can push this important improvement in the > > > community > > > >> so > > > >> > > that > > > >> > > >> > all > > > >> > > >> > > end-users can enjoy this significant improvement. > > > >> > > >> > > > > > >> > > >> > > I'd really appreciate you can join in and work with me to > > > push > > > >> > this > > > >> > > >> > feature > > > >> > > >> > > forward. > > > >> > > >> > > > > > >> > > >> > > Thanks very much. > > > >> > > >> > > > > > >> > > >> > > Ticket: HDFS-17366 < > > > >> > > https://issues.apache.org/jira/browse/HDFS-17366> > > > >> > > >> > > Design: NameNode Fine-grained locking based on directory > > tree > > > >> > > >> > > < > > > >> > > >> > > > > > >> > > >> > > > > >> > > >> > > > >> > > > > > >> > > > > >> > > > > > > https://docs.google.com/document/d/1bVBQcI4jfzS0UrczB7UhsrQTXmrERGvBV-a9W3HCCjk/edit?usp=sharing > > > >> > > >> > > > > > > >> > > >> > > > > > >> > > >> > > > > >> > > >> > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > > > > >
Re: Discussion about NameNode Fine-grained locking
; > Milestone 3: Replacing the global BM lock with directory > tree-based > > >> > > fine-grained lock. > > >> > > > > >> > > Each milestone can be merged into the trunk branch in time, which > > has > > >> > > multiple benefits: > > >> > > 1. Phased performance improvements can be quickly used by everyone > > >> > > 2. All developers can better understand the implementation ideas > of > > >> the > > >> > > fine-grained locking mechanism as soon as possible > > >> > > 3. Each milestone is developed based on the latest trunk branch to > > >> reduce > > >> > > conflicts > > >> > > > > >> > > If you have any concerns, please feel free to discuss them > together. > > >> > > I hope you can join us to push this project forward together, > > thanks. > > >> > > > > >> > > > > >> > > On Mon, 5 Feb 2024 at 11:33, haiyang hu > > >> wrote: > > >> > > > > >> > > > Thank you for raising the issue of this long-standing > bottleneck, > > >> this > > >> > > > will be a very important improvement! > > >> > > > > > >> > > > Hopefully can participate and push forward together. > > >> > > > > > >> > > > Best Regards~ > > >> > > > > > >> > > > Brahma Reddy Battula 于2024年2月3日周六 00:40写道： > > >> > > > > > >> > > >> Thanks for bringing this and considering all the history around > > >> this. > > >> > > >> One of the outstanding bottleneck(global lock) from a long > time. > > >> > > >> > > >> > > >> Hopefully we can push forward this time. > > >> > > >> > > >> > > >> > > >> > > >> On Fri, Feb 2, 2024 at 12:23 PM Hui Fei > > > >> > wrote: > > >> > > >> > > >> > > >> > Thanks for driving this. It's very meaningful. The > performance > > >> > > >> improvement > > >> > > >> > looks very good. > > >> > > >> > > > >> > > >> > Many users are facing the write performance issue. As far as > I > > >> know, > > >> > > >> some > > >> > > >> > companies already implemented the similar idea on their > > internal > > >> > > >> branches. > > >> > > >> > But the internal branch is very different from the community > > >> one. So > > >> > > >> it's > > >> > > >> > very hard to be in sync with the community. If this > improvement > > >> can > > >> > be > > >> > > >> > involved in the community, that would be great to both > end-user > > >> and > > >> > > the > > >> > > >> > community. > > >> > > >> > > > >> > > >> > It is very worth doing. > > >> > > >> > > > >> > > >> > Zengqiang XU 于2024年2月2日周五 11:07写道： > > >> > > >> > > > >> > > >> > > Hi everyone > > >> > > >> > > > > >> > > >> > > I have started a discussion about NameNode Fine-grained > > >> Locking to > > >> > > >> > improve > > >> > > >> > > performance of write operations in NameNode. > > >> > > >> > > > > >> > > >> > > I started this discussion again for serval main reasons: > > >> > > >> > > 1. We have implemented it and gained nearly 7x performance > > >> > > >> improvement in > > >> > > >> > > our prod environment > > >> > > >> > > 2. Many other companies made similar improvements based on > > >> their > > >> > > >> internal > > >> > > >> > > branch. > > >> > > >> > > 3. This topic has been discussed for a long time, but still > > >> > without > > >> > > >> any > > >> > > >> > > results. > > >> > > >> > > > > >> > > >> > > I hope we can push this important improvement in the > > community > > >> so > > >> > > that > > >> > > >> > all > > >> > > >> > > end-users can enjoy this significant improvement. > > >> > > >> > > > > >> > > >> > > I'd really appreciate you can join in and work with me to > > push > > >> > this > > >> > > >> > feature > > >> > > >> > > forward. > > >> > > >> > > > > >> > > >> > > Thanks very much. > > >> > > >> > > > > >> > > >> > > Ticket: HDFS-17366 < > > >> > > https://issues.apache.org/jira/browse/HDFS-17366> > > >> > > >> > > Design: NameNode Fine-grained locking based on directory > tree > > >> > > >> > > < > > >> > > >> > > > > >> > > >> > > > >> > > >> > > >> > > > > >> > > > >> > > > https://docs.google.com/document/d/1bVBQcI4jfzS0UrczB7UhsrQTXmrERGvBV-a9W3HCCjk/edit?usp=sharing > > >> > > >> > > > > > >> > > >> > > > > >> > > >> > > > >> > > >> > > >> > > > > > >> > > > > >> > > > >> > > > > > >
Re: Discussion about NameNode Fine-grained locking
scuss them together. > >> > > I hope you can join us to push this project forward together, > thanks. > >> > > > >> > > > >> > > On Mon, 5 Feb 2024 at 11:33, haiyang hu > >> wrote: > >> > > > >> > > > Thank you for raising the issue of this long-standing bottleneck, > >> this > >> > > > will be a very important improvement! > >> > > > > >> > > > Hopefully can participate and push forward together. > >> > > > > >> > > > Best Regards~ > >> > > > > >> > > > Brahma Reddy Battula 于2024年2月3日周六 00:40写道： > >> > > > > >> > > >> Thanks for bringing this and considering all the history around > >> this. > >> > > >> One of the outstanding bottleneck(global lock) from a long time. > >> > > >> > >> > > >> Hopefully we can push forward this time. > >> > > >> > >> > > >> > >> > > >> On Fri, Feb 2, 2024 at 12:23 PM Hui Fei > >> > wrote: > >> > > >> > >> > > >> > Thanks for driving this. It's very meaningful. The performance > >> > > >> improvement > >> > > >> > looks very good. > >> > > >> > > >> > > >> > Many users are facing the write performance issue. As far as I > >> know, > >> > > >> some > >> > > >> > companies already implemented the similar idea on their > internal > >> > > >> branches. > >> > > >> > But the internal branch is very different from the community > >> one. So > >> > > >> it's > >> > > >> > very hard to be in sync with the community. If this improvement > >> can > >> > be > >> > > >> > involved in the community, that would be great to both end-user > >> and > >> > > the > >> > > >> > community. > >> > > >> > > >> > > >> > It is very worth doing. > >> > > >> > > >> > > >> > Zengqiang XU 于2024年2月2日周五 11:07写道： > >> > > >> > > >> > > >> > > Hi everyone > >> > > >> > > > >> > > >> > > I have started a discussion about NameNode Fine-grained > >> Locking to > >> > > >> > improve > >> > > >> > > performance of write operations in NameNode. > >> > > >> > > > >> > > >> > > I started this discussion again for serval main reasons: > >> > > >> > > 1. We have implemented it and gained nearly 7x performance > >> > > >> improvement in > >> > > >> > > our prod environment > >> > > >> > > 2. Many other companies made similar improvements based on > >> their > >> > > >> internal > >> > > >> > > branch. > >> > > >> > > 3. This topic has been discussed for a long time, but still > >> > without > >> > > >> any > >> > > >> > > results. > >> > > >> > > > >> > > >> > > I hope we can push this important improvement in the > community > >> so > >> > > that > >> > > >> > all > >> > > >> > > end-users can enjoy this significant improvement. > >> > > >> > > > >> > > >> > > I'd really appreciate you can join in and work with me to > push > >> > this > >> > > >> > feature > >> > > >> > > forward. > >> > > >> > > > >> > > >> > > Thanks very much. > >> > > >> > > > >> > > >> > > Ticket: HDFS-17366 < > >> > > https://issues.apache.org/jira/browse/HDFS-17366> > >> > > >> > > Design: NameNode Fine-grained locking based on directory tree > >> > > >> > > < > >> > > >> > > > >> > > >> > > >> > > >> > >> > > > >> > > >> > https://docs.google.com/document/d/1bVBQcI4jfzS0UrczB7UhsrQTXmrERGvBV-a9W3HCCjk/edit?usp=sharing > >> > > >> > > > > >> > > >> > > > >> > > >> > > >> > > >> > >> > > > > >> > > > >> > > >> > > >
Re: Discussion about NameNode Fine-grained locking
rom a long time. >> > > >> >> > > >> Hopefully we can push forward this time. >> > > >> >> > > >> >> > > >> On Fri, Feb 2, 2024 at 12:23 PM Hui Fei >> > wrote: >> > > >> >> > > >> > Thanks for driving this. It's very meaningful. The performance >> > > >> improvement >> > > >> > looks very good. >> > > >> > >> > > >> > Many users are facing the write performance issue. As far as I >> know, >> > > >> some >> > > >> > companies already implemented the similar idea on their internal >> > > >> branches. >> > > >> > But the internal branch is very different from the community >> one. So >> > > >> it's >> > > >> > very hard to be in sync with the community. If this improvement >> can >> > be >> > > >> > involved in the community, that would be great to both end-user >> and >> > > the >> > > >> > community. >> > > >> > >> > > >> > It is very worth doing. >> > > >> > >> > > >> > Zengqiang XU 于2024年2月2日周五 11:07写道： >> > > >> > >> > > >> > > Hi everyone >> > > >> > > >> > > >> > > I have started a discussion about NameNode Fine-grained >> Locking to >> > > >> > improve >> > > >> > > performance of write operations in NameNode. >> > > >> > > >> > > >> > > I started this discussion again for serval main reasons: >> > > >> > > 1. We have implemented it and gained nearly 7x performance >> > > >> improvement in >> > > >> > > our prod environment >> > > >> > > 2. Many other companies made similar improvements based on >> their >> > > >> internal >> > > >> > > branch. >> > > >> > > 3. This topic has been discussed for a long time, but still >> > without >> > > >> any >> > > >> > > results. >> > > >> > > >> > > >> > > I hope we can push this important improvement in the community >> so >> > > that >> > > >> > all >> > > >> > > end-users can enjoy this significant improvement. >> > > >> > > >> > > >> > > I'd really appreciate you can join in and work with me to push >> > this >> > > >> > feature >> > > >> > > forward. >> > > >> > > >> > > >> > > Thanks very much. >> > > >> > > >> > > >> > > Ticket: HDFS-17366 < >> > > https://issues.apache.org/jira/browse/HDFS-17366> >> > > >> > > Design: NameNode Fine-grained locking based on directory tree >> > > >> > > < >> > > >> > > >> > > >> > >> > > >> >> > > >> > >> https://docs.google.com/document/d/1bVBQcI4jfzS0UrczB7UhsrQTXmrERGvBV-a9W3HCCjk/edit?usp=sharing >> > > >> > > > >> > > >> > > >> > > >> > >> > > >> >> > > > >> > > >> > >> >
Re: Discussion about NameNode Fine-grained locking
Thanks for suggestions. Actually Started working on this improvement. And cut the development branch :) >From the proposal doc and the current reviewing work, seems that it doesn't touch the existing logic codes too much. It keeps the original logic there. @Yuanbo @Zengqiang XU Could you share any internal improvement info Xiaoqiao mentioned above? Xiaoqiao He 于2024年2月26日周一 19:50写道： > Thanks for this meaningful proposal. Some nit comments: > a. Snapshot, Symbolic link and reserved feature are not mentioned at the > design doc, should it be considered > or different to this core design? > b. For the benchmark result, what Read/Write request ratio? And do you meet > any GC issues when reaching > `108K QPS`? If true, would you mind sharing STW time cost? > c. Is this deployed in your internal cluster now? If true, any performance > benefit differences compare to the > benchmark? > d. This is one huge feature IMO, If discussion passes, suggest creating a > single branch to develop and follow-up > works. > > Thanks again for this meaningful proposal. > > Best Regards, > - He Xiaoqiao > > > On Tue, Feb 20, 2024 at 5:38 PM Yuanbo Liu wrote: > > > Nice to see this feature brought up. We've implemented this feature > > internally and gained significant performance improvement. I'll be glad > to > > work on some jiras if necessary. > > > > > > On Tue, Feb 20, 2024 at 4:41 PM ZanderXu wrote: > > > > > Thank you everyone for reviewing this ticket. > > > > > > I think if there are no problems with the goal and the overall > solution, > > we > > > are ready to push this ticket forward and I will create some detailed > > > sub-tasks for this ticket. > > > > > > I will split this project into three milestones to make this project > > > cleaner for review and merge. > > > Milestone 1: Replacing the current global lock with two locks, global > FS > > > lock and global BM lock. End-user can choose which locking mode to use > > > through configuration. > > > Milestone 2: Replacing the global FS write lock with directory > tree-based > > > fine-grained lock. > > > Milestone 3: Replacing the global BM lock with directory tree-based > > > fine-grained lock. > > > > > > Each milestone can be merged into the trunk branch in time, which has > > > multiple benefits: > > > 1. Phased performance improvements can be quickly used by everyone > > > 2. All developers can better understand the implementation ideas of the > > > fine-grained locking mechanism as soon as possible > > > 3. Each milestone is developed based on the latest trunk branch to > reduce > > > conflicts > > > > > > If you have any concerns, please feel free to discuss them together. > > > I hope you can join us to push this project forward together, thanks. > > > > > > > > > On Mon, 5 Feb 2024 at 11:33, haiyang hu > wrote: > > > > > > > Thank you for raising the issue of this long-standing bottleneck, > this > > > > will be a very important improvement! > > > > > > > > Hopefully can participate and push forward together. > > > > > > > > Best Regards~ > > > > > > > > Brahma Reddy Battula 于2024年2月3日周六 00:40写道： > > > > > > > >> Thanks for bringing this and considering all the history around > this. > > > >> One of the outstanding bottleneck(global lock) from a long time. > > > >> > > > >> Hopefully we can push forward this time. > > > >> > > > >> > > > >> On Fri, Feb 2, 2024 at 12:23 PM Hui Fei > > wrote: > > > >> > > > >> > Thanks for driving this. It's very meaningful. The performance > > > >> improvement > > > >> > looks very good. > > > >> > > > > >> > Many users are facing the write performance issue. As far as I > know, > > > >> some > > > >> > companies already implemented the similar idea on their internal > > > >> branches. > > > >> > But the internal branch is very different from the community one. > So > > > >> it's > > > >> > very hard to be in sync with the community. If this improvement > can > > be > > > >> > involved in the community, that would be great to both end-user > and > > > the > > > >> > community. > > > >> > > > > >&
Re: Discussion about NameNode Fine-grained locking
Thanks for this meaningful proposal. Some nit comments: a. Snapshot, Symbolic link and reserved feature are not mentioned at the design doc, should it be considered or different to this core design? b. For the benchmark result, what Read/Write request ratio? And do you meet any GC issues when reaching `108K QPS`? If true, would you mind sharing STW time cost? c. Is this deployed in your internal cluster now? If true, any performance benefit differences compare to the benchmark? d. This is one huge feature IMO, If discussion passes, suggest creating a single branch to develop and follow-up works. Thanks again for this meaningful proposal. Best Regards, - He Xiaoqiao On Tue, Feb 20, 2024 at 5:38 PM Yuanbo Liu wrote: > Nice to see this feature brought up. We've implemented this feature > internally and gained significant performance improvement. I'll be glad to > work on some jiras if necessary. > > > On Tue, Feb 20, 2024 at 4:41 PM ZanderXu wrote: > > > Thank you everyone for reviewing this ticket. > > > > I think if there are no problems with the goal and the overall solution, > we > > are ready to push this ticket forward and I will create some detailed > > sub-tasks for this ticket. > > > > I will split this project into three milestones to make this project > > cleaner for review and merge. > > Milestone 1: Replacing the current global lock with two locks, global FS > > lock and global BM lock. End-user can choose which locking mode to use > > through configuration. > > Milestone 2: Replacing the global FS write lock with directory tree-based > > fine-grained lock. > > Milestone 3: Replacing the global BM lock with directory tree-based > > fine-grained lock. > > > > Each milestone can be merged into the trunk branch in time, which has > > multiple benefits: > > 1. Phased performance improvements can be quickly used by everyone > > 2. All developers can better understand the implementation ideas of the > > fine-grained locking mechanism as soon as possible > > 3. Each milestone is developed based on the latest trunk branch to reduce > > conflicts > > > > If you have any concerns, please feel free to discuss them together. > > I hope you can join us to push this project forward together, thanks. > > > > > > On Mon, 5 Feb 2024 at 11:33, haiyang hu wrote: > > > > > Thank you for raising the issue of this long-standing bottleneck, this > > > will be a very important improvement! > > > > > > Hopefully can participate and push forward together. > > > > > > Best Regards~ > > > > > > Brahma Reddy Battula 于2024年2月3日周六 00:40写道： > > > > > >> Thanks for bringing this and considering all the history around this. > > >> One of the outstanding bottleneck(global lock) from a long time. > > >> > > >> Hopefully we can push forward this time. > > >> > > >> > > >> On Fri, Feb 2, 2024 at 12:23 PM Hui Fei > wrote: > > >> > > >> > Thanks for driving this. It's very meaningful. The performance > > >> improvement > > >> > looks very good. > > >> > > > >> > Many users are facing the write performance issue. As far as I know, > > >> some > > >> > companies already implemented the similar idea on their internal > > >> branches. > > >> > But the internal branch is very different from the community one. So > > >> it's > > >> > very hard to be in sync with the community. If this improvement can > be > > >> > involved in the community, that would be great to both end-user and > > the > > >> > community. > > >> > > > >> > It is very worth doing. > > >> > > > >> > Zengqiang XU 于2024年2月2日周五 11:07写道： > > >> > > > >> > > Hi everyone > > >> > > > > >> > > I have started a discussion about NameNode Fine-grained Locking to > > >> > improve > > >> > > performance of write operations in NameNode. > > >> > > > > >> > > I started this discussion again for serval main reasons: > > >> > > 1. We have implemented it and gained nearly 7x performance > > >> improvement in > > >> > > our prod environment > > >> > > 2. Many other companies made similar improvements based on their > > >> internal > > >> > > branch. > > >> > > 3. This topic has been discussed for a long time, but still > without > > >> any > > >> > > results. > > >> > > > > >> > > I hope we can push this important improvement in the community so > > that > > >> > all > > >> > > end-users can enjoy this significant improvement. > > >> > > > > >> > > I'd really appreciate you can join in and work with me to push > this > > >> > feature > > >> > > forward. > > >> > > > > >> > > Thanks very much. > > >> > > > > >> > > Ticket: HDFS-17366 < > > https://issues.apache.org/jira/browse/HDFS-17366> > > >> > > Design: NameNode Fine-grained locking based on directory tree > > >> > > < > > >> > > > > >> > > > >> > > > https://docs.google.com/document/d/1bVBQcI4jfzS0UrczB7UhsrQTXmrERGvBV-a9W3HCCjk/edit?usp=sharing > > >> > > > > > >> > > > > >> > > > >> > > > > > >
Re: Discussion about NameNode Fine-grained locking
Nice to see this feature brought up. We've implemented this feature internally and gained significant performance improvement. I'll be glad to work on some jiras if necessary. On Tue, Feb 20, 2024 at 4:41 PM ZanderXu wrote: > Thank you everyone for reviewing this ticket. > > I think if there are no problems with the goal and the overall solution, we > are ready to push this ticket forward and I will create some detailed > sub-tasks for this ticket. > > I will split this project into three milestones to make this project > cleaner for review and merge. > Milestone 1: Replacing the current global lock with two locks, global FS > lock and global BM lock. End-user can choose which locking mode to use > through configuration. > Milestone 2: Replacing the global FS write lock with directory tree-based > fine-grained lock. > Milestone 3: Replacing the global BM lock with directory tree-based > fine-grained lock. > > Each milestone can be merged into the trunk branch in time, which has > multiple benefits: > 1. Phased performance improvements can be quickly used by everyone > 2. All developers can better understand the implementation ideas of the > fine-grained locking mechanism as soon as possible > 3. Each milestone is developed based on the latest trunk branch to reduce > conflicts > > If you have any concerns, please feel free to discuss them together. > I hope you can join us to push this project forward together, thanks. > > > On Mon, 5 Feb 2024 at 11:33, haiyang hu wrote: > > > Thank you for raising the issue of this long-standing bottleneck, this > > will be a very important improvement! > > > > Hopefully can participate and push forward together. > > > > Best Regards~ > > > > Brahma Reddy Battula 于2024年2月3日周六 00:40写道： > > > >> Thanks for bringing this and considering all the history around this. > >> One of the outstanding bottleneck(global lock) from a long time. > >> > >> Hopefully we can push forward this time. > >> > >> > >> On Fri, Feb 2, 2024 at 12:23 PM Hui Fei wrote: > >> > >> > Thanks for driving this. It's very meaningful. The performance > >> improvement > >> > looks very good. > >> > > >> > Many users are facing the write performance issue. As far as I know, > >> some > >> > companies already implemented the similar idea on their internal > >> branches. > >> > But the internal branch is very different from the community one. So > >> it's > >> > very hard to be in sync with the community. If this improvement can be > >> > involved in the community, that would be great to both end-user and > the > >> > community. > >> > > >> > It is very worth doing. > >> > > >> > Zengqiang XU 于2024年2月2日周五 11:07写道： > >> > > >> > > Hi everyone > >> > > > >> > > I have started a discussion about NameNode Fine-grained Locking to > >> > improve > >> > > performance of write operations in NameNode. > >> > > > >> > > I started this discussion again for serval main reasons: > >> > > 1. We have implemented it and gained nearly 7x performance > >> improvement in > >> > > our prod environment > >> > > 2. Many other companies made similar improvements based on their > >> internal > >> > > branch. > >> > > 3. This topic has been discussed for a long time, but still without > >> any > >> > > results. > >> > > > >> > > I hope we can push this important improvement in the community so > that > >> > all > >> > > end-users can enjoy this significant improvement. > >> > > > >> > > I'd really appreciate you can join in and work with me to push this > >> > feature > >> > > forward. > >> > > > >> > > Thanks very much. > >> > > > >> > > Ticket: HDFS-17366 < > https://issues.apache.org/jira/browse/HDFS-17366> > >> > > Design: NameNode Fine-grained locking based on directory tree > >> > > < > >> > > > >> > > >> > https://docs.google.com/document/d/1bVBQcI4jfzS0UrczB7UhsrQTXmrERGvBV-a9W3HCCjk/edit?usp=sharing > >> > > > > >> > > > >> > > >> > > >
Re: Discussion about NameNode Fine-grained locking
Thank you everyone for reviewing this ticket. I think if there are no problems with the goal and the overall solution, we are ready to push this ticket forward and I will create some detailed sub-tasks for this ticket. I will split this project into three milestones to make this project cleaner for review and merge. Milestone 1: Replacing the current global lock with two locks, global FS lock and global BM lock. End-user can choose which locking mode to use through configuration. Milestone 2: Replacing the global FS write lock with directory tree-based fine-grained lock. Milestone 3: Replacing the global BM lock with directory tree-based fine-grained lock. Each milestone can be merged into the trunk branch in time, which has multiple benefits: 1. Phased performance improvements can be quickly used by everyone 2. All developers can better understand the implementation ideas of the fine-grained locking mechanism as soon as possible 3. Each milestone is developed based on the latest trunk branch to reduce conflicts If you have any concerns, please feel free to discuss them together. I hope you can join us to push this project forward together, thanks. On Mon, 5 Feb 2024 at 11:33, haiyang hu wrote: > Thank you for raising the issue of this long-standing bottleneck, this > will be a very important improvement! > > Hopefully can participate and push forward together. > > Best Regards~ > > Brahma Reddy Battula 于2024年2月3日周六 00:40写道： > >> Thanks for bringing this and considering all the history around this. >> One of the outstanding bottleneck(global lock) from a long time. >> >> Hopefully we can push forward this time. >> >> >> On Fri, Feb 2, 2024 at 12:23 PM Hui Fei wrote: >> >> > Thanks for driving this. It's very meaningful. The performance >> improvement >> > looks very good. >> > >> > Many users are facing the write performance issue. As far as I know, >> some >> > companies already implemented the similar idea on their internal >> branches. >> > But the internal branch is very different from the community one. So >> it's >> > very hard to be in sync with the community. If this improvement can be >> > involved in the community, that would be great to both end-user and the >> > community. >> > >> > It is very worth doing. >> > >> > Zengqiang XU 于2024年2月2日周五 11:07写道： >> > >> > > Hi everyone >> > > >> > > I have started a discussion about NameNode Fine-grained Locking to >> > improve >> > > performance of write operations in NameNode. >> > > >> > > I started this discussion again for serval main reasons: >> > > 1. We have implemented it and gained nearly 7x performance >> improvement in >> > > our prod environment >> > > 2. Many other companies made similar improvements based on their >> internal >> > > branch. >> > > 3. This topic has been discussed for a long time, but still without >> any >> > > results. >> > > >> > > I hope we can push this important improvement in the community so that >> > all >> > > end-users can enjoy this significant improvement. >> > > >> > > I'd really appreciate you can join in and work with me to push this >> > feature >> > > forward. >> > > >> > > Thanks very much. >> > > >> > > Ticket: HDFS-17366 <https://issues.apache.org/jira/browse/HDFS-17366> >> > > Design: NameNode Fine-grained locking based on directory tree >> > > < >> > > >> > >> https://docs.google.com/document/d/1bVBQcI4jfzS0UrczB7UhsrQTXmrERGvBV-a9W3HCCjk/edit?usp=sharing >> > > > >> > > >> > >> >
Re: Discussion about NameNode Fine-grained locking
Thank you for raising the issue of this long-standing bottleneck, this will be a very important improvement! Hopefully can participate and push forward together. Best Regards~ Brahma Reddy Battula 于2024年2月3日周六 00:40写道： > Thanks for bringing this and considering all the history around this. > One of the outstanding bottleneck(global lock) from a long time. > > Hopefully we can push forward this time. > > > On Fri, Feb 2, 2024 at 12:23 PM Hui Fei wrote: > > > Thanks for driving this. It's very meaningful. The performance > improvement > > looks very good. > > > > Many users are facing the write performance issue. As far as I know, some > > companies already implemented the similar idea on their internal > branches. > > But the internal branch is very different from the community one. So it's > > very hard to be in sync with the community. If this improvement can be > > involved in the community, that would be great to both end-user and the > > community. > > > > It is very worth doing. > > > > Zengqiang XU 于2024年2月2日周五 11:07写道： > > > > > Hi everyone > > > > > > I have started a discussion about NameNode Fine-grained Locking to > > improve > > > performance of write operations in NameNode. > > > > > > I started this discussion again for serval main reasons: > > > 1. We have implemented it and gained nearly 7x performance improvement > in > > > our prod environment > > > 2. Many other companies made similar improvements based on their > internal > > > branch. > > > 3. This topic has been discussed for a long time, but still without any > > > results. > > > > > > I hope we can push this important improvement in the community so that > > all > > > end-users can enjoy this significant improvement. > > > > > > I'd really appreciate you can join in and work with me to push this > > feature > > > forward. > > > > > > Thanks very much. > > > > > > Ticket: HDFS-17366 <https://issues.apache.org/jira/browse/HDFS-17366> > > > Design: NameNode Fine-grained locking based on directory tree > > > < > > > > > > https://docs.google.com/document/d/1bVBQcI4jfzS0UrczB7UhsrQTXmrERGvBV-a9W3HCCjk/edit?usp=sharing > > > > > > > > > >
Re: Discussion about NameNode Fine-grained locking
Thanks for bringing this and considering all the history around this. One of the outstanding bottleneck(global lock) from a long time. Hopefully we can push forward this time. On Fri, Feb 2, 2024 at 12:23 PM Hui Fei wrote: > Thanks for driving this. It's very meaningful. The performance improvement > looks very good. > > Many users are facing the write performance issue. As far as I know, some > companies already implemented the similar idea on their internal branches. > But the internal branch is very different from the community one. So it's > very hard to be in sync with the community. If this improvement can be > involved in the community, that would be great to both end-user and the > community. > > It is very worth doing. > > Zengqiang XU 于2024年2月2日周五 11:07写道： > > > Hi everyone > > > > I have started a discussion about NameNode Fine-grained Locking to > improve > > performance of write operations in NameNode. > > > > I started this discussion again for serval main reasons: > > 1. We have implemented it and gained nearly 7x performance improvement in > > our prod environment > > 2. Many other companies made similar improvements based on their internal > > branch. > > 3. This topic has been discussed for a long time, but still without any > > results. > > > > I hope we can push this important improvement in the community so that > all > > end-users can enjoy this significant improvement. > > > > I'd really appreciate you can join in and work with me to push this > feature > > forward. > > > > Thanks very much. > > > > Ticket: HDFS-17366 <https://issues.apache.org/jira/browse/HDFS-17366> > > Design: NameNode Fine-grained locking based on directory tree > > < > > > https://docs.google.com/document/d/1bVBQcI4jfzS0UrczB7UhsrQTXmrERGvBV-a9W3HCCjk/edit?usp=sharing > > > > > >
Re: Discussion about NameNode Fine-grained locking
Thanks for driving this. It's very meaningful. The performance improvement looks very good. Many users are facing the write performance issue. As far as I know, some companies already implemented the similar idea on their internal branches. But the internal branch is very different from the community one. So it's very hard to be in sync with the community. If this improvement can be involved in the community, that would be great to both end-user and the community. It is very worth doing. Zengqiang XU 于2024年2月2日周五 11:07写道： > Hi everyone > > I have started a discussion about NameNode Fine-grained Locking to improve > performance of write operations in NameNode. > > I started this discussion again for serval main reasons: > 1. We have implemented it and gained nearly 7x performance improvement in > our prod environment > 2. Many other companies made similar improvements based on their internal > branch. > 3. This topic has been discussed for a long time, but still without any > results. > > I hope we can push this important improvement in the community so that all > end-users can enjoy this significant improvement. > > I'd really appreciate you can join in and work with me to push this feature > forward. > > Thanks very much. > > Ticket: HDFS-17366 <https://issues.apache.org/jira/browse/HDFS-17366> > Design: NameNode Fine-grained locking based on directory tree > < > https://docs.google.com/document/d/1bVBQcI4jfzS0UrczB7UhsrQTXmrERGvBV-a9W3HCCjk/edit?usp=sharing > > >
Discussion about NameNode Fine-grained locking
Hi everyone I have started a discussion about NameNode Fine-grained Locking to improve performance of write operations in NameNode. I started this discussion again for serval main reasons: 1. We have implemented it and gained nearly 7x performance improvement in our prod environment 2. Many other companies made similar improvements based on their internal branch. 3. This topic has been discussed for a long time, but still without any results. I hope we can push this important improvement in the community so that all end-users can enjoy this significant improvement. I'd really appreciate you can join in and work with me to push this feature forward. Thanks very much. Ticket: HDFS-17366 <https://issues.apache.org/jira/browse/HDFS-17366> Design: NameNode Fine-grained locking based on directory tree <https://docs.google.com/document/d/1bVBQcI4jfzS0UrczB7UhsrQTXmrERGvBV-a9W3HCCjk/edit?usp=sharing>
