Hi @zhangjian <1361320...@qq.com> , the dev branch HDFS-17531 is ready now. FYI.
Best Regards, - He Xiaoqiao On Tue, Jun 4, 2024 at 10:08 PM zhangjian <1361320...@qq.com> wrote: > Hi, Xiaoqiao He: > Can you help create a dev branch? I don't have the permission to create > it. > > Thank you very much. > - zhangjian > > > 2024年5月30日 11:57，Xiaoqiao He <hexiaoq...@apache.org> 写道： > > > > Great! It looks like there are no other nothing blockers. > > > > @zhangjian <1361320...@qq.com> If no other furthermore comments, we > should > > go to the next step: > > a. Create a dev branch for this proposal. > > b. Split this huge PR to some small JIRA and PRs. > > c. Involve some folks to review PR. > > > > Please ping here if you need any help. Thanks again. > > Good Luck! > > > > Best Regards, > > - He Xiaoqiao > > > > On Wed, May 29, 2024 at 11:46 AM Ayush Saxena <ayush...@gmail.com> > wrote: > > > >> Thanx for the details, sounds cool, good luck with the feature!!! > >> > >> -Ayush > >> > >>> On 29 May 2024, at 8:56 AM, zhangjian <1361320...@qq.com> wrote: > >>> > >>> Thank you for taking the time to review this proposal. > >>> Your opinion does point out the key issues to designing an asynchronous > >> router, but my proposal can address these issues: > >>> 1. My design does not affect the functionality of existing synchronous > >> routers in throwing stanby or retry exceptions and other aspects, the > async > >> router still inherits these implementations. > >>> 2. Currently, both asynchronous router and sync router support > >> backpressure on client requests when they exceed a certain limit ( > >>> asynchronous router : cannot obtain semaphores through the handler, > >>> sync router : block through handler synchronization, unable to obtain > >> available handler > >>> ) > >>> and return standby exception to allow the client to retry other routers > >> (RouterRpcFairnessPolicyController mechanism). > >>> > >>> Thank you again! > >>> zhangjian > >>> > >>>> 2024年5月29日 07:05，Ayush Saxena <ayush...@gmail.com> 写道： > >>>> Thanx folks, I had a very quick pass on the PDF and it looks good. > >>>> Maybe some doubts around the fact where it was mentioned that if a > >>>> Namenode returns a StandbyException or something on similar lines, the > >>>> Router will retry, I think we have some logic in RouterRpcClient > >>>> checking for such case, if it is StandByException it does try the > >>>> other Namenode, but for all other Retryable Exceptions, we return them > >>>> back to the client & let the client operate according to its Retry > >>>> Policy, I think we should preserve that behaviour, if the intentions > >>>> were to change it. > >>>> Regarding controlling the concurrency to prevent OOM at the router, > >>>> maybe we should consider rejecting the client requests beyond a > >>>> certain limit/backlog & return back a relevant Retriable Exception to > >>>> the client, so that it can retry on another Router rather than > >>>> overloading one Router when there are other available, most of the > >>>> deployments I believe would be running considerable number of Routers > >>>> Rest I scratched my head for possible scenario where things can go > >>>> south, but I think mostly the scenarios that came into my mind are > >>>> covered > >>>> Nothing blocker from my side, Good Luck!!! > >>>> -Ayush > >>>>> On Tue, 28 May 2024 at 21:52, Sangjin Lee <sj...@apache.org> wrote: > >>>>> Sounds good. Thanks for sharing your findings. > >>>>> On Sat, May 25, 2024 at 2:24 AM zhangjian <1361320...@qq.com> wrote: > >>>>>> Hello everyone, I conducted a performance comparison test between > >> sync and asynchronous router, and the test results showed that in > single ns > >> or multi ns scenarios, Asynchronous router in terms of throughput The > >> utilization of CPU and thread, as well as the average processing time of > >> client requests, are better than those of sync router, especially when > >> downstream ns have performance bottlenecks, The performance of the async > >> router is far greater than that of the sync router; And in terms of > >> isolation, Asynchronous router is also better than sync router. > >>>>>> Detailed testing PDF: > >> https://issues.apache.org/jira/browse/HDFS-17531 Comparison of Async > >> router & sync router performance.pdf > >>>>>> 2024年5月24日 14:13，Yuanbo Liu <liuyuanb...@gmail.com> 写道： > >>>>>> good job! > >>>>>> On Fri, May 24, 2024 at 1:57 AM zhangjian <1361320...@qq.com> > wrote: > >>>>>>> Hello everyone, currently, I have tested the performance of async > >> and sync router for a downstream ns: > >>>>>>> 1. The throughput, CPU, and thread performance of the async router > >> are better than those of the sync router, and its memory performance is > >> within an acceptable range compared to the synchronous router. > >>>>>>> 2. Asynchronous router can apply pressure downstream to better > >> utilize the performance of downstream ns, and can almost fill the call > >> queue of downstream ns. > >>>>>>> Due to the large size of the test result pdf, it cannot be sent via > >> email, > >>>>>>> please see: https://issues.apache.org/jira/browse/HDFS-17531 > >>>>>>>> 2024年5月23日 17:03，Xiaoqiao He <hexiaoq...@apache.org> 写道： > >>>>>>>> Great. Thanks for your addendum information. > >>>>>>>> cc @Ayush Saxena <ayush...@gmail.com> @inigo...@apache.org > >>>>>>>> <inigo...@apache.org> Any more feedback for this proposal? > >>>>>>>> IMO The feature of asynchronous router RPC is a helpful > >> improvement. For my > >>>>>>>> internal practice, it will improve the throughput of requests > >> forward > >>>>>>>> significantly > >>>>>>>> and is very valuable to push it forward. > >>>>>>>> Thanks again and good luck! > >>>>>>>> Best Regards, > >>>>>>>> - He Xiaoqiao > >>>>>>>> On Wed, May 22, 2024 at 9:59 AM zhangjian <1361320...@qq.com> > >> wrote: > >>>>>>>>> Hi, Sangjin Lee, thank you for your attention. I will use my free > >> time to > >>>>>>>>> do a performance comparison recently. > >>>>>>>>>> 2024年5月22日 03:42，Sangjin Lee <sj...@apache.org> 写道： > >>>>>>>>>> Thanks for the great proposal, Zhangjian. On point #3, I suspect > >> it > >>>>>>>>> should > >>>>>>>>>> be fairly straightforward to create a small isolated synthetic > >> test to > >>>>>>>>>> prove (or disprove) the benefits of this approach. By driving a > >>>>>>>>> controlled > >>>>>>>>>> amount of requests per second, you could see latency, memory, > >> CPU, etc. > >>>>>>>>>> Ideally, it should show meaningful improvements without much > >> degradation > >>>>>>>>> in > >>>>>>>>>> other metrics. Would you be able to spend some time doing that? > >>>>>>>>>> Thanks, > >>>>>>>>>> Sangjin > >>>>>>>>>> On Tue, May 21, 2024 at 5:13 AM zhangjian > >> <1361320...@qq.com.invalid> > >>>>>>>>> wrote: > >>>>>>>>>>> Hi, xiaoqiao he, thank you for your reply. > >>>>>>>>>>> 1.Currently, the server and client protocols within router can > be > >>>>>>>>>>> implemented by extends existing protocols and adding > asynchronous > >>>>>>>>>>> functionality, so it will not affect existing synchronization > >> protocols. > >>>>>>>>>>> RouterClientNamenodeProtocolServerSideTranslatorPB > >>>>>>>>>>> RouterClientProtocolTranslatorPB > >>>>>>>>>>> RouterGetUserMappingsProtocolServerSideTranslatorPB > >>>>>>>>>>> RouterGetUserMappingsProtocolTranslatorPB > >>>>>>>>>>> RouterNamenodeProtocolServerSideTranslatorPB > >>>>>>>>>>> RouterNamenodeProtocolTranslatorPB > >>>>>>>>>>> RouterRefreshUserMappingsProtocolServerSideTranslatorPB > >>>>>>>>>>> RouterRefreshUserMappingsProtocolTranslatorPB > >>>>>>>>>>> The following issues have implemented asynchronous callbacks > for > >>>>>>>>>>> Rpc.server, but I have not found any other modules to use > related > >>>>>>>>> functions > >>>>>>>>>>> Server HADOOP-11552 HADOOP-17046 > >>>>>>>>>>> In the implementation of asynchronous Rpc.client, this issue is > >> directly > >>>>>>>>>>> used > >>>>>>>>>>> Client HADOOP-13226 > >>>>>>>>>>> Therefore, I believe that asynchronous routers are safe for > >> modifying > >>>>>>>>> the > >>>>>>>>>>> RPC protocol, RPC server, and client > >>>>>>>>>>> 2. Forwarding requests to multiple downstream ns, the > >> synchronous router > >>>>>>>>>>> handler adds requests from multiple downstream ns to the thread > >> pool > >>>>>>>>>>> (RouterRpcClient.executorService), and then waits for responses > >> from all > >>>>>>>>>>> downstream ns before returning. Since threads in the thread > pool > >> also > >>>>>>>>>>> process rpc requests synchronously, similar to a handler, the > >> number of > >>>>>>>>>>> threads in the thread pool directly affects the performance of > >>>>>>>>>>> invoiceConcurrent, which in turn affects the performance of the > >> handler. > >>>>>>>>>>> In asynchronous router implementation, the handler calls > >>>>>>>>> invoiceConcurrent > >>>>>>>>>>> to simply convert a request into multiple requests and add them > >> to the > >>>>>>>>> asyn > >>>>>>>>>>> handler thread pool, which can then process the next request in > >> the call > >>>>>>>>>>> queue; When a connection thread of a downstream ns receives a > >> response, > >>>>>>>>> it > >>>>>>>>>>> will hand it over to the async response for processing. The > async > >>>>>>>>> response > >>>>>>>>>>> thread will determine whether it has received all responses > from > >> the > >>>>>>>>>>> downstream ns. If it does, it will continue to process the > >> response. > >>>>>>>>>>> Otherwise, the async response thread will process the next > >> response. The > >>>>>>>>>>> asynchronous router uses CompletableFuture.allOf() to implement > >>>>>>>>>>> asynchronous invoiceConcurrent, and the handler, async handler, > >> async > >>>>>>>>>>> response, and connection thread still does not need to wait > >>>>>>>>> synchronously. > >>>>>>>>>>> In addition, synchronous routers not only have drawbacks in > >> multi ns > >>>>>>>>>>> environments, but also in single downstream ns situations, it > is > >> often > >>>>>>>>>>> difficult to decide how many handlers to set for the router, > >> setting it > >>>>>>>>> too > >>>>>>>>>>> much will waste thread resources, and setting it too small will > >> not be > >>>>>>>>> able > >>>>>>>>>>> to give pressure to downstream ns; Asynchronous routers can > push > >>>>>>>>> requests > >>>>>>>>>>> to downstream ns without considering how to set handlers. > >> Asynchronous > >>>>>>>>>>> routers can also better connect to more downstream storage > >> services that > >>>>>>>>>>> support the HDFS protocol, with better scalability. > >>>>>>>>>>> 3.Since I have not yet deployed asynchronous routers to our own > >> cluster, > >>>>>>>>>>> there is no performance comparison. However, theoretically, I > >> believe > >>>>>>>>> that > >>>>>>>>>>> asynchronous routers will occupy more memory than synchronous > >> routers. > >>>>>>>>>>> However, I do not believe that it will occupy a lot, especially > >> since we > >>>>>>>>>>> can control the maximum number of requests entering the router, > >> as > >>>>>>>>>>> CompletableFuture is stable and widely used; In other aspects, > it > >>>>>>>>> should be > >>>>>>>>>>> far superior to synchronous routers, especially in downstream > >> scenarios > >>>>>>>>>>> with more ns.If anyone is interested, you can also help to > make a > >>>>>>>>>>> performance comparison > >>>>>>>>>>>> 2024年5月21日 11:39，Xiaoqiao He <hexiaoq...@apache.org> 写道： > >>>>>>>>>>>> Thanks for this great proposal! > >>>>>>>>>>>> Some questions after reviewing the design doc (sorry didn't > >> review PR > >>>>>>>>>>>> carefully which is too large.) > >>>>>>>>>>>> 1. This solution will involve RPC framework update, will it > >> affect > >>>>>>>>> other > >>>>>>>>>>>> modules and how to > >>>>>>>>>>>> keep other modules off these changes. > >>>>>>>>>>>> 2. Some RPC requests should be forward concurrently to all > >> downstream > >>>>>>>>> NS, > >>>>>>>>>>>> will it cover > >>>>>>>>>>>> this case in this solution. > >>>>>>>>>>>> 3. Considering there is one init-version implementation, did > you > >>>>>>>>> collect > >>>>>>>>>>>> some benchmark vs > >>>>>>>>>>>> the current synchronous model of DFSRouter? > >>>>>>>>>>>> Thanks again. > >>>>>>>>>>>> Best Regards, > >>>>>>>>>>>> - He Xiaoqiao > >>>>>>>>>>>> On Tue, May 21, 2024 at 11:21 AM zhangjian > >> <1361320...@qq.com.invalid> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>> Thank you for your positive attitude towards this feature. > You > >> can > >>>>>>>>> debug > >>>>>>>>>>>>> the UTs provided in PR to better understand the current > >> asynchronous > >>>>>>>>>>>>> calling function. > >>>>>>>>>>>>>> 2024年5月21日 02:04，Simbarashe Dzinamarira < > >> simbadz...@apache.org> 写道： > >>>>>>>>>>>>>> Excited to see this feature as well. I'll spend more time > >>>>>>>>> understanding > >>>>>>>>>>>>> the > >>>>>>>>>>>>>> proposal and implementation. > >>>>>>>>>>>>>> On Mon, May 20, 2024 at 7:55 AM zhangjian > >> <1361320...@qq.com.invalid > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> Hi, Yuanbo liu, thank you for your interest in this > >> feature, I > >>>>>>>>> think > >>>>>>>>>>>>> the > >>>>>>>>>>>>>>> difficulty of an asynchronous router is not only to > implement > >>>>>>>>>>>>> asynchronous > >>>>>>>>>>>>>>> functions, but also to consider the readability and > >> reusability of > >>>>>>>>> the > >>>>>>>>>>>>>>> code, so as to facilitate the development of the community. > >> I also > >>>>>>>>>>>>> planned > >>>>>>>>>>>>>>> to do the virtual thread you mentioned at the beginning, > >> virtual > >>>>>>>>>>> Threads > >>>>>>>>>>>>>>> can achieve asynchronousization elegantly at the code > level, > >> but the > >>>>>>>>>>>>>>> biggest problem is that it is not easy to upgrade the jdk > >> version, > >>>>>>>>> no > >>>>>>>>>>>>>>> matter in the community or in the actual production > >> environment. > >>>>>>>>>>>>> Therefore, > >>>>>>>>>>>>>>> I later used CompletableFuture, which is currently > supported > >> by jdk > >>>>>>>>> 8, > >>>>>>>>>>>>> to > >>>>>>>>>>>>>>> achieve asynchronousization. The router is stateless, and > >> the router > >>>>>>>>>>> rpc > >>>>>>>>>>>>>>> process is very clear. Therefore, even if CompletableFuture > >> itself > >>>>>>>>> is > >>>>>>>>>>>>> not > >>>>>>>>>>>>>>> as readable as the virtual thread, if we design it well, we > >> can make > >>>>>>>>>>> the > >>>>>>>>>>>>>>> asynchronous process look very clear. > >>>>>>>>>>>>>>>> 2024年5月20日 10:56，Yuanbo Liu <liuyuanb...@gmail.com> 写道： > >>>>>>>>>>>>>>>> Nice to see this feature brought up. I tried to implement > >> this > >>>>>>>>>>> feature > >>>>>>>>>>>>> in > >>>>>>>>>>>>>>>> our internal clusters, and know that it's a very > complicated > >>>>>>>>> feature, > >>>>>>>>>>>>> CC > >>>>>>>>>>>>>>>> hdfs-dev to bring more discussion. > >>>>>>>>>>>>>>>> By the way, I'm not sure whether virtual thread of higher > >> jdk will > >>>>>>>>>>> help > >>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>> this case. > >>>>>>>>>>>>>>>> On Mon, May 20, 2024 at 10:10 AM zhangjian > >>>>>>>>> <1361320...@qq.com.invalid > >>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>> Hello everyone, currently there are some shortcomings in > >> the RPC > >>>>>>>>> of > >>>>>>>>>>>>> HDFS > >>>>>>>>>>>>>>>>> router： > >>>>>>>>>>>>>>>>> Currently the router's handler thread is synchronized, > >> when the > >>>>>>>>>>>>>>> *handler* thread > >>>>>>>>>>>>>>>>> adds the call to connection.calls, it needs to wait until > >> the > >>>>>>>>>>>>>>> *connection* notifies > >>>>>>>>>>>>>>>>> the call to complete, and then Only after the response is > >> put into > >>>>>>>>>>> the > >>>>>>>>>>>>>>>>> response queue can a new call be obtained from the call > >> queue and > >>>>>>>>>>>>>>>>> processed. Therefore, the concurrency performance of the > >> router is > >>>>>>>>>>>>>>> limited > >>>>>>>>>>>>>>>>> by the number of handlers; a simple example is as > follows: > >> If the > >>>>>>>>>>>>>>> number of > >>>>>>>>>>>>>>>>> handlers is 1 and the maximum number of calls in the > >> connection > >>>>>>>>>>> thread > >>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>> 10, then even if the connection thread can send 10 > >> requests to the > >>>>>>>>>>>>>>>>> downstream ns, since the number of handlers is 1, the > >> router can > >>>>>>>>>>> only > >>>>>>>>>>>>>>>>> process one request after another. > >>>>>>>>>>>>>>>>> Since the performance of router rpc is mainly limited by > >> the > >>>>>>>>> number > >>>>>>>>>>> of > >>>>>>>>>>>>>>>>> handlers, the most effective way to improve rpc > performance > >>>>>>>>>>> currently > >>>>>>>>>>>>>>> is to > >>>>>>>>>>>>>>>>> increase the number of handlers. Letting the router > create > >> a large > >>>>>>>>>>>>>>> number > >>>>>>>>>>>>>>>>> of handler threads will also increase the number of > thread > >>>>>>>>> switches > >>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>> cannot maximize the use of machine performance. > >>>>>>>>>>>>>>>>> There are usually multiple ns downstream of the router. > If > >> the > >>>>>>>>>>> handler > >>>>>>>>>>>>>>>>> forwards the request to an ns with poor performance, it > >> will cause > >>>>>>>>>>> the > >>>>>>>>>>>>>>>>> handler to wait for a long time. Due to the reduction of > >> available > >>>>>>>>>>>>>>>>> handlers, the router's ability to handle ns requests with > >> normal > >>>>>>>>>>>>>>>>> performance will be reduced. From the perspective of the > >> client, > >>>>>>>>> the > >>>>>>>>>>>>>>>>> performance of the downstream ns of the router has > >> deteriorated at > >>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>> time. We often find that the call queue of the downstream > >> ns is > >>>>>>>>> not > >>>>>>>>>>>>>>> high, > >>>>>>>>>>>>>>>>> but the call queue of the router is very high. > >>>>>>>>>>>>>>>>> Therefore, although the main function of the router is to > >> federate > >>>>>>>>>>> and > >>>>>>>>>>>>>>>>> handle requests from multiple NSs, the current > synchronous > >> RPC > >>>>>>>>>>>>>>> performance > >>>>>>>>>>>>>>>>> cannot satisfy the scenario where there are many NSs > >> downstream of > >>>>>>>>>>> the > >>>>>>>>>>>>>>>>> router. Even if the concurrent performance of the router > >> can be > >>>>>>>>>>>>>>> improved by > >>>>>>>>>>>>>>>>> increasing the number of handlers, it is still relatively > >> slow. > >>>>>>>>> More > >>>>>>>>>>>>>>>>> threads will increase the CPU context switching time, and > >> in fact > >>>>>>>>>>> many > >>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>> the handler threads are in a blocked state, which is > >> undoubtedly a > >>>>>>>>>>>>>>> waste of > >>>>>>>>>>>>>>>>> thread resources. When a request enters the router, there > >> is no > >>>>>>>>>>>>>>> guarantee > >>>>>>>>>>>>>>>>> that there will be a running handler at this time. > >>>>>>>>>>>>>>>>> Therefore, I consider asynchronous router rpc. Please > view > >> the > >>>>>>>>>>> issues: > >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-17531 for > the > >>>>>>>>> complete > >>>>>>>>>>>>>>>>> solution. > >>>>>>>>>>>>>>>>> And you can also view this PR: > >>>>>>>>>>>>>>> https://github.com/apache/hadoop/pull/6838, > >>>>>>>>>>>>>>>>> which is just a demo, but it completes the core > >> asynchronous RPC > >>>>>>>>>>>>>>> function. > >>>>>>>>>>>>>>>>> If you think asynchronous routing is feasible, we can > >> consider > >>>>>>>>>>>>> splitting > >>>>>>>>>>>>>>>>> this PR for easy review in the future. > >>>>>>>>>>>>>>>>> The PDF is attached and can also be viewed through > issues. > >>>>>>>>>>>>>>>>> Welcome everyone to exchange and discuss! > >>>>>>>>> > >> --------------------------------------------------------------------- > >>>>>>>>>>>>>>> To unsubscribe, e-mail: > >> common-dev-unsubscr...@hadoop.apache.org > >>>>>>>>>>>>>>> For additional commands, e-mail: > >> common-dev-h...@hadoop.apache.org > >>>>>>>>>>>>> > >> --------------------------------------------------------------------- > >>>>>>>>>>>>> To unsubscribe, e-mail: > hdfs-dev-unsubscr...@hadoop.apache.org > >>>>>>>>>>>>> For additional commands, e-mail: > >> hdfs-dev-h...@hadoop.apache.org > >>>>>>>>>>> > >> --------------------------------------------------------------------- > >>>>>>>>>>> To unsubscribe, e-mail: > common-dev-unsubscr...@hadoop.apache.org > >>>>>>>>>>> For additional commands, e-mail: > >> common-dev-h...@hadoop.apache.org > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > >>>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org > >> > >> > > > >
