ic4y opened a new pull request, #6463: URL: https://github.com/apache/kyuubi/pull/6463
# :mag: Description ## Issue References 🔗 <!-- Append the issue number after #. If there is no issue for you to link create one or --> <!-- If there are no issues to link, please provide details here. --> This pull request fixes none ## Describe Your Solution 🔧 The engine startup concurrency limit is mainly used to restrict high load on the Kyuubi server. Therefore, we should immediately release the startupProcessSemaphore after the engine startup process exits (i.e., after the spark-submit process exits) as the load has already disappeared by then. Otherwise, we would have to wait until the program successfully starts after applying for resources on YARN to release the startupProcessSemaphore. This can block the submission of tasks from other queues when resources are insufficient in some YARN queues, greatly affecting the resource utilization of the Kyuubi server. -------------------------------------------------------------------------------------------------------------------------- 引擎启动并发限制主要是为了限制kyuubi服务器上的高负载。所以我们在引起启动成功进程退出后（spark-submit进程退出）负载已经消失，我们应该立即释放startupProcessSemaphore。否则还需要等程序在yarn上成功申请资源启动后才能释放startupProcessSemaphore，这在yarn某些队列资源不足的时候会阻塞其他队列的任务提交，极大的影响kyuubi server的资源利用率。 ## Types of changes :bookmark: <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: --> - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 I tested it on a cluster of 50 kyuubi Servers, and kyuubi server resource utilization increased by 70% #### Behavior Without This Pull Request :coffin: #### Behavior With This Pull Request :tada: #### Related Unit Tests --- # Checklist 📝 <!--- Go over all the following points, and put an `x` in all the boxes that apply. --> <!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
