Hi all, I want to hear your thoughts of merging YARN resource profile branch into trunk in the next few weeks. The goal is to get it in for Hadoop 3.0 beta1.
*Regarding to testing:* We did extensive tests for the feature in the last several months. Comparing to latest trunk. - For SLS benchmark: We didn't see observable performance gap from simulated test based on 8K nodes SLS traces (1 PB memory). We got 3k+ containers allocated per second. - For microbenchmark: We use performance test cases added by YARN-6775, it shows around 5% performance regression comparing to trunk. *Regarding to API stability: * Most new added @Public APIs are @Unstable (We're going to convert some new added @Public/@Evolving to @Unstable in the cleanup JIRA as well), we want to get this included by beta1 so we get some feedbacks before declaring stable API. There're few pending cleanups under YARN-3926 umbrella JIRA. Besides these cleanups, this feature works from end-to-end, we will do another iteration of end-to-end tests after cleanup patches got committed. We would love to get your thoughts before opening a voting thread. Special thanks to a team of folks who worked hard and contributed towards this efforts including design discussion / patch / reviews, etc.: Varun Vasudev, Sunil Govind, Daniel Templeton, Vinod Vavilapalli, Yufei Gu, Karthik Kambatla, Jason Lowe, Arun Suresh. Thanks, Wangda Tan
