As a cluster resource management platform, generic history service is necessary for job-tracing, troubleshooting, fault tolerance, and performance tuning. Like StarterLog in Condor, accounting logs in Torque, and so on. However, generic history service should avoid getting involved into too many details since some applications may have modules to trace their own histories.
+1 non-binding On Fri, Jan 17, 2014 at 2:53 PM, Zhijie Shen <zs...@hortonworks.com> wrote: > Hi folks, > > As previously discussed here (http://markmail.org/message/iscvp7cedrtvmd6p > ), > I would like to call a vote to merge the YARN-321 branch for Generic > Application History Server into trunk. > > *Scope of the changes* > > > The changes enable ResourceManager to record the historic information of > the application, the application attempt and the container in terms of > events via a history writer. In addition, the changes setup up an > application history server, which allows users to access the recorded > information via RPC interface, web UI and REST APIs. > > *Details of development* > > > - Development of the feature is tracked in the jira - > https://issues.apache.org/jira/browse/YARN-321. > - Development has been done in a separate branch - > https://svn.apache.org/repos/asf/hadoop/common/branches/YARN-321. > - The feature development involved about 35 subtasks. > - The up-to-date design is posted at - > https://issues.apache.org/jira/secure/attachment/12619638/Generic > Application > History - Design-20131219.pdf< > https://issues.apache.org/jira/secure/attachment/12619638/Generic%C2%A0Application%20History%20-%20Design-20131219.pdf > > > > - The uber merge patch Jira - > https://issues.apache.org/jira/browse/YARN-1587 > > *Testing* > > > A number of unit tests have been added as a part of the feature. In > addition, we’ve also done end-to-end functional tests, and performance > tests for HDFS-based history storage and history events processing. Last > but not least, we have updated branch YARN-321 against the latest trunk, > edited merge conflicts, fixed test failures caused by merge, and corrected > a bunch of bad source code issues. The uber merge patch that contains all > the diff between branch YARN-321 and trunk has been run through Jenkins. > > *Pending work* > > > - Make it work in secure mode > - Pending bug fixes > > We wish to merge the branch now instead of waiting for later. The main > reason for this is that as the branch grew in size, the cost of its > maintenance became huge. Once the feature is merged into trunk, we will > continue to work on pending work like security stuff, to test and fix any > bugs that may be found on the trunk, and to refactor the code about to > share some pieces in PRC and web interfaces. > > *Release status* > > > If the security stuff and the pending fixes arrive by the time everything > else planned for Release 2.4 is done, we can include it as well. This is > what we are striving for. Otherwise, we will call AHS not-feature-complete > and not stable. > > The bulk of the design and implementation was done by Mayank Bansal and me > with contributions from Devaraj K and Vinod Kumar Vavilapalli amongst > others. Also, thanks to Robert Joseph Evans and Sandy Ryza for providing > feedback on the design discussions. > > > > This vote runs for a week and closes on 1/24/2014 at 11:59 pm PT. > > Thanks, > Zhijie > > -- > Zhijie Shen > Hortonworks Inc. > http://hortonworks.com/ > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
