----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/42340/#review114776 -----------------------------------------------------------
Ship it! Ship It! - Jaimin Jetly On Jan. 15, 2016, 9:36 p.m., Swapan Shridhar wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/42340/ > ----------------------------------------------------------- > > (Updated Jan. 15, 2016, 9:36 p.m.) > > > Review request for Ambari, Jaimin Jetly and Sumit Mohanty. > > > Bugs: AMBARI-14679 > https://issues.apache.org/jira/browse/AMBARI-14679 > > > Repository: ambari > > > Description > ------- > > When Spark is added later to a cluster with YARN deployed then yarn-site is > not modified. > > If Spark is added to an existing cluster with YARN using HDP 2.4, then > yarn-site:yarn.nodemanager.aux-services is not modified to include > spark_shuffle. > > Recommendation shown is : > > > Recommendations: > {'yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes': '', > 'yarn.nodemanager.container-executor.class': > 'org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor', > 'yarn.nodemanager.aux-services.spark_shuffle.class': > 'org.apache.spark.network.yarn.YarnShuffleService', > 'yarn.scheduler.minimum-allocation-vcores': '1', > 'yarn.scheduler.maximum-allocation-vcores': '1', > 'yarn.nodemanager.resource.memory-mb': '1024', > 'yarn.scheduler.minimum-allocation-mb': '341', > 'yarn.nodemanager.resource.cpu-vcores': '1', 'yarn.nodemanager.aux-services': > 'mapreduce_shuffle, 'yarn.scheduler.maximum-allocation-mb': '1024', > 'yarn.nodemanager.linux-container-executor.group': 'hadoop'} > > But with SPARK added, it should be : > > 'yarn.nodemanager.aux-services': 'mapreduce_shuffle,spark_shuffle' > > > Fix : Added the YARN property (yarn.nodemanager.aux-services, > yarn.nodemanager.aux-services.spark_shuffle.class) dependency on property > (spark.yarn.queue). > > > Diffs > ----- > > > ambari-server/src/main/resources/stacks/HDP/2.4/services/YARN/configuration/yarn-site.xml > PRE-CREATION > > Diff: https://reviews.apache.org/r/42340/diff/ > > > Testing > ------- > > - Tested the behavior. "spark_shuffle" gets shown in recommendations. > Screenshot Attached. > -> New cluster install > -> Adding SPARK service after cluster install. > - Build done. > > > File Attachments > ---------------- > > Screen Shot 2016-01-15 at 1.36.01 PM.png > > https://reviews.apache.org/media/uploaded/files/2016/01/15/0396ac33-6932-461a-abba-ab2138c854ec__Screen_Shot_2016-01-15_at_1.36.01_PM.png > > > Thanks, > > Swapan Shridhar > >
