[ https://issues.apache.org/jira/browse/YARN-702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670153#comment-13670153 ]
Siddharth Seth commented on YARN-702: ------------------------------------- bq. The issue in hbase is that we'd like to have one single conf that gets the new settings but the minicluster creates a new copy of the conf passed in instead of augmenting it. Thus when hbase MR jobs run, we need to copy out a whole bunch of settings back out. There's some discussion about this on HBASE-7904. https://issues.apache.org/jira/browse/HBASE-7904?focusedCommentId=13611311&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13611311 In the steps mentioned, is the same config object used ? Since MR is the last step, is it possible to replace this configuration with the conf returned by the MiniMRCluster (getConfiguration()). If the MiniMRCluster were to use the conf object passed in as a parameter, it would end up adding a bunch of keys to this config (Everything defined in yarn-default and yarn-site) - effectively the same as a getConfiguration on the cluster. I think it's useful for the cluster to run with a different config, and not share this with the multiple jobs that may be submitted to it. What may be possible is for the MiniMRCluster to copy these configs into the user specified config. That would be very similar to a blanket copy. As mentioned before, I think it'll be useful to expose a limited configuration set (RM address, isMiniCluster, etc) which can then be merged into client configs. Alternately, have the MiniCluster merge in this limited set. This could be a start towards separating cluster specific configuration and client specific configuration. > minicluster classpath construction requires user to set yarn.is.minicluster > in the job conf > ------------------------------------------------------------------------------------------- > > Key: YARN-702 > URL: https://issues.apache.org/jira/browse/YARN-702 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager > Affects Versions: 2.0.4-alpha > Reporter: Sandy Ryza > Assignee: Sandy Ryza > > YARN-129 improved classpath construction for miniclusters by, when > yarn.is.minicluster is set, adding the current JVM's classpath to the > ContainerLaunchContext for the MR AM and tasks. An issue with this is that > it requires the user to set yarn.is.minicluster on the mapreduce side in the > job conf, if they are not copying to RM conf into the jobconf. > I think it would be better to bypass the ContainerLaunchContext and instead > have the nodemanager check the property, and if it is true, do the classpath > additions there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira