[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13624087#comment-13624087 ]
Devaraj Das commented on HBASE-8258: ------------------------------------ {quote} When it comes out we deal with it. The isolation of the particular relevant yarn configs demonstrate that the vmem stuff is red herring and unrelated as of hadoop 2.0.2. {quote} Are you saying that we fix the problems in HBase assuming hadoop-2.0.2-alpha as the underlying hadoop version? So far, the assumption I have been working with is that we wanted to get HBase working with 2.0.4-alpha (and this whole thread started with trying to make HBase work with hadoo-2.0.3-alpha, ref HBASE-7904). bq. copyConfigurationValues seems overly broad and doesn't expose the root cause. Hmm.. One can dump all the yarn configurations (yarn.* config keys) before and after the copyConfigurationValues call, and figure out what changed. That might be a worthwhile experiment. So let's say that we identify two or three configs. We should then change copyConfigurationValues to be more restrictive in copying? bq. You also skipped the last two questions – without getting to the root, this still doesn't explain why it works for other hbase-mr tests and not here, and how to make it so that any other mr tests doesn't have to have a special copyConfiguraitonValues equivalent call Sorry about missing these earlier... On the first one, as per the last hadoopQA run without the fixes with hadoop-2.0.4-SNAPSHOT (https://issues.apache.org/jira/browse/HBASE-8258?focusedCommentId=13621696&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13621696), almost all of the mapreduce tests failed. I haven't looked at all the tests that passed but one quick check on one of the passing test in the mapreduce package like TestTableSplit.java doesn't submit jobs, and maybe that's why they passed. On the other question, if the approach of calling copyConfigurationValues in the common part of the test code makes the tests work (as per the last 1 or 2 hadoopqa runs), I think we should make a note of this in some doc. We can get to this detail once we all agree on the proposed patch. bq. My fear is that there are fixes we need in upstream so lets figure them out sooner rather than later (figuring them out after 2.0.4-alpha means hbase 0.96 can't release till 2.0.5-alpha/beta whenever that is). Stack, are you suggesting that we should depend on 2.0.4-alpha sooner rather than later, maybe even check with 2.0.4-SNAPSHOT on a periodic basis until 2.0.4-alpha is released? > Make mapreduce tests pass on hadoop2 > ------------------------------------ > > Key: HBASE-8258 > URL: https://issues.apache.org/jira/browse/HBASE-8258 > Project: HBase > Issue Type: Bug > Components: mapreduce > Reporter: stack > Priority: Blocker > Fix For: 0.95.1 > > Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, > 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch > > > HBASE-7904 was a first attempt at making this work but it got lost in the > weeds. > This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o > breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira