[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha
[ https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13620462#comment-13620462 ] Siddharth Seth commented on HBASE-7904: --- bq. This issue started out as an incompability issue (see original Ted Yu comments at head of issue) If this was about MiniMRCluster being deprecated, the replacement is MiniMRClientClusterFactory which exists in branch-1 as well. Don't think this is an incompatible change. If using the new interface, MiniMRClientCluster.getConfig() can be used instead of MiniMRCluster.createJobConf(). bq. Yes, hbase tests can be run in parallel: e.g. TestTableMapReduce and TestHFileOutputFormat run at same time. Our surefire runs our test suite by firing off N JVMs each running a distinct, long-running test. Alright. So no parallel tests against a single cluster, but multiple clusters can run at the same time since these two test classes spin up their own clusters. MAPREDUCE-5083 becomes important to prevent the clusters clobbering each other. Will pull that into the 2.0.4-alpha branch. The summary posted over at YARN-449 (with these additions) should be enough to get the hbase mapreduce tests passing. I'm guessing the HBase specific changes will eventually make it in via separate jiras. Make mapreduce jobs pass based on 2.0.4-alpha - Key: HBASE-7904 URL: https://issues.apache.org/jira/browse/HBASE-7904 Project: HBase Issue Type: Task Components: mapreduce Reporter: Ted Yu Priority: Critical Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt 2.0.3-alpha has been released. We should upgrade the dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha
[ https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619159#comment-13619159 ] Siddharth Seth commented on HBASE-7904: --- Posted a summary at https://issues.apache.org/jira/browse/YARN-449?focusedCommentId=13619145page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13619145 bq. Did something change in the hadoop2 snapshot? A yarn patch got committed? Nothing that should've affected HBase (given the previous comment about clusters not running in parallel) bq. I suppose I could go look myself but I ask because supposedly there was an incompatibility issue. Do you happen to know what this incompatibility was (and against which version) ? bq. Regarding TestImportExport - I'm not sure why this test is using the HBaseTestingUtility differently than other tests. It was missing setting YARN conf in certain places. bq. We just want something that worked like minimr in hadoop1. We start the cluster do our tests, not in //, then shut down the cluster. I misunderstood you when you said // above. Yes we run our tests in //. We do not set up a minimr cluster once and run tests in // against it. I'm still not clear about this. (not so relevant now that the tests are passing, but would be nice to know). Yes we run our tests in // - this is a little confusing. As an example, can TestTableMapReduce run in parallel with TestHFileOutputFormat ?, or can multiple tests within TestTableMapReduce run in parallel, or something else. Make mapreduce jobs pass based on 2.0.4-alpha - Key: HBASE-7904 URL: https://issues.apache.org/jira/browse/HBASE-7904 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Priority: Critical Fix For: 0.95.0, 0.98.0 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt 2.0.3-alpha has been released. We should upgrade the dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7904) Upgrade hadoop 2.0 dependency to 2.0.4-alpha
[ https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13611080#comment-13611080 ] Siddharth Seth commented on HBASE-7904: --- [~saint@gmail.com] Using the config returned by MiniMRCluster is preferred to avoid similar problems in the future. Am guessing yarn.resourcemanager.address and yarn.resourcemanager.scheduler.address were set as a result of a similar change. YARN-470 along with a follow up to disable mem monitoring in the MiniCluster is for the OOM issue. MAPREDUCE-5083 is to allow parallel minimr tests to execute. Do HBase tests run in parallel? mapred.local.dir set in HBaseTestingUtility will not have an affect on a YARN cluster. Upgrade hadoop 2.0 dependency to 2.0.4-alpha Key: HBASE-7904 URL: https://issues.apache.org/jira/browse/HBASE-7904 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Priority: Critical Fix For: 0.95.0 Attachments: 7904.txt, 7904-v2-hadoop-2.0.txt, 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 7904-v5-hadoop-2.0.txt, 7904-v5.txt, hbase-7904-v3.txt 2.0.3-alpha has been released. We should upgrade the dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7904) Upgrade hadoop 2.0 dependency to 2.0.4-alpha
[ https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13611311#comment-13611311 ] Siddharth Seth commented on HBASE-7904: --- bq. Usually configs are out in xml files that we read into our context? The hbase Configuration adds the hbase-*.xml files to its base set. If tests need specific configs., we add them in xml and then make sure the Configuration reads them in before test runs. What can't we do that in this case rather than do this config. copy? To aceess the MiniMRCluster, certain client configs are required. This, along with a lot of defaults, are provided by the cluster after it is setup. Is it possible to use this, and then apply addiional configs on top. Something like - config from HDFS is the base (I see fs.default.name being set manually) - If MiniMR is required, set it up using the base HDFS config. - HBase configs, and test specific configs on top of this (without attempting to set MR/HDFS parameters) I'm sure HBase has similar client configs - which need to be loaded to access the MiniHBaseCluster. How are clients expected to use this ? bq. Why we even specifying addresses? Why are there not defaults that just work in say the standalone case so downstream projects don't even have to be concerned w/ setting this stuff? To allow parallel MiniClusters (different ports). bq. After reading reams of commentary by you pasting logs both here and over in yarn issues, I am still not clear what the problem is. It seems like my comment above on exit code 137 being code for OOME seems to be pertinent given cited YARN issue to disable mem checks (could we up the heap and have this stuff pass?) but I have no clue on how far we are from resolution or what action to take upstream. I think I now know what is wanted regards config reading over in YARN-449 seeing Siddharth Seth comments there but that issue seems to have run off into the weeds Figuring what is needed should not be this hard. 137 implies a kill -9 afaik. That's what the NodeManager/TT will eventually send out to containers. Note: This could be seen in case of successful MR tasks as well. Up the heap is not the main issue here. JVMs are not going OOM, they're being killed by YARN resource monitoring. After MR-5083 is committed (and snapshot published to mvn), can we trigger another run of the HBase tests - that should eliminate potential random failures caused by parallel tests. Upgrade hadoop 2.0 dependency to 2.0.4-alpha Key: HBASE-7904 URL: https://issues.apache.org/jira/browse/HBASE-7904 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Priority: Critical Fix For: 0.95.0 Attachments: 7904.txt, 7904-v2-hadoop-2.0.txt, 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 7904-v5-hadoop-2.0.txt, 7904-v5.txt, hbase-7904-v3.txt 2.0.3-alpha has been released. We should upgrade the dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7904) Upgrade hadoop 2.0 dependency to 2.0.4-alpha
[ https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13611396#comment-13611396 ] Siddharth Seth commented on HBASE-7904: --- bq. // clusters are default? Support the usecase where downstream projects run tests in parallel, and each test spins up a cluster (Assuming hbase fits into this). bq. I see by reading over in MR-5083, that it is a vmem limit. Can I up the vmem limit now w/o waiting on mr-5083 by setting some configs on minimr? (Assuming you mean MR-5094) yarn.nodemanager.vmem-pmem-ratio (default is 2.1f) Again, this is env specific. Afaik, the hbase build env does not see the OOM, not yet anyway. Upgrade hadoop 2.0 dependency to 2.0.4-alpha Key: HBASE-7904 URL: https://issues.apache.org/jira/browse/HBASE-7904 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Priority: Critical Fix For: 0.95.0 Attachments: 7904.txt, 7904-v2-hadoop-2.0.txt, 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 7904-v5-hadoop-2.0.txt, 7904-v5.txt, hbase-7904-v3.txt 2.0.3-alpha has been released. We should upgrade the dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
