[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626290#comment-13626290 ] stack commented on HBASE-8258: -- [~ted_yu] Did you even look to see why the tests fail? What you have pasted, a list of test failures that seem unrelated, is close to zero help. [~jmhsieh] bq. I disagree with stack's comment... Fair enough. If can get this to work w/ current hadoop2, good. If fails against 2.0.4-SNAPSHOT, we should figure why but we would be in a better position if it first was working (and we knew why it had not been working previously). Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch, hbase-8258.simple.v2.patch, hbase-8258.simple.v3.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[
https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626635#comment-13626635
]
Jonathan Hsieh commented on HBASE-8258:
---
I only changed two lines that could modify all other tests
{code}
+// Tests were failing because this process used 6GB of virtual memory and
was getting killed.
+// we up the VM usable so that processes don't get killed.
+conf.setInt(yarn.nodemanager.resource.memory-mb, 8 * 1024);
+conf.setFloat(yarn.nodemanager.vmem-pmem-ratio, 8.0f);
+
{code}
doing the math this could mean allocating 64gb of vmem which may be excessive.
I removed the *.memory-mb line (I believe the default is 1.5gb -- maxing out
vmem at 12gb).
Make mapreduce tests pass on hadoop2
Key: HBASE-8258
URL: https://issues.apache.org/jira/browse/HBASE-8258
Project: HBase
Issue Type: Bug
Components: mapreduce
Reporter: stack
Priority: Blocker
Fix For: 0.95.1
Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt,
8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch,
hbase-8258.simple.v2.patch, hbase-8258.simple.v3.patch,
hbase-8258.simple.v4.patch
HBASE-7904 was a first attempt at making this work but it got lost in the
weeds.
This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o
breaking mapreduce on hadoop1)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626648#comment-13626648 ] Jonathan Hsieh commented on HBASE-8258: --- precommit breakage not due to this code. N fixed (and initially broke) the precommit builds last night with these commits: http://svn.apache.org/viewvc?view=revisionrevision=1465726 http://svn.apache.org/viewvc?view=revisionrevision=1465934 http://svn.apache.org/viewvc?view=revisionrevision=1465964 Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch, hbase-8258.simple.v2.patch, hbase-8258.simple.v3.patch, hbase-8258.simple.v4.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[
https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626672#comment-13626672
]
Hadoop QA commented on HBASE-8258:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12577796/hbase-8258.simple.v4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.security.access.TestAccessController
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5220//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5220//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5220//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5220//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5220//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5220//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5220//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5220//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5220//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5220//console
This message is automatically generated.
Make mapreduce tests pass on hadoop2
Key: HBASE-8258
URL: https://issues.apache.org/jira/browse/HBASE-8258
Project: HBase
Issue Type: Bug
Components: mapreduce
Reporter: stack
Priority: Blocker
Fix For: 0.95.1
Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt,
8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch,
hbase-8258.simple.v2.patch, hbase-8258.simple.v3.patch,
hbase-8258.simple.v4.patch
HBASE-7904 was a first attempt at making this work but it got lost in the
weeds.
This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o
breaking mapreduce on hadoop1)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626770#comment-13626770 ] Andrew Purtell commented on HBASE-8258: --- bq. doing the math this could mean allocating 64gb of vmem which may be excessive It is. This wouldn't work on EC2 jenkins as each instance has 15 GB total RAM and 0 GB swap. A max of 12 GB is still excessive IMHO. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch, hbase-8258.simple.v2.patch, hbase-8258.simple.v3.patch, hbase-8258.simple.v4.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626800#comment-13626800 ] Jonathan Hsieh commented on HBASE-8258: --- Without the memory-mb and vmem-pmem lines, yarn jobs were failing out with ~250 mb physical memory and 6.2gb vmem. I haven't been using the same memory constraints on my test runs, but since these are the only lines that could affect those tests. [~apurtell] do you have an alternate suggestion? (2x vmem seems reasonable) That amount of vmem implies that either our client uses way too many threads (allocating vmem for each threads' stack) or there is something funny in yarn. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch, hbase-8258.simple.v2.patch, hbase-8258.simple.v3.patch, hbase-8258.simple.v4.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626823#comment-13626823 ] Andrew Purtell commented on HBASE-8258: --- bq. do you have an alternate suggestion? (2x vmem seems reasonable) Not here, let's get the mr tests passing. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch, hbase-8258.simple.v2.patch, hbase-8258.simple.v3.patch, hbase-8258.simple.v4.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626840#comment-13626840 ] stack commented on HBASE-8258: -- Would it be of help breaking up some of the big MR unit tests? Or just moving them over to hbase-it (I don't want to paper-over a possible incompatibility or some mess in minimrcluster... ) Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch, hbase-8258.simple.v2.patch, hbase-8258.simple.v3.patch, hbase-8258.simple.v4.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626849#comment-13626849 ] Jonathan Hsieh commented on HBASE-8258: --- bq. Would it be of help breaking up some of the big MR unit tests? I don't think that will really matter. My builds are coming up clean or with failures on unrelated unit tests. I plan on committing this afternoon and will monitor the trunk on hadoop2 build. If the TestImportExports come back clean, I'll port to 0.95 and possibly 0.94. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch, hbase-8258.simple.v2.patch, hbase-8258.simple.v3.patch, hbase-8258.simple.v4.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626852#comment-13626852 ] Ted Yu commented on HBASE-8258: --- Did TestHFileOutputFormat pass based on your patch, Jon ? Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch, hbase-8258.simple.v2.patch, hbase-8258.simple.v3.patch, hbase-8258.simple.v4.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626896#comment-13626896 ] Jonathan Hsieh commented on HBASE-8258: --- Looks like my internal runs haven't been running the large tests, so the specific test and several of the others you pointed out were not run. I'm running them now. Note that from recent trunk-on-hadoop2 [1] runs, it looks like that the TestHFileOutputFormat test has been broken for a while. I'll move the HBASE-6330 in here as a sub-issue and commit the path to deal with TestImportExport this afternoon. I'll also follow up on the other broken tests. [1] https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/488/testReport/ Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch, hbase-8258.simple.v2.patch, hbase-8258.simple.v3.patch, hbase-8258.simple.v4.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626904#comment-13626904 ] Devaraj Das commented on HBASE-8258: Good to know your progress, Jon on hadoop-2.0.2. On the vmem stuff, can't you disable the checks via some yarn configuration? (I know that these checks have been disabled by default in the newer not-yet-released hadoop versions but I am not sure whether such config semantics already exist) Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch, hbase-8258.simple.v2.patch, hbase-8258.simple.v3.patch, hbase-8258.simple.v4.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626908#comment-13626908 ] Jonathan Hsieh commented on HBASE-8258: --- Deveraj, the addition of the disable checks is new hadoop-2.0.4 (YARN-470) i believe. Bumping the vmem number higher is a workaround that attempts to achieve the same result (avoiding the vmem checker from killing the job). Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch, hbase-8258.simple.v2.patch, hbase-8258.simple.v3.patch, hbase-8258.simple.v4.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626910#comment-13626910 ] Jonathan Hsieh commented on HBASE-8258: --- Seeing HBASE-6330 and its parent, HBASE-6891, I'm more inclined to close this as a dupe, commit he simple.v4 patch as HBASE-6330, and then use HBASE-6891 as the parent issue. Some of the issues over there are resolved as later and we can reopen those issues with proper details as they get worked. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch, hbase-8258.simple.v2.patch, hbase-8258.simple.v3.patch, hbase-8258.simple.v4.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626917#comment-13626917 ] Ted Yu commented on HBASE-8258: --- I think we should keep this JIRA open - it was opened for making mapreduce jobs pass based on 2.0.4-alpha. If we resolve this one, another (the 3rd) would be opened when 2.0.4-alpha comes out. bq. commit he simple.v4 patch as HBASE-6330 +1 Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch, hbase-8258.simple.v2.patch, hbase-8258.simple.v3.patch, hbase-8258.simple.v4.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626945#comment-13626945 ] Jonathan Hsieh commented on HBASE-8258: --- This issue's current title is 'Make mapreduce tests pass on hadoop2'. The overall effort should first go into properly fixing the unit tests against hadoop2 in our current build. From my runs I thought it was just one test but it looks like I missed a bunch. So this should be treated as an umbrella issue which makes it a dupe of HBASE-6891 (where efforts had started a while back). It doesn't make sense to have an issue open for something that cannot be worked on. Since 2.0.4 isn't released whatever fixes we'd commit along this line would have the same problem as building on SNAPSHOTs -- it may break because of changes between the hbase commit and when 2.0.4 is released. Once hadoop 2.0.4 is released, we can test against it then and if there are new issues with it, open jiras and fix it then. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch, hbase-8258.simple.v2.patch, hbase-8258.simple.v3.patch, hbase-8258.simple.v4.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13627220#comment-13627220 ] Ted Yu commented on HBASE-8258: --- Your action is quick, Jon :-) Secure bulk load would not work, based on current hdfs branch 2.0 When 2.0.4-alpha gets released, I plan to log a JIRA to fix that. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Assignee: Jonathan Hsieh Priority: Blocker Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch, hbase-8258.simple.v2.patch, hbase-8258.simple.v3.patch, hbase-8258.simple.v4.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13627250#comment-13627250 ] Jonathan Hsieh commented on HBASE-8258: --- bq. Secure bulk load would not work, based on current hdfs branch 2.0 bq. When 2.0.4-alpha gets released, I plan to log a JIRA to fix that. I think you mean that the current trunk on the hdfs's repo's branch-2 will fail and that after 2.0.4 comes out you'll file a jira to fix it. Sounds good to me! Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Assignee: Jonathan Hsieh Priority: Blocker Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch, hbase-8258.simple.v2.patch, hbase-8258.simple.v3.patch, hbase-8258.simple.v4.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13627253#comment-13627253 ] Ted Yu commented on HBASE-8258: --- That was what I meant. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Sub-task Components: mapreduce Reporter: stack Assignee: Jonathan Hsieh Priority: Blocker Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch, hbase-8258.simple.v2.patch, hbase-8258.simple.v3.patch, hbase-8258.simple.v4.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[
https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13625519#comment-13625519
]
Jonathan Hsieh commented on HBASE-8258:
---
As a follow on to [~apurtell], you can always pick the profile and then add a
-Dhadoop-two.version=version with your favorite hadoop2 version -- e.g.:
{{mvn dependency:list -Dhadoop.profile=2.0 -Dhadoop-two.version=2.0.3-alpha}}
Make mapreduce tests pass on hadoop2
Key: HBASE-8258
URL: https://issues.apache.org/jira/browse/HBASE-8258
Project: HBase
Issue Type: Bug
Components: mapreduce
Reporter: stack
Priority: Blocker
Fix For: 0.95.1
Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt,
8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch
HBASE-7904 was a first attempt at making this work but it got lost in the
weeds.
This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o
breaking mapreduce on hadoop1)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13625521#comment-13625521 ] Jonathan Hsieh commented on HBASE-8258: --- Looks like the failures in the hadoop1 verson is the theory from a previous part of the conversation -- writing to different hdfs's from HBASE-6330 [1]. [1] https://issues.apache.org/jira/browse/HBASE-6330?focusedCommentId=13420691page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13420691 Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13625660#comment-13625660 ] Jonathan Hsieh commented on HBASE-8258: --- Fully qualifying the export path to hdfs makes the job work in hadoop1. In hadoop 1: export was exporting to a random dir in hdfs, while import was expecting data from the hadoop file system. * export to: hdfs://localhost:46368/user/jon/test-data/72b45ff2-9203-4791-aa28-2b95e44f5d95/hadoop/mapred-working-dir/outputdir/part-m-0 h * import from: hdfs://localhost:46368/user/jon/outputdir/ In hadoop 2: export was exporting to the hdfs while import was expecting data form hdfs. * export to: hdfs://localhost:60549/user/jon/outputdir/part-m-0 * import from: hdfs://localhost:60549/user/jon/outputdir/ So at least for the export tests, I'm going to use fully qualified hdfs paths for the export output Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626096#comment-13626096 ] Jonathan Hsieh commented on HBASE-8258: --- Full suite passes for me with patch applied against hadoop2 and hadoop1. Will commit tomorrow afternoon unless I hear otherwise. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch, hbase-8258.simple.v2.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626111#comment-13626111 ] Ted Yu commented on HBASE-8258: --- From https://builds.apache.org/job/PreCommit-HBASE-Build/5193/console : /home/jenkins/tools/maven/latest/bin/mvn clean test -DskipTests -DHBasePatchProcess /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkJavacWarnings.txt 21 Trunk compilation is broken? Once the patch is checked in, do we keep this JIRA open for the upcoming 2.0.4-alpha or open new JIRA ? Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch, hbase-8258.simple.v2.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626120#comment-13626120 ] Jonathan Hsieh commented on HBASE-8258: --- [~te...@apache.org] we close this and then when 2.0.4 comes out we file a new jira to deal with the problems it introduces there. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch, hbase-8258.simple.v2.patch, hbase-8258.simple.v3.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626122#comment-13626122 ] Jonathan Hsieh commented on HBASE-8258: --- I'm suprised that our script doesn't report if a patch files to apply. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch, hbase-8258.simple.v2.patch, hbase-8258.simple.v3.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[
https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626211#comment-13626211
]
Ted Yu commented on HBASE-8258:
---
I got the following test failures running on top of 2.0.2-alpha:
{code}
Running org.apache.hadoop.hbase.regionserver.TestRSKilledWhenMasterInitializing
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 128.739 sec
FAILURE!
--
Running org.apache.hadoop.hbase.regionserver.TestJoinedScanners
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 238.319 sec
FAILURE!
--
Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Tests run: 10, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 273.141 sec
FAILURE!
--
Running org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel
Tests run: 4, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 526.378 sec
FAILURE!
Running org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential
Tests run: 4, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 534.981 sec
FAILURE!
--
Running org.apache.hadoop.hbase.TestDrainingServer
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 72.381 sec
FAILURE!
--
Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting
Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 345.015 sec
FAILURE!
Running org.apache.hadoop.hbase.master.TestTableLockManager
Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 655.895 sec
FAILURE!
{code}
TestHFileOutputFormat is in the scope of this JIRA.
Make mapreduce tests pass on hadoop2
Key: HBASE-8258
URL: https://issues.apache.org/jira/browse/HBASE-8258
Project: HBase
Issue Type: Bug
Components: mapreduce
Reporter: stack
Priority: Blocker
Fix For: 0.95.1
Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt,
8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch,
hbase-8258.simple.v2.patch, hbase-8258.simple.v3.patch
HBASE-7904 was a first attempt at making this work but it got lost in the
weeds.
This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o
breaking mapreduce on hadoop1)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[
https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626214#comment-13626214
]
Ted Yu commented on HBASE-8258:
---
After test suite completed, I saw two hanging JVMs:
{code}
pool-1-thread-1 prio=10 tid=0x7f2d006b2000 nid=0x3a2c waiting on
condition [0x7f2cb1c92000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0xedb7c3d8 (a
java.util.concurrent.FutureTask$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.bulkLoadPhase(LoadIncrementalHFiles.java:324)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:264)
at
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles.runTest(TestLoadIncrementalHFiles.java:158)
at
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles.testSimpleLoad(TestLoadIncrementalHFiles.java:86)
{code}
{code}
pool-1-thread-1 prio=10 tid=0x7f15a069c000 nid=0x5584 waiting on
condition [0x7f15662e6000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0xdaf6f150 (a
java.util.concurrent.FutureTask$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplitPhase(LoadIncrementalHFiles.java:374)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:261)
at
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery.populateTable(TestLoadIncrementalHFilesSplitRecovery.java:148)
at
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery.testSplitWhileBulkLoadPhase(TestLoadIncrementalHFilesSplitRecovery.java:298)
{code}
Make mapreduce tests pass on hadoop2
Key: HBASE-8258
URL: https://issues.apache.org/jira/browse/HBASE-8258
Project: HBase
Issue Type: Bug
Components: mapreduce
Reporter: stack
Priority: Blocker
Fix For: 0.95.1
Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt,
8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch,
hbase-8258.simple.v2.patch, hbase-8258.simple.v3.patch
HBASE-7904 was a first attempt at making this work but it got lost in the
weeds.
This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o
breaking mapreduce on hadoop1)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13623662#comment-13623662 ] Devaraj Das commented on HBASE-8258: I think we have to move to the next release of hadoop (2.0.4+). That fixes some issues needed by HBase tests to work. On the usage of _merge_ versus _copyConfiguration_, I think we should use copyConfiguration and if not at least identify what configs we really need to copy from the possibly updated configuration. If I am not mistaken, I think that is the main point of confusion in this jira. Right, [~jmhsieh]? [~yuzhih...@gmail.com], do you mind trying the usage of copyConfiguration instead of merge? Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[
https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13623732#comment-13623732
]
Hadoop QA commented on HBASE-8258:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12577217/8258-v2-hadoop-2.0.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
42 warning messages.
{color:red}-1 javac{color}. The applied patch generated 16 javac compiler
warnings (more than the trunk's current 12 warnings).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.security.token.TestTokenAuthentication
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles.testSimpleLoad(TestLoadIncrementalHFiles.java:86)
at
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery.testSplitWhileBulkLoadPhase(TestLoadIncrementalHFilesSplitRecovery.java:298)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5152//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5152//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5152//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5152//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5152//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5152//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5152//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5152//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5152//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5152//console
This message is automatically generated.
Make mapreduce tests pass on hadoop2
Key: HBASE-8258
URL: https://issues.apache.org/jira/browse/HBASE-8258
Project: HBase
Issue Type: Bug
Components: mapreduce
Reporter: stack
Priority: Blocker
Fix For: 0.95.1
Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt,
8258-v2-hadoop-2.0.txt
HBASE-7904 was a first attempt at making this work but it got lost in the
weeds.
This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o
breaking mapreduce on hadoop1)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[
https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13623739#comment-13623739
]
Jonathan Hsieh commented on HBASE-8258:
---
bq. As I said above, 0.95 needs a stable hadoop 2 release (2.0.4-alpha).
2.0.4 is not released yet http://hadoop.apache.org/releases.html. When it is,
we'll deal with it.
bq. SNAPSHOT artifacts would allow us to find the problem much sooner.
-1 to building hbase on SNAPSHOT default.
bq. One such config param is YarnConfiguration.IS_MINI_YARN_CLUSTER which must
be set true in order for MiniMRCluster to work.
From my experiments I'm not sure if that is relevant to the current problems.
I've narrowed it down to only needing to set these two yarn configs (because
something overrides their defaults). All the rest of the yarn config values
are not needed.
{code}
yarn.resourcemanager.scheduler.address : localhost:49649
yarn.resourcemanager.address : localhost:48993
{code}
yarn.rm.schedule.addrss is default port 8030 and yarn.resourcemanager.address
is default port 8032. The need for the different value here explains why we
get the ConnectException from the mr client trying to talk to 0.0.0.0:8032.
The follow-on questions this leads me to are:
* Why is do we need to override these in our minimrcluster tests?
* Why isn't this need documented and where should it be?
* Why isn't this a problem on other HBase-MR jobs?
* What can we do to properly propagate this so that future MR tests won't have
this problem?
I feel that these need to be answered and hopefully addressed before we commit
a fix.
bq. Can you explain folding a little bit ?
I'm using folding and merging as synonyms -- what the copyConfigurationValues
does.
Make mapreduce tests pass on hadoop2
Key: HBASE-8258
URL: https://issues.apache.org/jira/browse/HBASE-8258
Project: HBase
Issue Type: Bug
Components: mapreduce
Reporter: stack
Priority: Blocker
Fix For: 0.95.1
Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt,
8258-v2-hadoop-2.0.txt
HBASE-7904 was a first attempt at making this work but it got lost in the
weeds.
This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o
breaking mapreduce on hadoop1)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13623907#comment-13623907 ] Jonathan Hsieh commented on HBASE-8258: --- bq. 2.0.4 has some fixes that impact HBase tests (MAPREDUCE-5083, YARN-470 at the least). I think we should wait for the release to happen before we do anything more on this issue. When it comes out we deal with it. The isolation of the particular relevant yarn configs demonstrate that the vmem stuff is red herring and unrelated as of hadoop 2.0.2. bq. This (the need to copy configuration values) was discovered and introduced way back in HBASE-6601. Ted's patch that had the reference to merge made it so that all MR tests could get all the (updated) configuration key/vals. In reality, copyConfigurationValues alone is sufficient. Ted's current patch has it that way (all MR tests get the updated configs via copyConfigurationValues). I think the javadoc comment on copyConfigurationValues is already a nice document. Maybe, we could enhance that a bit.. copyConfigurationValues seems overly broad and doesn't expose the root cause. You also skipped the last two questions -- without getting to the root, this still doesn't explain why it works for other hbase-mr tests and not here, and how to make it so that any other mr tests doesn't have to have a special copyConfiguraitonValues equivalent call. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13623937#comment-13623937 ] stack commented on HBASE-8258: -- bq. I think we should wait for the release to happen before we do anything more on this issue. My fear is that there are fixes we need in upstream so lets figure them out sooner rather than later (figuring them out after 2.0.4-alpha means hbase 0.96 can't release till 2.0.5-alpha/beta whenever that is). On copyConfigurationValues, we could do that but it blankets over questions Jon is asking of why the breakage and only in one of our MR tests. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13624073#comment-13624073 ] Jonathan Hsieh commented on HBASE-8258: --- bq. Why is do we need to override these in our minimrcluster tests? bq. Why isn't this a problem on other HBase-MR jobs? The other MR tests use the HBaseTestingUtility's getConfiguration conf. When the mini-mr cluster is started it adds values to it. The current TestImportExport gets the values from HBaseTestingUtility.getMiniCluster.getConf which does not have yarn/mr values added. That's why we had to the extra copy junk. If we use the pattern used in the other MR jobs, we don't have this problem. There is one catch -- we need to make a copy of the conf for the job conf by using new Configuration(htu.getConfiguration). Without that I was having data from previous exports from one test interfere with subsequent runs in suite. bq. Why isn't this need documented and where should it be? bq. What can we do to properly propagate this so that future MR tests won't have this problem? I will improve javadoc in HBaseTestingUtility to explain the proper way of getting configurations for hbase unit test mr jobs. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[
https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13624084#comment-13624084
]
Nick Dimiduk commented on HBASE-8258:
-
bq. My fear is that there are fixes we need in upstream so lets figure them out
sooner rather than later
Good point.
bq. copyConfigurationValues seems overly broad and doesn't expose the root
cause.
Cluster management in this test is pretty difficult to follow. I think the
underlying cause is that the job configuration is built against the wrong
cluster config. Let's see if I understand this correctly.
Back in {{beforeClass}} we have:
{code}
@BeforeClass
public static void beforeClass() throws Exception {
cluster = UTIL.startMiniCluster();
UTIL.startMiniMapReduceCluster();
}
{code}
I guess these are two different clusters, one running HBase, managed via
{{cluster}}, and the other running MR, managed by {{UTIL}}.
{{testSimpleCase}} starts by populating a table on {{cluster}} (HBase cluster),
it then creates the Export job using the configuration of {{cluster}} (HBase
cluster), even though that's the cluster lacking MR. Thus, it's necessary to
copy the MR details from {{UTIL}} (MR cluster) into the Export job's config.
Since the test is, at it's essence, running a MR job, wouldn't the test be made
more stable (and realistic to user jobs) if instead the HBase connection
details were copied into a MR job config build from {{UTIL}} (MR cluster)?
Further, that copy can be done in one place, way back in {{beforeClass}}. That
way, the code repeated for each test is limited.
{code}
// in beforeClass:
this.baseConf = new Configuration(UTIL.getConfiguration());
... // copy hbase configs into baseConf
// in each test
GenericOptionsParser opts = new GenericOptionsParser(new
Configuration(baseConf), args);
Configuration exportConf = opts.getConfiguration();
args = opts.getRemainingArgs();
{code}
This can be further simplified if the jobs under test implement the {{Tool}}
interface (like I did for ImportTsv in HBASE-8011) and deprecating the
{{createSubmittableJob}} nonsense. Then it becomes something closer to:
{code}
ToolRunner.run(new Configuration(baseConf), new Export(), args);
{code}
Make mapreduce tests pass on hadoop2
Key: HBASE-8258
URL: https://issues.apache.org/jira/browse/HBASE-8258
Project: HBase
Issue Type: Bug
Components: mapreduce
Reporter: stack
Priority: Blocker
Fix For: 0.95.1
Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt,
8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch
HBASE-7904 was a first attempt at making this work but it got lost in the
weeds.
This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o
breaking mapreduce on hadoop1)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[
https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13624087#comment-13624087
]
Devaraj Das commented on HBASE-8258:
{quote}
When it comes out we deal with it.
The isolation of the particular relevant yarn configs demonstrate that the vmem
stuff is red herring and unrelated as of hadoop 2.0.2.
{quote}
Are you saying that we fix the problems in HBase assuming hadoop-2.0.2-alpha as
the underlying hadoop version? So far, the assumption I have been working with
is that we wanted to get HBase working with 2.0.4-alpha (and this whole thread
started with trying to make HBase work with hadoo-2.0.3-alpha, ref HBASE-7904).
bq. copyConfigurationValues seems overly broad and doesn't expose the root
cause.
Hmm.. One can dump all the yarn configurations (yarn.* config keys) before and
after the copyConfigurationValues call, and figure out what changed. That might
be a worthwhile experiment. So let's say that we identify two or three configs.
We should then change copyConfigurationValues to be more restrictive in copying?
bq. You also skipped the last two questions – without getting to the root, this
still doesn't explain why it works for other hbase-mr tests and not here, and
how to make it so that any other mr tests doesn't have to have a special
copyConfiguraitonValues equivalent call
Sorry about missing these earlier...
On the first one, as per the last hadoopQA run without the fixes with
hadoop-2.0.4-SNAPSHOT
(https://issues.apache.org/jira/browse/HBASE-8258?focusedCommentId=13621696page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13621696),
almost all of the mapreduce tests failed. I haven't looked at all the tests
that passed but one quick check on one of the passing test in the mapreduce
package like TestTableSplit.java doesn't submit jobs, and maybe that's why they
passed.
On the other question, if the approach of calling copyConfigurationValues in
the common part of the test code makes the tests work (as per the last 1 or 2
hadoopqa runs), I think we should make a note of this in some doc. We can get
to this detail once we all agree on the proposed patch.
bq. My fear is that there are fixes we need in upstream so lets figure them out
sooner rather than later (figuring them out after 2.0.4-alpha means hbase 0.96
can't release till 2.0.5-alpha/beta whenever that is).
Stack, are you suggesting that we should depend on 2.0.4-alpha sooner rather
than later, maybe even check with 2.0.4-SNAPSHOT on a periodic basis until
2.0.4-alpha is released?
Make mapreduce tests pass on hadoop2
Key: HBASE-8258
URL: https://issues.apache.org/jira/browse/HBASE-8258
Project: HBase
Issue Type: Bug
Components: mapreduce
Reporter: stack
Priority: Blocker
Fix For: 0.95.1
Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt,
8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch
HBASE-7904 was a first attempt at making this work but it got lost in the
weeds.
This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o
breaking mapreduce on hadoop1)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13624088#comment-13624088 ] Jonathan Hsieh commented on HBASE-8258: --- [~ndimiduk] That is very close the explanation that I have. The minimrcluster for yarn adds values to the configuration so if we refer to the same one and copy it later, we get the updated yarn settings. This is what all the other MR tests do. I need to look into why we need to do the copy of the config to prevent interference (likely some if the value is set don't update it logic). I agree that using the tool interface makes sense as well -- feels like a nice follow up issue. I've been running with the version I posted on trunk against hadoop 2.0 -- will report on that and against hadoop 1.0 when results return. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[
https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13624103#comment-13624103
]
Hadoop QA commented on HBASE-8258:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12577283/8258-v4-hadoop-2.0.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
42 warning messages.
{color:red}-1 javac{color}. The applied patch generated 16 javac compiler
warnings (more than the trunk's current 12 warnings).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestZooKeeper
org.apache.hadoop.hbase.security.token.TestTokenAuthentication
org.apache.hadoop.hbase.security.access.TestAccessController
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles.testSimpleLoad(TestLoadIncrementalHFiles.java:86)
at
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery.testSplitWhileBulkLoadPhase(TestLoadIncrementalHFilesSplitRecovery.java:299)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5155//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5155//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5155//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5155//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5155//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5155//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5155//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5155//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5155//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5155//console
This message is automatically generated.
Make mapreduce tests pass on hadoop2
Key: HBASE-8258
URL: https://issues.apache.org/jira/browse/HBASE-8258
Project: HBase
Issue Type: Bug
Components: mapreduce
Reporter: stack
Priority: Blocker
Fix For: 0.95.1
Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt,
8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch
HBASE-7904 was a first attempt at making this work but it got lost in the
weeds.
This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o
breaking mapreduce on hadoop1)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13624104#comment-13624104 ] stack commented on HBASE-8258: -- [~devaraj] bq. Stack, are you suggesting that we should depend on 2.0.4-alpha sooner rather than later, maybe even check with 2.0.4-SNAPSHOT on a periodic basis until 2.0.4-alpha is released? I suggest sooner so if any issues in hdfs/mr/yarn, we can help out our parent project by surfacing them quicker (For example, I don't mind shipping a 0.95.x Development Series release against a hadoop SNAPSHOT; I don't see it as a problem given the quick decay half-life of 0.95.x release). Thanks Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13624105#comment-13624105 ] Jonathan Hsieh commented on HBASE-8258: --- bq. Are you saying that we fix the problems in HBase assuming hadoop-2.0.2-alpha as the underlying hadoop version? Yes. Its been what this build has been broken against for a long time so let's get that fixed first. Haven't tried 2.0.3 yet, but after this gets fixed we can go there. *One problem at a time and ideally per jira.* bq. So let's say that we identify two or three configs. We should then change copyConfigurationValues to be more restrictive in copying? Experiment was already done, see [1] -- yarn.resourcemanager.scheduler.address, yarn.resourcemanager.address were the main ones. That lead to the follow up questions, which lead me to see that the whole copying stuff is moot. bq. On the first one, as per the last hadoopQA run without the fixes with hadoop-2.0.4-SNAPSHOT (https://issues.apache.org/jira/browse/HBASE-8258?focusedCommentId=13621696page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13621696), almost all of the mapreduce tests failed. I'm going to ignore tests run against a hadoop SNAPSHOT. bq. Stack, are you suggesting that we should depend on 2.0.4-alpha sooner rather than later, maybe even check with 2.0.4-SNAPSHOT on a periodic basis until 2.0.4-alpha is released? I think he means that not fixing this until 2.0.4 comes out is not a good idea, but I'll let him answer. [1] https://issues.apache.org/jira/browse/HBASE-8258?focusedCommentId=13623739page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13623739 Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[
https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13624163#comment-13624163
]
Hadoop QA commented on HBASE-8258:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12577297/hbase-8258-simple.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestImportExport
org.apache.hadoop.hbase.master.TestTableLockManager
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5157//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5157//console
This message is automatically generated.
Make mapreduce tests pass on hadoop2
Key: HBASE-8258
URL: https://issues.apache.org/jira/browse/HBASE-8258
Project: HBase
Issue Type: Bug
Components: mapreduce
Reporter: stack
Priority: Blocker
Fix For: 0.95.1
Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt,
8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch
HBASE-7904 was a first attempt at making this work but it got lost in the
weeds.
This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o
breaking mapreduce on hadoop1)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[
https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13624173#comment-13624173
]
Jonathan Hsieh commented on HBASE-8258:
---
Hm.. On my second try, it looks like hadoop1 has got some failures, but the
hadoop2 build is good.
On my first run against hadoop2 I had problems with vmem overuse:
{code}
2013-04-05 14:57:45,983 WARN [AsyncDispatcher event handler]
resourcemanager.RMAuditLogger(255): USER=jenkins OPERATION=Application
Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App
failed with state: FAILED PERMISSIONS=Application
application_1365199040071_0001 failed 1 times due to AM Container for
appattempt_1365199040071_0001_01 exited with exitCode: 143 due to:
Container [pid=24860,containerID=container_1365199040071_0001_01_01] is
running beyond virtual memory limits. Current usage: 269.3mb of 2.0gb physical
memory used; 6.3gb of 4.2gb virtual memory used. Killing container.
{code}
I've uncommented the pmem/vmem lines in the patch to make hadoop2 happy.
{code}
// Tests were failing because this process used 6GB of virtual memory and
was getting killed.
// we up the VM usable so that processes don't get killed.
conf.setInt(yarn.nodemanager.resource.memory-mb, 8 * 1024);
conf.setFloat(yarn.nodemanager.vmem-pmem-ratio, 8.0f);
{code}
Digging into the hadoop 1 issues now.
Make mapreduce tests pass on hadoop2
Key: HBASE-8258
URL: https://issues.apache.org/jira/browse/HBASE-8258
Project: HBase
Issue Type: Bug
Components: mapreduce
Reporter: stack
Priority: Blocker
Fix For: 0.95.1
Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt,
8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch
HBASE-7904 was a first attempt at making this work but it got lost in the
weeds.
This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o
breaking mapreduce on hadoop1)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13624178#comment-13624178 ] Ted Yu commented on HBASE-8258: --- [~jmhsieh]: What do you think of Stack's comment: bq. For example, I don't mind shipping a 0.95.x Development Series release against a hadoop SNAPSHOT; I don't see it as a problem given the quick decay half-life of 0.95.x release Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13624191#comment-13624191 ] Devaraj Das commented on HBASE-8258: bq. I suggest sooner so if any issues in hdfs/mr/yarn, we can help out our parent project by surfacing them quicker (For example, I don't mind shipping a 0.95.x Development Series release against a hadoop SNAPSHOT; I don't see it as a problem given the quick decay half-life of 0.95.x release). Hmm.. fair point, [~stack], given 0.95.x is not meant for production. [~jmhsieh], what do you think about this suggestion. The vmem issues you encountered won't happen with hadoop-2.0.4 since the YARN fixes are committed to not do vmem checks by default. Your patch AFAICT is not sufficient to ride HBase over the hadoop-2.0.3 / 2.0.4 releases. We need some variant of Ted's patch for having HBase pass the unit tests successfully over the 2.0.4 release.. (otherwise we will have this saga all over again in a few weeks). Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13624205#comment-13624205 ] Jonathan Hsieh commented on HBASE-8258: --- [~te...@apache.org] I disagree with stack's comment. I think moving 2.0.3 (a released version) would be reasonable, but would prefer handling that after we get this tackled. I'm against 2.0.4-SNAPSHOT (or any *-SNAPSHOT hadoop). We need be on known good points and ideally move to other known good points. If it ends up that 2.0.3 breaks us somewhere, I feel that we should stand pat where we are stable until there is a version that works for us or if we are forced to upgrade. If 2.0.4 (released) is the safe point and it is released, I'm all for moving to it. We have enough problems with flaky tests, and I'd rather not have to consider hadoop code changes shifting under us as another source of bugs. I'm concerned if upgrading breaks us -- this means hadoop broke compatibility. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13624209#comment-13624209 ] Andrew Purtell commented on HBASE-8258: --- bq. We have enough problems with flaky tests, and I'd rather not have to consider hadoop code changes shifting under us as another source of bugs. I'm concerned if upgrading breaks us -- this means hadoop broke compatibility. +1 On the other hand, I see no problem setting up builds on Apache Jenkins or the EC2 Jenkins which redefines hadoop.version for the build. THen you can specify 2.0.4-SNAPSHOT or whatever you like. Call the builds HBase-2.x-SNAPSHOT or similar. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt, 8258-v2-hadoop-2.0.txt, 8258-v4-hadoop-2.0.txt, hbase-8258-simple.patch HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[
https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622371#comment-13622371
]
Jonathan Hsieh commented on HBASE-8258:
---
The old theory I mentioned above is not relevant. Here's the new idea I'm
working on.
* We are getting an UndeclaredThrowableException when waiting for the MR job to
complete, because
* we get a ConnectException from the mr client trying to talk to 0.0.0.0:8032
(the resource manager port), because
* ... there is a gap here between the local runs and the jenkins runs ...
* the Yarn MRAppMaster for the is getting killed the Yarn nodemanager[1],
(separate processes) because
* virtual memory is being over provisioned in the MRAppMaster
[1]
{code}
2013-04-04 04:46:18,895 WARN [AsyncDispatcher event handler]
resourcemanager.RMAuditLogger(255): USER=jenkins OPERATION=Application
Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App
failed with state: FAILED PERMISSIONS=Application
application_1365075961471_0001 failed 1 times due to AM Container for
appattempt_1365075961471_0001_01 exited with exitCode: 143 due to:
Container [pid=31600,containerID=container_1365075961471_0001_01_01] is
running beyond virtual memory limits. Current usage: 259.7mb of 2.0gb physical
memory used; 5.6gb of 4.2gb virtual memory used. Killing container.
{code}
Make mapreduce tests pass on hadoop2
Key: HBASE-8258
URL: https://issues.apache.org/jira/browse/HBASE-8258
Project: HBase
Issue Type: Bug
Components: mapreduce
Reporter: stack
Priority: Blocker
Fix For: 0.95.1
Attachments: 8258-plain.txt
HBASE-7904 was a first attempt at making this work but it got lost in the
weeds.
This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o
breaking mapreduce on hadoop1)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622388#comment-13622388 ] Jonathan Hsieh commented on HBASE-8258: --- [~devaraj] I'm not getting the same error you are getting. Are you working ontop of the current hadoop2 in the pom 2.0.2-alpha or a different version? (I hvae build with ted's 8258-plain.txt patch applied). Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622406#comment-13622406 ] stack commented on HBASE-8258: -- [~jmhsieh] You saw the discussion over in hbase-7904 about YARN doing process kill because of vmem excess? Ted added this addendum over in hbase-7904: https://issues.apache.org/jira/secure/attachment/12576220/7904-addendum.txt Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622415#comment-13622415 ] Jonathan Hsieh commented on HBASE-8258: --- [~stack] yeah, saw that -- I think instead of depending on the new flag in 2.0.4, we should be able to just bump those numbers to bypass the failures. Currently trying this. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622873#comment-13622873 ] Ted Yu commented on HBASE-8258: --- If MAPREDUCE-5094 goes into 2.0.4, there should be no need to set vmem config parameter. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622954#comment-13622954 ] Jonathan Hsieh commented on HBASE-8258: --- [~te...@apache.org] I'd like to get this fixed without having to upgrade hadoop versions to a snapshot version. I'd also like an explanation of *why* the changes fix problems -- vague statements necessary configuration values without explanation does not help. Ideally this is in the code as comments or at the least in the jira. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13622951#comment-13622951 ] Devaraj Das commented on HBASE-8258: [~yuzhih...@gmail.com], since MAPREDUCE-5094 is not checked in yet, do you want to put the configs for bumping up the vmem (https://issues.apache.org/jira/secure/attachment/12576220/7904-addendum.txt) so that we get a more realistic hadoopqa run. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[
https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13623091#comment-13623091
]
Ted Yu commented on HBASE-8258:
---
[~jmhsieh]:
bq. I'd like to get this fixed without having to upgrade hadoop versions to a
snapshot version.
Did you mean that we should wait for 2.0.4-alpha to come out ? I think for
0.95, we should use 2.0.4-alpha which is to be released soon.
If we use 2.0.3-alpha, we would get (due to capacity-scheduler.xml missing in
artifact):
https://issues.apache.org/jira/browse/HBASE-7904?focusedCommentId=13584497page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13584497
In case you haven't read comments from Siddharth, you can find them starting
with this one :
https://issues.apache.org/jira/browse/HBASE-7904?focusedCommentId=13611080page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13611080
That was how the HBaseConfiguration.merge() call in HBaseTestingUtility.java
came about.
Changes in TestImportExport.java were to align with remaining tests in that
file. The comment below would provide some more context:
https://issues.apache.org/jira/browse/HBASE-7904?focusedCommentId=13617700page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13617700
bq. vague statements necessary configuration values without explanation
I think you were referring to the following:
{code}
// copy or add the necessary configuration values from the map reduce
config to the hbase config
{code}
Such comments have been in TestImportExport. This command can show us the
history:
{code}
$ svn blame
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
| grep 'necessary configurati'
{code}
For SecureBulkLoadEndpoint.java, here is the background:
Secure bulk loading needs to establish staging directory on hdfs. So it calls:
{code}
fs = FileSystem.get(conf);
{code}
The conf above is an instance of CompoundConfiguration. NameNodeProxies would
set up RPC engine by modifying conf.
This results in:
{code}
2013-03-25 22:36:19,493 ERROR [IPC Server handler 2 on 40900]
access.SecureBulkLoadEndpoint$1(240): Failed to complete bulk load
java.lang.UnsupportedOperationException: Immutable Configuration
at
org.apache.hadoop.hbase.CompoundConfiguration.setClass(CompoundConfiguration.java:474)
at org.apache.hadoop.ipc.RPC.setProtocolEngine(RPC.java:193)
at
org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:249)
at
org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:168)
at
org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:129)
at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:421)
at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:388)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:125)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2277)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:86)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2311)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2293)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:317)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:163)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:224)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:218)
{code}
In 8258-v1-hadoop-2.0.txt, I tried to limit the scope of changes while
satisfying requirement from hadoop 2.0
That is, clone a mutable Configuration (an HBaseConfiguration to be more exact)
from the CompoundConfiguration and pass it to FileSystem.get().
I am open to discussion about proposed changes. I can upload the patch onto
review board so that opinion on each change can be expressed better.
I am open to other proposals as well.
Make mapreduce tests pass on hadoop2
Key: HBASE-8258
URL: https://issues.apache.org/jira/browse/HBASE-8258
Project: HBase
Issue Type: Bug
Components: mapreduce
Reporter: stack
Priority: Blocker
Fix For: 0.95.1
Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt
HBASE-7904 was a first attempt at making this work but it got lost in the
weeds.
This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o
breaking mapreduce on hadoop1)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[
https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13623096#comment-13623096
]
Hadoop QA commented on HBASE-8258:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12577098/8258-v1-hadoop-2.0.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
42 warning messages.
{color:red}-1 javac{color}. The applied patch generated 16 javac compiler
warnings (more than the trunk's current 12 warnings).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.security.token.TestTokenAuthentication
org.apache.hadoop.hbase.master.TestTableLockManager
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery.testSplitWhileBulkLoadPhase(TestLoadIncrementalHFilesSplitRecovery.java:298)
at
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles.testSimpleLoad(TestLoadIncrementalHFiles.java:86)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5144//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5144//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5144//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5144//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5144//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5144//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5144//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5144//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5144//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5144//console
This message is automatically generated.
Make mapreduce tests pass on hadoop2
Key: HBASE-8258
URL: https://issues.apache.org/jira/browse/HBASE-8258
Project: HBase
Issue Type: Bug
Components: mapreduce
Reporter: stack
Priority: Blocker
Fix For: 0.95.1
Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt
HBASE-7904 was a first attempt at making this work but it got lost in the
weeds.
This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o
breaking mapreduce on hadoop1)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13623114#comment-13623114 ] Jonathan Hsieh commented on HBASE-8258: --- bq. Did you mean that we should wait for 2.0.4-alpha to come out ? I think for 0.95, we should use 2.0.4-alpha which is to be released soon. I don't want to build off of things that aren't released yet, especially a SNAPSHOT jar -- it will change underneath us and cause us pain by introducing another variable that culd introduces problems under us. bq. That was how the HBaseConfiguration.merge() call in HBaseTestingUtility.java came about It says what we did but I'm not able to get a simple explanation of why from it. At least for the limited test cases, the copyConfigurationValues changes seem to fix the problem (without the merge call). Extraneous calls mean we didn't understand how or why it fixed the problem. The comment on copyConfigurationValues() is a little more helpful (it gives me a clue why, but doesn't get down to which particular variables are important. Without understanding this we don't know if and why all the other mr tests on need these tweaks. Should it get folded in to all them? Why or why not? How can we make it so that future tests don't run into these problems? bq. Such comments have been in TestImportExport. It is more important to convey why things are they way they are in the comments or summarize blocks of code do a single idea that we can verify in review or in code reading. We shouldn't propagate vague comments from the past -- we should make them clearer for the next person. bq. For SecureBulkLoadEndpoint.java, here is the background: From my testing at the moment (on hadoop 2.0.2-alpha) this is out of scope. If and when it becomes an issue with hadoop 2.0.4-beta, we'll address it then. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13623357#comment-13623357 ] Ted Yu commented on HBASE-8258: --- bq. it will change underneath us and cause us pain As I said above, 0.95 needs a stable hadoop 2 release (2.0.4-alpha). For trunk, building against SNAPSHOT artifacts would allow us to find the problem much sooner. If I understand Roman's point of view correctly, he wants to discover regression in upstream project in a similar way. bq. but doesn't get down to which particular variables are important. One such config param is YarnConfiguration.IS_MINI_YARN_CLUSTER which must be set true in order for MiniMRCluster to work. See YARN-129 bq. Should it get folded in to all them? Can you explain folding a little bit ? bq. we should make them clearer for the next person. Agreed. Will add comment once solution is finalized. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 Attachments: 8258-plain.txt, 8258-v1-hadoop-2.0.txt HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621526#comment-13621526 ] Devaraj Das commented on HBASE-8258: Am looking at this. Currently, trying to identify which tests fail. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[
https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621559#comment-13621559
]
Jonathan Hsieh commented on HBASE-8258:
---
builds.apache.org's attempts [1] have been fairly flakey in general recently
for tests that are generally fail due to interference from other processes /
unit test suites.
From an recent runs on beefy physical hardware, I see these test cases
failing regularly:
{code}
org.apache.hadoop.hbase.mapreduce.TestImportExport.testSimpleCase
org.apache.hadoop.hbase.mapreduce.TestImportExport.testMetaExport
org.apache.hadoop.hbase.mapreduce.TestImportExport.testExportScannerBatching
org.apache.hadoop.hbase.mapreduce.TestImportExport.testWithFilter
org.apache.hadoop.hbase.mapreduce.TestImportExport.testWithDeletes
org.apache.hadoop.hbase.mapreduce.TestRowCounter.testRowCounterNoColumn
org.apache.hadoop.hbase.mapreduce.TestRowCounter.testRowCounterHiddenColumn
org.apache.hadoop.hbase.mapreduce.TestRowCounter.testRowCounterExclusiveColumn
{code}
TestImportExport seems flakey (50:50) while TestRowCounter seems to failed
consistently for while.
[1] https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0
Make mapreduce tests pass on hadoop2
Key: HBASE-8258
URL: https://issues.apache.org/jira/browse/HBASE-8258
Project: HBase
Issue Type: Bug
Components: mapreduce
Reporter: stack
Priority: Blocker
Fix For: 0.95.1
HBASE-7904 was a first attempt at making this work but it got lost in the
weeds.
This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o
breaking mapreduce on hadoop1)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[
https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621613#comment-13621613
]
Devaraj Das commented on HBASE-8258:
Thanks [~jmhsieh] for the list of the tests.
Maybe I am missing something, but when I run the command:
{noformat}
mvn -PlocalTests -Dhadoop.profile=2.0 -Dtest=TestImportExport test
{noformat}
the test fails but the error is very fundamental. There is a
{noformat}
java.lang.NoSuchMethodError:
org.apache.hadoop.hdfs.MiniDFSCluster.lt;initgt;(ILorg/apache/hadoop/conf/Configuration;IZZZLorg/apache/hadoop/hdfs/server/common/HdfsConstants$StartupOption;[Ljava/lang/String;[Ljava/lang/String;[J)V
{noformat}
Make mapreduce tests pass on hadoop2
Key: HBASE-8258
URL: https://issues.apache.org/jira/browse/HBASE-8258
Project: HBase
Issue Type: Bug
Components: mapreduce
Reporter: stack
Priority: Blocker
Fix For: 0.95.1
HBASE-7904 was a first attempt at making this work but it got lost in the
weeds.
This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o
breaking mapreduce on hadoop1)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[ https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621618#comment-13621618 ] Jonathan Hsieh commented on HBASE-8258: --- Hm.. you are right -- on my setup on April 1 (before the revert of HBASE-7904) trunk runs I have 21 tests in org.apache.hadoop.hbase.mapreduce. The TestImportExport cases are actually flaking between failure and non being reported (hanging?). Here's my theory from a previous attempt. I believe several months ago I narrowed the problem to hbase and the yarn/mr2 instances writing to different hdfs's. I might have this flipped but it was something like hbase would write and read from a the minihdfs cluster's hfds. So a table would be written to the minihdfs. The mr2 export job would read from hbase, but write data to a its different local file system. The import job would check for files in its minihdfs cluster and not find them and then fail. I'm in the process of confirming if this is still the case. Make mapreduce tests pass on hadoop2 Key: HBASE-8258 URL: https://issues.apache.org/jira/browse/HBASE-8258 Project: HBase Issue Type: Bug Components: mapreduce Reporter: stack Priority: Blocker Fix For: 0.95.1 HBASE-7904 was a first attempt at making this work but it got lost in the weeds. This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o breaking mapreduce on hadoop1) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[
https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621633#comment-13621633
]
Hadoop QA commented on HBASE-8258:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12576898/8258-plain.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5122//console
This message is automatically generated.
Make mapreduce tests pass on hadoop2
Key: HBASE-8258
URL: https://issues.apache.org/jira/browse/HBASE-8258
Project: HBase
Issue Type: Bug
Components: mapreduce
Reporter: stack
Priority: Blocker
Fix For: 0.95.1
Attachments: 8258-plain.txt
HBASE-7904 was a first attempt at making this work but it got lost in the
weeds.
This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o
breaking mapreduce on hadoop1)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[
https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621647#comment-13621647
]
Hadoop QA commented on HBASE-8258:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12576904/8258-plain.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 javac{color}. The patch appears to cause mvn compile goal to
fail.
{color:red}-1 findbugs{color}. The patch appears to cause Findbugs
(version 1.3.9) to fail.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5124//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5124//console
This message is automatically generated.
Make mapreduce tests pass on hadoop2
Key: HBASE-8258
URL: https://issues.apache.org/jira/browse/HBASE-8258
Project: HBase
Issue Type: Bug
Components: mapreduce
Reporter: stack
Priority: Blocker
Fix For: 0.95.1
Attachments: 8258-plain.txt
HBASE-7904 was a first attempt at making this work but it got lost in the
weeds.
This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o
breaking mapreduce on hadoop1)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8258) Make mapreduce tests pass on hadoop2
[
https://issues.apache.org/jira/browse/HBASE-8258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13621696#comment-13621696
]
Hadoop QA commented on HBASE-8258:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12576907/8258-plain.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
42 warning messages.
{color:red}-1 javac{color}. The applied patch generated 16 javac compiler
warnings (more than the trunk's current 12 warnings).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestMultiTableInputFormat
org.apache.hadoop.hbase.mapreduce.TestRowCounter
org.apache.hadoop.hbase.mapreduce.TestImportTsv
org.apache.hadoop.hbase.mapreduce.TestImportExport
org.apache.hadoop.hbase.mapred.TestTableMapReduce
org.apache.hadoop.hbase.security.token.TestTokenAuthentication
org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
org.apache.hadoop.hbase.mapreduce.TestWALPlayer
org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper
org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery.testSplitWhileBulkLoadPhase(TestLoadIncrementalHFilesSplitRecovery.java:298)
at
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles.testSimpleLoad(TestLoadIncrementalHFiles.java:86)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5126//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5126//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5126//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5126//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5126//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5126//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5126//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5126//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5126//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5126//console
This message is automatically generated.
Make mapreduce tests pass on hadoop2
Key: HBASE-8258
URL: https://issues.apache.org/jira/browse/HBASE-8258
Project: HBase
Issue Type: Bug
Components: mapreduce
Reporter: stack
Priority: Blocker
Fix For: 0.95.1
Attachments: 8258-plain.txt
HBASE-7904 was a first attempt at making this work but it got lost in the
weeds.
This is a new attempt at making hbase mapreduce jobs run on hadoop2 (w/o
breaking mapreduce on hadoop1)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
