[
https://issues.apache.org/jira/browse/HBASE-7605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13627999#comment-13627999
]
Jonathan Hsieh commented on HBASE-7605:
---------------------------------------
tl;dr Test is correct but there is some regression when using hadoop2. Going
to reduce the size of the test so that it finishes in a reasonable amount of
time and passes. Would like to close with patch that makes it work but will
file jira to investigate why the big h1 vs h2 perf difference.
On my test runs against hadoop2, I get failures due to timeouts with
TestMiniClusterLoadParallel and TestMiniClusterLoadSequential.
With hadoop1 runs from the ec2 test trunk on hadoop2 instance [1], each of
these individual cases take this long to run.
*Sequential
{code}
Test name Duration Status
loadTest[0] 1 min 11 sec Passed
loadTest[1] 48 sec Passed
loadTest[2] 22 sec Passed
loadTest[3] 27 sec Passed
{code}
*Parallel
{code}
Test name Duration Status
loadTest[0] 1 min 18 sec Passed
loadTest[1] 51 sec Passed
loadTest[2] 19 sec Passed
loadTest[3] 20 sec Passed
{code}
I ran then locally on my box with hadoop1 profile an got this:
With hadoop2 I did some experiments on my local machine, bumping up Sequential
from 180s to 360s timeouts per test. They passed, but barely before timing
out. For some reason they are 2-3x slower. (Not good)
{code}
<testcase time="348.761"
classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel"
name="loadTest[0]"/>
<testcase time="343.577"
classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel"
name="loadTest[1]"/>
<testcase time="46.397"
classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel"
name="loadTest[2]"/>
<testcase time="45.845"
classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel"
name="loadTest[3]"/>
...
<testcase time="358.088"
classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential"
name="loadTest[0]"/>
<testcase time="357.654"
classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential"
name="loadTest[1]"/>
<testcase time="61.306"
classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential"
name="loadTest[2]"/>
<testcase time="61.263"
classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential"
name="loadTest[3]"/>
{code}
Instead of bumping timeout up, I'm going to change the tests so that it does an
order of magnitude less work, and finishes on the order of 10-30s instead of
100-300s.
Here are the hadoop2 vs hadoop1 results (bumping down from 10000 keys to 1000
keys)
{code}
hadoop1:
<testcase time="14.929"
classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel"
name="loadTest[0]"/>
<testcase time="12.747"
classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel"
name="loadTest[1]"/>
<testcase time="12.005"
classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel"
name="loadTest[2]"/>
<testcase time="11.663"
classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel"
name="loadTest[3]"/>
<testcase time="15.956"
classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential"
name="loadTest[0]"/>
<testcase time="14.321"
classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential"
name="loadTest[1]"/>
<testcase time="11.541"
classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential"
name="loadTest[2]"/>
<testcase time="11.588"
classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential"
name="loadTest[3]"/>
hadoop2:
<testcase time="43.703"
classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel"
name="loadTest[0]"/>
<testcase time="41.542"
classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel"
name="loadTest[1]"/>
<testcase time="10.711"
classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel"
name="loadTest[2]"/>
<testcase time="8.801"
classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel"
name="loadTest[3]"/>
<testcase time="45.986"
classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential"
name="loadTest[0]"/>
<testcase time="42.037"
classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential"
name="loadTest[1]"/>
<testcase time="12.56"
classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential"
name="loadTest[2]"/>
<testcase time="12.01"
classname="org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential"
name="loadTest[3]"/>
{code}
[1]http://54.241.6.143/job/HBase-TRUNK-Hadoop-2/org.apache.hbase$hbase-server/53/
> TestMiniClusterLoadSequential fails in trunk build on hadoop 2
> --------------------------------------------------------------
>
> Key: HBASE-7605
> URL: https://issues.apache.org/jira/browse/HBASE-7605
> Project: HBase
> Issue Type: Sub-task
> Reporter: Ted Yu
> Priority: Critical
> Fix For: 0.95.1
>
>
> From HBase-TRUNK-on-Hadoop-2.0.0 #354:
> loadTest[0](org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential):
> test timed out after 120000 milliseconds
> loadTest[1](org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential):
> test timed out after 120000 milliseconds
> loadTest[2](org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential):
> test timed out after 120000 milliseconds
> loadTest[3](org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential):
> test timed out after 120000 milliseconds
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
