[jira] [Commented] (HBASE-11647) MOB integration testing
[ https://issues.apache.org/jira/browse/HBASE-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14637840#comment-14637840 ] Hudson commented on HBASE-11647: FAILURE: Integrated in HBase-TRUNK #6672 (See [https://builds.apache.org/job/HBase-TRUNK/6672/]) HBASE-11647 MOB integration testing. (Jingcheng Du) (anoopsamjohn: rev 3e563c5cc72152cec0742956b7ec61b1841e78ac) * hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestIngestWithMOB.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/LoadTestDataGeneratorWithMOB.java HBASE-11647 MOB integration testing. - addendum(Jingcheng Du) (anoopsamjohn: rev 0c86d83e1f966b5e0c72e53664f7e9ff5a71e488) * hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestIngestWithMOB.java HBASE-11647 addendum to fix compile issues. (anoopsamjohn: rev e5d3850776174a63ddc2e0b5ead58409ca7c8706) * hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestIngestWithMOB.java MOB integration testing --- Key: HBASE-11647 URL: https://issues.apache.org/jira/browse/HBASE-11647 Project: HBase Issue Type: Sub-task Components: Performance, test Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-11647-without-sweep-tool-V2.diff, HBASE-11647-without-sweep-tool-V3.diff, HBASE-11647-without-sweep-tool-addendum.diff, HBASE-11647-without-sweep-tool.diff, HBASE-11647.diff, HBASE-11647_addendum2.patch The integration testings include the integration function testing and performance testing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11647) MOB integration testing
[
https://issues.apache.org/jira/browse/HBASE-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146620#comment-14146620
]
Jonathan Hsieh commented on HBASE-11647:
This can be run by using this command line:
{code}
bin/hbase org.apache.hadoop.hbase.IntegrationTestingestWithMOB
{code}
MOB integration testing
---
Key: HBASE-11647
URL: https://issues.apache.org/jira/browse/HBASE-11647
Project: HBase
Issue Type: Sub-task
Components: Performance, test
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Fix For: hbase-11339
Attachments: HBASE-11647-without-sweep-tool-V2.diff,
HBASE-11647-without-sweep-tool-V3.diff,
HBASE-11647-without-sweep-tool-addendum.diff,
HBASE-11647-without-sweep-tool.diff, HBASE-11647.diff,
HBASE-11647_addendum2.patch
The integration testings include the integration function testing and
performance testing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11647) MOB integration testing
[ https://issues.apache.org/jira/browse/HBASE-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126852#comment-14126852 ] Anoop Sam John commented on HBASE-11647: +1 for this addendum. Will commit now. MOB integration testing --- Key: HBASE-11647 URL: https://issues.apache.org/jira/browse/HBASE-11647 Project: HBase Issue Type: Sub-task Components: Performance, test Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-11647-without-sweep-tool-V2.diff, HBASE-11647-without-sweep-tool-V3.diff, HBASE-11647-without-sweep-tool-addendum.diff, HBASE-11647-without-sweep-tool.diff, HBASE-11647.diff The integration testings include the integration function testing and performance testing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11647) MOB integration testing
[ https://issues.apache.org/jira/browse/HBASE-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126861#comment-14126861 ] Anoop Sam John commented on HBASE-11647: Thanks for the addendum Jingcheng. Pushed to the branch. MOB integration testing --- Key: HBASE-11647 URL: https://issues.apache.org/jira/browse/HBASE-11647 Project: HBase Issue Type: Sub-task Components: Performance, test Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-11647-without-sweep-tool-V2.diff, HBASE-11647-without-sweep-tool-V3.diff, HBASE-11647-without-sweep-tool-addendum.diff, HBASE-11647-without-sweep-tool.diff, HBASE-11647.diff The integration testings include the integration function testing and performance testing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11647) MOB integration testing
[ https://issues.apache.org/jira/browse/HBASE-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124308#comment-14124308 ] Anoop Sam John commented on HBASE-11647: +1. Will commit now. MOB integration testing --- Key: HBASE-11647 URL: https://issues.apache.org/jira/browse/HBASE-11647 Project: HBase Issue Type: Sub-task Components: Performance, test Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-11647-without-sweep-tool-V2.diff, HBASE-11647-without-sweep-tool-V3.diff, HBASE-11647-without-sweep-tool.diff, HBASE-11647.diff The integration testings include the integration function testing and performance testing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11647) MOB integration testing
[ https://issues.apache.org/jira/browse/HBASE-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117266#comment-14117266 ] ramkrishna.s.vasudevan commented on HBASE-11647: Just using LTT and passing the required MOB related args can we run this test without using it as IT? Not a major just asking? MOB integration testing --- Key: HBASE-11647 URL: https://issues.apache.org/jira/browse/HBASE-11647 Project: HBase Issue Type: Sub-task Components: Performance, test Reporter: Jingcheng Du Assignee: Jingcheng Du Attachments: HBASE-11647-without-sweep-tool-V2.diff, HBASE-11647-without-sweep-tool.diff, HBASE-11647.diff The integration testings include the integration function testing and performance testing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11647) MOB integration testing
[
https://issues.apache.org/jira/browse/HBASE-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117590#comment-14117590
]
Anoop Sam John commented on HBASE-11647:
+1
{code}
+ public void initialize(String[] args) {
+super.initialize(args);
+String mobColumnFamily = args[0];
+int minMobDataSize = Integer.parseInt(args[1]);
+int maxMobDataSize = Integer.parseInt(args[2]);
{code}
minor : Just check the length of args to be 3. If not throw
IllegalArgumentException.
MOB integration testing
---
Key: HBASE-11647
URL: https://issues.apache.org/jira/browse/HBASE-11647
Project: HBase
Issue Type: Sub-task
Components: Performance, test
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Attachments: HBASE-11647-without-sweep-tool-V2.diff,
HBASE-11647-without-sweep-tool.diff, HBASE-11647.diff
The integration testings include the integration function testing and
performance testing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11647) MOB integration testing
[
https://issues.apache.org/jira/browse/HBASE-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117648#comment-14117648
]
Hadoop QA commented on HBASE-11647:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12665738/HBASE-11647-without-sweep-tool-V2.diff
against trunk revision .
ATTACHMENT ID: 12665738
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestRegionRebalancing
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10665//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10665//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10665//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10665//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10665//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10665//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10665//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10665//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10665//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10665//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10665//console
This message is automatically generated.
MOB integration testing
---
Key: HBASE-11647
URL: https://issues.apache.org/jira/browse/HBASE-11647
Project: HBase
Issue Type: Sub-task
Components: Performance, test
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Attachments: HBASE-11647-without-sweep-tool-V2.diff,
HBASE-11647-without-sweep-tool.diff, HBASE-11647.diff
The integration testings include the integration function testing and
performance testing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11647) MOB integration testing
[
https://issues.apache.org/jira/browse/HBASE-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117898#comment-14117898
]
Hadoop QA commented on HBASE-11647:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12665835/HBASE-11647-without-sweep-tool-V3.diff
against trunk revision .
ATTACHMENT ID: 12665835
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.regionserver.TestReplicationThrottler
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10667//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10667//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10667//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10667//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10667//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10667//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10667//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10667//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10667//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10667//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10667//console
This message is automatically generated.
MOB integration testing
---
Key: HBASE-11647
URL: https://issues.apache.org/jira/browse/HBASE-11647
Project: HBase
Issue Type: Sub-task
Components: Performance, test
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Attachments: HBASE-11647-without-sweep-tool-V2.diff,
HBASE-11647-without-sweep-tool-V3.diff, HBASE-11647-without-sweep-tool.diff,
HBASE-11647.diff
The integration testings include the integration function testing and
performance testing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11647) MOB integration testing
[
https://issues.apache.org/jira/browse/HBASE-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113612#comment-14113612
]
Anoop Sam John commented on HBASE-11647:
IntegrationTestIngestMOB
columnDescriptor.setValue(MobConstants.IS_MOB, true); You should pass this
as boolean than String?
columnDescriptor.setValue(MobConstants.MOB_THRESHOLD,
String.valueOf(threshold)); - Pass it as long than String?
{code}
} else if(dataGen instanceof LoadTestDataGeneratorMOB) {
LOG.info(Using LoadTestDataGeneratorMOB);
String mobCf = clazzAndArgs[1];
int minMobDataSize = Integer.parseInt(clazzAndArgs[2]);
int maxMobDataSize = Integer.parseInt(clazzAndArgs[3]);
LoadTestDataGeneratorMOB mobDatGen =
(LoadTestDataGeneratorMOB)dataGen;
mobDatGen.configureMob(mobCf.getBytes(), minMobDataSize,
maxMobDataSize);
args = clazzAndArgs.length==4? new String[0] :
Arrays.copyOfRange(clazzAndArgs, 4, clazzAndArgs.length);
} else {
args = clazzAndArgs.length == 1 ? new String[0] :
Arrays.copyOfRange(clazzAndArgs, 1,
clazzAndArgs.length);
}
dataGen.initialize(args);
{code}
hmm this is bad! Why we need this way here? Note the else part and passing all
the args to initialize() You can handle these MOB specific args in
LoadTestDataGeneratorMOB#initialize
{code}
sb.append(COLON);
sb.append(minMobDataSize);
sb.append(COLON);
sb.append(maxMobDataSize);
{code}
Can we not hard code this but take as arg also? Can go with current values as
def value.
{code}
public class LoadTestDataGeneratorMOB
extends MultiThreadedAction.DefaultDataGenerator {
private byte[] mobCf;
private int minMobColDataSize;
private int maxMobColDataSize;
{code}
Unused state variables. Pls remove.
MOB integration testing
---
Key: HBASE-11647
URL: https://issues.apache.org/jira/browse/HBASE-11647
Project: HBase
Issue Type: Sub-task
Components: Performance, test
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Attachments: HBASE-11647-without-sweep-tool.diff, HBASE-11647.diff
The integration testings include the integration function testing and
performance testing.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11647) MOB integration testing
[
https://issues.apache.org/jira/browse/HBASE-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107716#comment-14107716
]
Jonathan Hsieh commented on HBASE-11647:
I like where these tests are going. Can you post some examples of how you kick
off the test in the jira?
As an aside, the documentation in the IntegrationTestIngest could be improved,
as could the docs in this the IntegrationTestIngestMOB.
Explicitly call out this his an extension of the IntegrationTestIngest. that
uses LoadTestTool to generate and writes mob sized data into hbase and verify
it.
{quote}
+/**
+ * Integration Test for MOB ingest.
+ */
+@Category(IntegrationTests.class)
+public class IntegrationTestIngestMOB extends IntegrationTestIngest {
{quote}
Please provide some way of getting usage instructions and what the
LoadTestDataGeneraetyorMob:x:y:z:w args are!
{quote}
+ public static void main(String[] args) throws Exception {
+Configuration conf = HBaseConfiguration.create();
+IntegrationTestingUtility.setUseDistributedCluster(conf);
+int ret = ToolRunner.run(conf, new IntegrationTestIngestMOB(), args);
+System.exit(ret);
+ }
{quote}
Add a comment here saying we add a another value generator that has different
cols data size bounds to expcicitly test the mobs.
{quote}
+/**
+ * A load test data generator for MOB
+ */
+public class LoadTestDataGeneratorMOB
+extends MultiThreadedAction.DefaultDataGenerator {
+
{quote}
This instanceof is a bad smell -- it breaks encapsulation -- can we do this in
a cleaner way? Maybe add in a String... or Object... arg so that we can
handle all of these and without having to do the instanceof? At the least,
please leave a TODO here to refactor so that we just use an interface and
inheritance properly to avoid the instanceof.
{quote}
}
+ } else if(dataGen instanceof LoadTestDataGeneratorMOB) {
+LOG.info(Using LoadTestDataGeneratorMOB);
+String mobCf = clazzAndArgs[1];
+int minMobDataSize = Integer.parseInt(clazzAndArgs[2]);
+int maxMobDataSize = Integer.parseInt(clazzAndArgs[3]);
+LoadTestDataGeneratorMOB mobDatGen = (LoadTestDataGeneratorMOB)dataGen;
+mobDatGen.configureMob(mobCf.getBytes(), minMobDataSize,
maxMobDataSize);
+args = clazzAndArgs.length==4? new String[0] :
Arrays.copyOfRange(clazzAndArgs, 4, clazzAndArgs.length);
} else {
{quote}
MOB integration testing
---
Key: HBASE-11647
URL: https://issues.apache.org/jira/browse/HBASE-11647
Project: HBase
Issue Type: Sub-task
Components: Performance, test
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Attachments: HBASE-11647.diff
The integration testings include the integration function testing and
performance testing.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11647) MOB integration testing
[ https://issues.apache.org/jira/browse/HBASE-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14086319#comment-14086319 ] Jonathan Hsieh commented on HBASE-11647: This idelaly would be a set of tests derived from IntegrationTest that would constantly read and write MOB sized data to a cf with the mob feature enabled. Along with the regular chaos monkeys, we would also need actions that would trigger the ttl and sweep jobs and also alter mob thresholds and cause normal compactions. MOB integration testing --- Key: HBASE-11647 URL: https://issues.apache.org/jira/browse/HBASE-11647 Project: HBase Issue Type: Sub-task Components: Performance, test Reporter: Jingcheng Du Assignee: Jingcheng Du The integration testings include the integration function testing and performance testing. -- This message was sent by Atlassian JIRA (v6.2#6252)
