[jira] [Updated] (HBASE-7938) Add integration test for ImportTsv/LoadIncrementalHFiles workflow
[ https://issues.apache.org/jira/browse/HBASE-7938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-7938: Release Note: The new IntegrationTestImportTsv is written in the style of the recently added IntegrationTestLoadAndVerify. It is designed to be run from maven or directly against a distributed cluster. It shares core test logic with TestImportTsv. To run it with maven, first refresh your install jars, and the run the test: $ mvn clean package install -DskipTests $ cd hbase-it $ mvn failsafe:integration-test -Dit.test=IntegrationTestImportTsv NOTE: Running from maven fails because HFileOutputFormat is creating the partition file on localFs, not HDFS. Or run it stand-alone: $ HBASE_CLASSPATH="hbase-it/target/hbase-it--tests.jar" \ ./bin/hbase org.apache.hadoop.hbase.mapreduce.IntegrationTestImportTsv When run stand-alone, it supports all the usual Hadoop Tool arguments, particularly -conf. ImportTsv was refactored to make code sharing easier. It is upgraded from MediumTest to LargeTest because it take 180+ seconds to run on my machine. Tests for the TsvParser are separated into their own SmallTest. > Add integration test for ImportTsv/LoadIncrementalHFiles workflow > - > > Key: HBASE-7938 > URL: https://issues.apache.org/jira/browse/HBASE-7938 > Project: HBase > Issue Type: Sub-task > Components: mapreduce >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 0.95.0, 0.98.0 > > Attachments: > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch, > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch, > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch > > > We have existing unit tests for smoke-testing the packaged MR jobs, however > they do not create a runtime environment that is true to running on a real MR > cluster. This is particularly true in regard to classpaths (HBASE-7934) but > also other static state (HBASE-4802). An integration test that can be pointed > to run on a pseudo-distributed Hadoop deployed on localhost would find these > kinds of problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7938) Add integration test for ImportTsv/LoadIncrementalHFiles workflow
[ https://issues.apache.org/jira/browse/HBASE-7938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-7938: Summary: Add integration test for ImportTsv/LoadIncrementalHFiles workflow (was: Add integration test for various MapReduce workflows) > Add integration test for ImportTsv/LoadIncrementalHFiles workflow > - > > Key: HBASE-7938 > URL: https://issues.apache.org/jira/browse/HBASE-7938 > Project: HBase > Issue Type: Bug > Components: mapreduce >Reporter: Nick Dimiduk > Fix For: 0.95.0, 0.98.0, 0.94.7 > > > We have existing unit tests for smoke-testing the packaged MR jobs, however > they do not create a runtime environment that is true to running on a real MR > cluster. This is particularly true in regard to classpaths (HBASE-7934) but > also other static state (HBASE-4802). An integration test that can be pointed > to run on a pseudo-distributed Hadoop deployed on localhost would find these > kinds of problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7938) Add integration test for ImportTsv/LoadIncrementalHFiles workflow
[ https://issues.apache.org/jira/browse/HBASE-7938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-7938: Attachment: 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch This patch depends on HBASE-8011. The new IntegrationTestImportTsv is written in the style of the recently added IntegrationTestLoadAndVerify. It is designed to be run from maven or directly against a distributed cluster. It shares core test logic with TestImportTsv, from which it inherits. To run it with maven, first refresh your install jars, and the run the test: $ mvn clean package install -DskipTests $ cd hbase-it $ mvn test-compile failsafe:integration-test -Dit.test=IntegrationTestImportTsv Or run it stand-alone: $ HBASE_CLASSPATH="hbase-it/target/hbase-it--tests.jar" \ ./bin/hbase org.apache.hadoop.hbase.mapreduce.IntegrationTestImportTsv When run stand-alone, it supports all the usual Hadoop Tool arguments, particularly -conf. ImportTsv was refactored to make code sharing easier. It is upgraded from MediumTest to LargeTest because it take 180+ seconds to run on my machine. Tests for the TsvParser are separated into their own SmallTest. > Add integration test for ImportTsv/LoadIncrementalHFiles workflow > - > > Key: HBASE-7938 > URL: https://issues.apache.org/jira/browse/HBASE-7938 > Project: HBase > Issue Type: Bug > Components: mapreduce >Reporter: Nick Dimiduk > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch > > > We have existing unit tests for smoke-testing the packaged MR jobs, however > they do not create a runtime environment that is true to running on a real MR > cluster. This is particularly true in regard to classpaths (HBASE-7934) but > also other static state (HBASE-4802). An integration test that can be pointed > to run on a pseudo-distributed Hadoop deployed on localhost would find these > kinds of problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7938) Add integration test for ImportTsv/LoadIncrementalHFiles workflow
[ https://issues.apache.org/jira/browse/HBASE-7938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-7938: Issue Type: Sub-task (was: Bug) Parent: HBASE-8084 > Add integration test for ImportTsv/LoadIncrementalHFiles workflow > - > > Key: HBASE-7938 > URL: https://issues.apache.org/jira/browse/HBASE-7938 > Project: HBase > Issue Type: Sub-task > Components: mapreduce >Reporter: Nick Dimiduk > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch > > > We have existing unit tests for smoke-testing the packaged MR jobs, however > they do not create a runtime environment that is true to running on a real MR > cluster. This is particularly true in regard to classpaths (HBASE-7934) but > also other static state (HBASE-4802). An integration test that can be pointed > to run on a pseudo-distributed Hadoop deployed on localhost would find these > kinds of problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7938) Add integration test for ImportTsv/LoadIncrementalHFiles workflow
[ https://issues.apache.org/jira/browse/HBASE-7938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-7938: Assignee: Nick Dimiduk Status: Patch Available (was: Open) > Add integration test for ImportTsv/LoadIncrementalHFiles workflow > - > > Key: HBASE-7938 > URL: https://issues.apache.org/jira/browse/HBASE-7938 > Project: HBase > Issue Type: Sub-task > Components: mapreduce >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch > > > We have existing unit tests for smoke-testing the packaged MR jobs, however > they do not create a runtime environment that is true to running on a real MR > cluster. This is particularly true in regard to classpaths (HBASE-7934) but > also other static state (HBASE-4802). An integration test that can be pointed > to run on a pseudo-distributed Hadoop deployed on localhost would find these > kinds of problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7938) Add integration test for ImportTsv/LoadIncrementalHFiles workflow
[ https://issues.apache.org/jira/browse/HBASE-7938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-7938: Fix Version/s: (was: 0.94.7) > Add integration test for ImportTsv/LoadIncrementalHFiles workflow > - > > Key: HBASE-7938 > URL: https://issues.apache.org/jira/browse/HBASE-7938 > Project: HBase > Issue Type: Sub-task > Components: mapreduce >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 0.95.0, 0.98.0 > > Attachments: > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch > > > We have existing unit tests for smoke-testing the packaged MR jobs, however > they do not create a runtime environment that is true to running on a real MR > cluster. This is particularly true in regard to classpaths (HBASE-7934) but > also other static state (HBASE-4802). An integration test that can be pointed > to run on a pseudo-distributed Hadoop deployed on localhost would find these > kinds of problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7938) Add integration test for ImportTsv/LoadIncrementalHFiles workflow
[ https://issues.apache.org/jira/browse/HBASE-7938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-7938: Status: Open (was: Patch Available) > Add integration test for ImportTsv/LoadIncrementalHFiles workflow > - > > Key: HBASE-7938 > URL: https://issues.apache.org/jira/browse/HBASE-7938 > Project: HBase > Issue Type: Sub-task > Components: mapreduce >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 0.95.0, 0.98.0 > > Attachments: > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch > > > We have existing unit tests for smoke-testing the packaged MR jobs, however > they do not create a runtime environment that is true to running on a real MR > cluster. This is particularly true in regard to classpaths (HBASE-7934) but > also other static state (HBASE-4802). An integration test that can be pointed > to run on a pseudo-distributed Hadoop deployed on localhost would find these > kinds of problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7938) Add integration test for ImportTsv/LoadIncrementalHFiles workflow
[ https://issues.apache.org/jira/browse/HBASE-7938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-7938: Attachment: 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch New patch makes use of the subdirectory features of the test util's test directory method. Running via mvn and IntegrationTestsDriver locally, will report back. [~enis] care to give this patch a go as well? > Add integration test for ImportTsv/LoadIncrementalHFiles workflow > - > > Key: HBASE-7938 > URL: https://issues.apache.org/jira/browse/HBASE-7938 > Project: HBase > Issue Type: Sub-task > Components: mapreduce >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 0.95.0, 0.98.0 > > Attachments: > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch, > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch > > > We have existing unit tests for smoke-testing the packaged MR jobs, however > they do not create a runtime environment that is true to running on a real MR > cluster. This is particularly true in regard to classpaths (HBASE-7934) but > also other static state (HBASE-4802). An integration test that can be pointed > to run on a pseudo-distributed Hadoop deployed on localhost would find these > kinds of problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7938) Add integration test for ImportTsv/LoadIncrementalHFiles workflow
[ https://issues.apache.org/jira/browse/HBASE-7938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-7938: Hadoop Flags: Reviewed Status: Patch Available (was: Open) > Add integration test for ImportTsv/LoadIncrementalHFiles workflow > - > > Key: HBASE-7938 > URL: https://issues.apache.org/jira/browse/HBASE-7938 > Project: HBase > Issue Type: Sub-task > Components: mapreduce >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 0.95.0, 0.98.0 > > Attachments: > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch, > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch > > > We have existing unit tests for smoke-testing the packaged MR jobs, however > they do not create a runtime environment that is true to running on a real MR > cluster. This is particularly true in regard to classpaths (HBASE-7934) but > also other static state (HBASE-4802). An integration test that can be pointed > to run on a pseudo-distributed Hadoop deployed on localhost would find these > kinds of problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7938) Add integration test for ImportTsv/LoadIncrementalHFiles workflow
[ https://issues.apache.org/jira/browse/HBASE-7938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-7938: Status: Open (was: Patch Available) > Add integration test for ImportTsv/LoadIncrementalHFiles workflow > - > > Key: HBASE-7938 > URL: https://issues.apache.org/jira/browse/HBASE-7938 > Project: HBase > Issue Type: Sub-task > Components: mapreduce >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 0.95.0, 0.98.0 > > Attachments: > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch, > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch > > > We have existing unit tests for smoke-testing the packaged MR jobs, however > they do not create a runtime environment that is true to running on a real MR > cluster. This is particularly true in regard to classpaths (HBASE-7934) but > also other static state (HBASE-4802). An integration test that can be pointed > to run on a pseudo-distributed Hadoop deployed on localhost would find these > kinds of problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7938) Add integration test for ImportTsv/LoadIncrementalHFiles workflow
[ https://issues.apache.org/jira/browse/HBASE-7938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-7938: Attachment: 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch How do you guys let me get away with this slop? New patch, cleans up: - deleted redundant code from TestImportTsv, using existing code in the test utilities. - added a couple helpers to HBaseCommonTestingUtility. - set dataTestDirOnTestFS to deleteOnExit when TestFS is local. This matched behavior of when TestFS is not local. - IntegrationTestImportTsv no longer extends TestImportTsv, primarily because the IT suffered a kind of split-brain syndrom when resolving the Configuration. Instead of inheritance, just helpers in test utils and protected static methods from ImportTsv. The test works as intended when run stand-alone, but FAILS when run via maven. It looks like HFileOutputFormat is creating the paritions file on localFs instead of HDFS. I don't yet know why, but my most recent patch on HBASE-4285 gets rid of that nonsense, at least in part. I'll try rebasing that patch onto this one and see how the test behaves. > Add integration test for ImportTsv/LoadIncrementalHFiles workflow > - > > Key: HBASE-7938 > URL: https://issues.apache.org/jira/browse/HBASE-7938 > Project: HBase > Issue Type: Sub-task > Components: mapreduce >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 0.95.0, 0.98.0 > > Attachments: > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch, > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch, > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch > > > We have existing unit tests for smoke-testing the packaged MR jobs, however > they do not create a runtime environment that is true to running on a real MR > cluster. This is particularly true in regard to classpaths (HBASE-7934) but > also other static state (HBASE-4802). An integration test that can be pointed > to run on a pseudo-distributed Hadoop deployed on localhost would find these > kinds of problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7938) Add integration test for ImportTsv/LoadIncrementalHFiles workflow
[ https://issues.apache.org/jira/browse/HBASE-7938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-7938: Status: Patch Available (was: Open) > Add integration test for ImportTsv/LoadIncrementalHFiles workflow > - > > Key: HBASE-7938 > URL: https://issues.apache.org/jira/browse/HBASE-7938 > Project: HBase > Issue Type: Sub-task > Components: mapreduce >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 0.95.0, 0.98.0 > > Attachments: > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch, > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch, > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch > > > We have existing unit tests for smoke-testing the packaged MR jobs, however > they do not create a runtime environment that is true to running on a real MR > cluster. This is particularly true in regard to classpaths (HBASE-7934) but > also other static state (HBASE-4802). An integration test that can be pointed > to run on a pseudo-distributed Hadoop deployed on localhost would find these > kinds of problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7938) Add integration test for ImportTsv/LoadIncrementalHFiles workflow
[ https://issues.apache.org/jira/browse/HBASE-7938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-7938: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed this to 0.95 and trunk. Thanks Nick. > Add integration test for ImportTsv/LoadIncrementalHFiles workflow > - > > Key: HBASE-7938 > URL: https://issues.apache.org/jira/browse/HBASE-7938 > Project: HBase > Issue Type: Sub-task > Components: mapreduce >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 0.95.0, 0.98.0 > > Attachments: > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch, > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch, > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch > > > We have existing unit tests for smoke-testing the packaged MR jobs, however > they do not create a runtime environment that is true to running on a real MR > cluster. This is particularly true in regard to classpaths (HBASE-7934) but > also other static state (HBASE-4802). An integration test that can be pointed > to run on a pseudo-distributed Hadoop deployed on localhost would find these > kinds of problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
