This is an automated email from the ASF dual-hosted git repository.
zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 67ef629486b HIVE-26021: Change integration tests under DBInstallBase
to Checkin tests (Stamatis Zampetakis, reviewed by Peter Vary)
67ef629486b is described below
commit 67ef629486ba38b1d3e0f400bee0073fa3c4e989
Author: Stamatis Zampetakis <zabe...@gmail.com>
AuthorDate: Mon Jun 20 13:38:20 2022 +0200
HIVE-26021: Change integration tests under DBInstallBase to Checkin tests
(Stamatis Zampetakis, reviewed by Peter Vary)
Drop failsafe plugin since it is no longer needed.
Rename tests to better reflect their purpose (not integration tests
anymore) but more importantly to allow Jenkins splitTests step (
https://plugins.jenkins.io/parallel-test-executor/) to pick them up
automatically.
Extend test coverage to include mysql and mssql install/upgrade tests.
Closes #3399
---
Jenkinsfile | 11 +-----
standalone-metastore/DEV-README | 39 ++++++++--------------
standalone-metastore/metastore-server/pom.xml | 31 -----------------
.../hive/metastore/dbinstall/DbInstallBase.java | 3 ++
.../dbinstall/{ITestDerby.java => TestDerby.java} | 2 +-
.../dbinstall/{ITestMssql.java => TestMssql.java} | 2 +-
.../dbinstall/{ITestMysql.java => TestMysql.java} | 2 +-
.../{ITestOracle.java => TestOracle.java} | 2 +-
.../{ITestPostgres.java => TestPostgres.java} | 2 +-
9 files changed, 22 insertions(+), 72 deletions(-)
diff --git a/Jenkinsfile b/Jenkinsfile
index 79052c64e42..09b6cd377d8 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -93,7 +93,7 @@ OPTS+="
-Dorg.slf4j.simpleLogger.log.org.apache.maven.plugin.surefire.SurefirePl
OPTS+=" -Dmaven.repo.local=$PWD/.git/m2"
git config extra.mavenOpts "$OPTS"
OPTS=" $M_OPTS -Dmaven.test.failure.ignore "
-if [ -s inclusions.txt ]; then OPTS+="
-Dsurefire.includesFile=$PWD/inclusions.txt"; sed -i '/\\/ITest/d'
$PWD/inclusions.txt;fi
+if [ -s inclusions.txt ]; then OPTS+="
-Dsurefire.includesFile=$PWD/inclusions.txt";fi
if [ -s exclusions.txt ]; then OPTS+="
-Dsurefire.excludesFile=$PWD/exclusions.txt";fi
mvn $OPTS '''+args+'''
du -h --max-depth=1
@@ -279,15 +279,6 @@ reinit_metastore $dbType
time docker rm -f dev_$dbType || true
'''
}
- stage('verify') {
- try {
- sh """#!/bin/bash -e
-mvn verify -DskipITests=false -Dit.test=ITest${dbType.capitalize()}
-Dtest=nosuch -pl standalone-metastore/metastore-server
-Dmaven.test.failure.ignore -B
-"""
- } finally {
- junit '**/TEST-*.xml'
- }
- }
}
}
}
diff --git a/standalone-metastore/DEV-README b/standalone-metastore/DEV-README
index f0f5cacfbcf..731a09bfc06 100644
--- a/standalone-metastore/DEV-README
+++ b/standalone-metastore/DEV-README
@@ -10,7 +10,7 @@ checkin tests before loading a patch.
To run just the checkin tests:
'mvn test
-Dtest.groups=org.apache.hadoop.hive.metastore.annotation.MetastoreCheckinTest'
-To run all of the tests (exclusive of the databases tests, see below):
+To run all the tests:
'mvn test -Dtest.groups=""'. At the moment this takes around 25 minutes.
When adding a test, if you want it to run as part of the unit tests annotate it
@@ -23,47 +23,34 @@ quick test can be done for the unit tests and more in depth
testing as part
of the checkin tests.
--------------------------------------------------------------------------------
-There are integration tests for testing installation and upgrade of the
+There are checkin tests for testing installation and upgrade of the
metastore on Derby, MySQL (actually MariaDB is used), Oracle, Postgres, and
SQLServer.
-Each ITest runs two tests, one that installs the latest version of the
+For each DB type we runs two tests, one that installs the latest version of the
database and one that installs the latest version minus one and then upgrades
the database.
-To run the tests you will need to explicitly turn on integration testing by
-setting skipITests variable to false. The tests rely on Docker so the latter
-needs to be installed and configured properly (e.g., memory more than 3.5GB).
-
-Run all tests:
-
-mvn verify -DskipITests=false -Dtest=nosuch
+The tests (except Derby) rely on Docker so the latter needs to be installed
and configured
+properly (e.g., memory more than 3.5GB).
Run a single test:
-mvn verify -DskipITests=false -Dit.test=ITestMysql -Dtest=nosuch
+mvn test -Dtest.groups=MetastoreCheckinTest -Dtest=TestDerby
Supported databases for testing:
--Dit.test=ITestDerby
--Dit.test=ITestMysql
--Dit.test=ITestOracle
--Dit.test=ITestPostgres
--Dit.test=ITestMssql
+-Dtest=TestDerby
+-Dtest=TestMysql
+-Dtest=TestOracle
+-Dtest=TestPostgres
+-Dtest=TestMssql
By adding -Dverbose.schematool the Schema Tool output becomes more detailed.
-Logs for tests are located under
standalone-metastore/metastore-common/target/failsafe-reports
-
If you wish to use one of these containers to run your own tests against a
non-Derby version of the metastore, you can do that as well. You must specify
-that only the install test be run (change -Dit.test=ITestMysql in the example
-above to -Dit.test=ITestMysql#install) and tell it to leave the docker
container
+that only the install test be run (-Dtest=TestMysql#install) and tell it to
leave the docker container
running by adding -Dmetastore.itest.no.stop.container=true. You will then need
to stop and remove the container yourself once you have finished. The
container
is recreated for each run of the test, so you cannot rerun the test until you
have stopped and removed it. You can construct the connection values to put in
-metastore-site.xml from the information in the appropriate ITest file (e.g.,
-from ITestMysql you can find that the JDBC URL is
-"jdbc:mysql://localhost:3306/hivedb", the JDBC driver is
-"org.mariadb.jdbc.Driver", and the password is "hivepassword". The user is
-always "hiveuser".
-
+metastore-site.xml from the information in the DatabaseRule and its subclasses.
diff --git a/standalone-metastore/metastore-server/pom.xml
b/standalone-metastore/metastore-server/pom.xml
index 049caf5bf2e..d8a454e9545 100644
--- a/standalone-metastore/metastore-server/pom.xml
+++ b/standalone-metastore/metastore-server/pom.xml
@@ -546,37 +546,6 @@
</execution>
</executions>
</plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-failsafe-plugin</artifactId>
- <executions>
- <execution>
- <goals>
- <goal>integration-test</goal>
- <goal>verify</goal>
- </goals>
- </execution>
- </executions>
- <configuration>
- <redirectTestOutputToFile>true</redirectTestOutputToFile>
- <reuseForks>false</reuseForks>
- <argLine>-Xmx2048m</argLine>
- <failIfNoTests>false</failIfNoTests>
- <systemPropertyVariables>
- <log4j.debug>true</log4j.debug>
- <java.io.tmpdir>${test.tmp.dir}</java.io.tmpdir>
- <test.tmp.dir>${test.tmp.dir}</test.tmp.dir>
- <hive.in.test>true</hive.in.test>
- <derby.version>${derby.version}</derby.version>
-
<derby.stream.error.file>${test.tmp.dir}/derby.log</derby.stream.error.file>
- </systemPropertyVariables>
- <additionalClasspathElements>
-
<additionalClasspathElement>${log4j.conf.dir}</additionalClasspathElement>
- </additionalClasspathElements>
- <skipITs>${skipITests}</skipITs>
- <!-- set this to false to run these tests -->
- </configuration>
- </plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
diff --git
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/DbInstallBase.java
b/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/DbInstallBase.java
index 2ebba393bc4..939299d491a 100644
---
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/DbInstallBase.java
+++
b/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/DbInstallBase.java
@@ -18,10 +18,13 @@
package org.apache.hadoop.hive.metastore.dbinstall;
import org.apache.hadoop.hive.metastore.HiveMetaException;
+import org.apache.hadoop.hive.metastore.annotation.MetastoreCheckinTest;
import org.apache.hadoop.hive.metastore.dbinstall.rules.DatabaseRule;
import org.junit.Assert;
import org.junit.Test;
+import org.junit.experimental.categories.Category;
+@Category(MetastoreCheckinTest.class)
public abstract class DbInstallBase {
private static final String FIRST_VERSION = "1.2.0";
diff --git
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/ITestDerby.java
b/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/TestDerby.java
similarity index 96%
rename from
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/ITestDerby.java
rename to
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/TestDerby.java
index 2a0f7807e81..9ed3f1f3eb9 100644
---
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/ITestDerby.java
+++
b/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/TestDerby.java
@@ -24,7 +24,7 @@ import org.junit.Rule;
/**
* Mssql-specific DbInstallBase child test class.
*/
-public class ITestDerby extends DbInstallBase {
+public class TestDerby extends DbInstallBase {
@Rule
public final DatabaseRule databaseRule = new Derby(true);
diff --git
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/ITestMssql.java
b/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/TestMssql.java
similarity index 96%
rename from
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/ITestMssql.java
rename to
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/TestMssql.java
index e94d392b005..ab9d41aee0c 100644
---
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/ITestMssql.java
+++
b/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/TestMssql.java
@@ -24,7 +24,7 @@ import org.junit.Rule;
/**
* Mssql-specific DbInstallBase child test class.
*/
-public class ITestMssql extends DbInstallBase {
+public class TestMssql extends DbInstallBase {
@Rule
public final DatabaseRule databaseRule = new Mssql();
diff --git
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/ITestMysql.java
b/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/TestMysql.java
similarity index 96%
rename from
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/ITestMysql.java
rename to
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/TestMysql.java
index 1c364682525..26ed4d7494c 100644
---
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/ITestMysql.java
+++
b/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/TestMysql.java
@@ -24,7 +24,7 @@ import org.junit.Rule;
/**
* Mysql-specific DbInstallBase child test class.
*/
-public class ITestMysql extends DbInstallBase {
+public class TestMysql extends DbInstallBase {
@Rule
public final DatabaseRule databaseRule = new Mysql();
diff --git
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/ITestOracle.java
b/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/TestOracle.java
similarity index 96%
rename from
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/ITestOracle.java
rename to
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/TestOracle.java
index b2de064a760..b5777c9c0d3 100644
---
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/ITestOracle.java
+++
b/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/TestOracle.java
@@ -24,7 +24,7 @@ import org.junit.Rule;
/**
* Oracle-specific DbInstallBase child test class.
*/
-public class ITestOracle extends DbInstallBase {
+public class TestOracle extends DbInstallBase {
@Rule
public final DatabaseRule databaseRule = new Oracle();
diff --git
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/ITestPostgres.java
b/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/TestPostgres.java
similarity index 96%
rename from
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/ITestPostgres.java
rename to
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/TestPostgres.java
index 1e43d4f8ce0..558e227a0b7 100644
---
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/ITestPostgres.java
+++
b/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/TestPostgres.java
@@ -24,7 +24,7 @@ import org.junit.Rule;
/**
* Postgres-specific DbInstallBase child test class.
*/
-public class ITestPostgres extends DbInstallBase {
+public class TestPostgres extends DbInstallBase {
@Rule
public final DatabaseRule databaseRule = new Postgres();
