[
https://issues.apache.org/jira/browse/HIVE-25594?focusedWorklogId=675885&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-675885
]
ASF GitHub Bot logged work on HIVE-25594:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 04/Nov/21 01:38
Start Date: 04/Nov/21 01:38
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #2742:
URL: https://github.com/apache/hive/pull/2742#discussion_r741839431
##########
File path:
itests/util/src/main/java/org/apache/hadoop/hive/ql/externalDB/AbstractExternalDB.java
##########
@@ -71,27 +59,8 @@ public ProcessResults(String stdout, String stderr, int rc) {
}
}
- public static AbstractExternalDB initalizeExternalDB(String
externalDBType) throws IOException {
- AbstractExternalDB abstractExternalDB;
- switch (externalDBType) {
- case "mysql":
- abstractExternalDB = new MySQLExternalDB();
- break;
- case "postgres":
- abstractExternalDB = new PostgresExternalDB();
- break;
- default:
- throw new IOException("unsupported external database type " +
externalDBType);
- }
- return abstractExternalDB;
- }
-
- public AbstractExternalDB(String externalDBType) {
- this.externalDBType = externalDBType;
- }
-
- protected String getDockerContainerName() {
- return String.format("qtestExternalDB-%s", externalDBType);
+ private final String getDockerContainerName() {
Review comment:
Logged https://issues.apache.org/jira/browse/HIVE-25667 for tracking
this further.
##########
File path:
itests/util/src/main/java/org/apache/hadoop/hive/ql/qoption/QTestDatabaseHandler.java
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.qoption;
+
+import org.apache.hadoop.hive.ql.QTestUtil;
+import org.apache.hadoop.hive.ql.externalDB.AbstractExternalDB;
+import org.apache.hadoop.hive.ql.externalDB.MSSQLServer;
+import org.apache.hadoop.hive.ql.externalDB.MariaDB;
+import org.apache.hadoop.hive.ql.externalDB.MySQLExternalDB;
+import org.apache.hadoop.hive.ql.externalDB.Oracle;
+import org.apache.hadoop.hive.ql.externalDB.PostgresExternalDB;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.EnumMap;
+import java.util.Map;
+
+/**
+ * An option handler for spinning (resp. stopping) databases before (resp.
after) running a test.
+ *
+ * Syntax: qt:database:DatabaseType:path_to_init_script
+ *
+ * The database type ({@link DatabaseType}) is obligatory argument. The
initialization script can be omitted.
+ *
+ * Current limitations:
+ * <ol>
+ * <li>Only one test/file per run</li>
+ * <li>Does not support parallel execution</li>
+ * <li>Cannot instantiate more than one database of the same type per
test</li>
+ * </ol>
+ */
+public class QTestDatabaseHandler implements QTestOptionHandler {
+ private static final Logger LOG =
LoggerFactory.getLogger(QTestDatabaseHandler.class);
+
+ private enum DatabaseType {
+ POSTGRES {
+ @Override
+ AbstractExternalDB create() {
+ return new PostgresExternalDB();
+ }
+ }, MYSQL {
+ @Override
+ AbstractExternalDB create() {
+ return new MySQLExternalDB();
+ }
+ }, MARIADB {
+ @Override
+ AbstractExternalDB create() {
+ return new MariaDB();
+ }
+ }, MSSQL {
+ @Override
+ AbstractExternalDB create() {
+ return new MSSQLServer();
+ }
+ }, ORACLE {
+ @Override
+ AbstractExternalDB create() {
+ return new Oracle();
+ }
+ };
+
+ abstract AbstractExternalDB create();
+ }
+
+ private final Map<DatabaseType, String> databaseToScript = new
EnumMap<>(DatabaseType.class);
+
+ @Override
+ public void processArguments(String arguments) {
+ String[] args = arguments.split(":");
+ if (args.length == 0) {
+ throw new IllegalArgumentException("No arguments provided");
+ }
+ if (args.length > 2) {
+ throw new IllegalArgumentException(
+ "Too many arguments. Expected {dbtype:script}. Found: " +
Arrays.toString(args));
+ }
+ DatabaseType dbType = DatabaseType.valueOf(args[0].toUpperCase());
+ String initScript = args.length == 2 ? args[1] : "";
+ if (databaseToScript.containsKey(dbType)) {
+ throw new IllegalArgumentException(dbType + " database is already
defined in this file.");
+ }
+ databaseToScript.put(dbType, initScript);
+ }
+
+ @Override
+ public void beforeTest(QTestUtil qt) throws Exception {
+ if (databaseToScript.isEmpty()) {
+ return;
+ }
+ for (Map.Entry<DatabaseType, String> dbEntry :
databaseToScript.entrySet()) {
+ String scriptsDir = QTestUtil.getScriptsDir(qt.getConf());
+ Path dbScript = Paths.get(scriptsDir, dbEntry.getValue());
+ AbstractExternalDB db = dbEntry.getKey().create();
Review comment:
This suggestion would be definitely very useful and necessary if we want
to support multiple databases of the same type running side by side or enabling
database reuse among qfiles. Leaving this for
https://issues.apache.org/jira/browse/HIVE-25668 where we are going to have
more concrete requirements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 675885)
Time Spent: 3h 40m (was: 3.5h)
> Setup JDBC databases in tests via QT options
> --------------------------------------------
>
> Key: HIVE-25594
> URL: https://issues.apache.org/jira/browse/HIVE-25594
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
> Reporter: Stamatis Zampetakis
> Assignee: Stamatis Zampetakis
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> The goal of this jira is to add a new QT option for setting up JDBC DBMS and
> using it in qtests which need a JDBC endpoint up and running. It can be used
> in tests with external JDBC tables, connectors, etc.
> A sample file using the proposed option ({{qt:database}}) is shown below.
> {code:sql}
> --!qt:database:postgres:init_sript_1234.sql:cleanup_script_1234.sql
> CREATE EXTERNAL TABLE country (name varchar(80))
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.database.type" = "POSTGRES",
> "hive.sql.jdbc.driver" = "org.postgresql.Driver",
> "hive.sql.jdbc.url" = "jdbc:postgresql://localhost:5432/qtestDB",
> "hive.sql.dbcp.username" = "qtestuser",
> "hive.sql.dbcp.password" = "qtestpassword",
> "hive.sql.table" = "country");
> EXPLAIN CBO SELECT COUNT(*) from country;
> SELECT COUNT(*) from country;
> {code}
> This builds upon HIVE-25423 but proposes to use JDBC datasources without the
> need for a using a specific CLI driver. Furthermore, the proposed QT option
> syntax allows using customised init/cleanup scripts for the JDBC datasource
> per test.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
