kazuhirokomoda opened a new issue, #14936: URL: https://github.com/apache/druid/issues/14936
### Affected Version - Druid: 0.22.1 - MariaDB: (On-Prem) MariaDB-Galera 10.5.12 cluster - Java: OpenJDK 11.0.12 ### Description Trying to use MariaDB as metadata storage, but facing MySQLTransientException when trying to [run `metadata-init` tool](https://druid.apache.org/docs/0.22.1/operations/metadata-migration.html#create-druid-tables), during `druid_pendingSegments` table creation. The issue does not happen when I try the same steps with MariaDB Docker container on a server. Thank you for your support in advance. #### Steps to reproduce 1. Download the [0.22.1 release](https://www.apache.org/dyn/closer.cgi?path=/druid/0.22.1/apache-druid-0.22.1-bin.tar.gz). 2. In your terminal, extract Druid and change directories to the distribution directory (as shown in [official doc](https://druid.apache.org/docs/0.22.1/tutorials/index.html#step-1-install-druid)) 3. Download [mariadb-java-client-2.7.3.jar](https://repo1.maven.org/maven2/org/mariadb/jdbc/mariadb-java-client/2.7.3/mariadb-java-client-2.7.3.jar) and move it under `./extensions/mysql-metadata-storage` 4. (Prepare MariaDB cluster) 5. Run this command, following [official doc](https://druid.apache.org/docs/0.22.1/operations/metadata-migration.html#create-druid-tables), note that `-Ddruid.metadata.mysql.driver.driverClassName=org.mariadb.jdbc.Driver` is added ```shell java -classpath "lib/*" -Dlog4j.configurationFile=conf/druid/cluster/_common/log4j2.xml -Ddruid.extensions.directory="extensions" -Ddruid.extensions.loadList=[\"mysql-metadata-storage\"] -Ddruid.metadata.storage.type=mysql -Ddruid.metadata.mysql.driver.driverClassName=org.mariadb.jdbc.Driver org.apache.druid.cli.Main tools metadata-init --connectURI="jdbc:mysql://<host>:<port>/<database>" --user <user> --password <password> --base druid ``` 6. Facing the following MySQLTransientException exception <details> <summary><i>log output including MySQLTransientException exception</i></summary> ```shell WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/home/druid/apache-druid-0.22.1/lib/guice-4.1.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1 WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 2023-08-30T04:28:34,518 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.2.5.Final 2023-08-30T04:28:34,794 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [mysql-metadata-storage], jars: mysql-metadata-storage-0.22.1.jar, mariadb-java-client-2.7.3.jar 2023-08-30T04:28:34,892 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [mysql-metadata-storage], jars: mysql-metadata-storage-0.22.1.jar, mariadb-java-client-2.7.3.jar 2023-08-30T04:28:35,373 INFO [main] org.apache.druid.server.emitter.EmitterModule - Using emitter [NoopEmitter{}] for metrics and alerts, with dimensions [{version=0.22.1}]. 2023-08-30T04:28:35,375 INFO [main] org.apache.druid.server.metrics.MetricsModule - Loaded 2 monitors: org.apache.druid.query.ExecutorServiceMonitor, org.apache.druid.server.initialization.jetty.JettyServerModule$JettyMonitor 2023-08-30T04:28:35,382 INFO [main] org.apache.druid.metadata.storage.mysql.MySQLConnector - Loading "MySQL" metadata connector driver org.mariadb.jdbc.Driver 2023-08-30T04:28:35,428 INFO [main] org.apache.druid.metadata.storage.mysql.MySQLConnector - Configured MySQL as metadata storage 2023-08-30T04:28:35,519 INFO [main] org.apache.druid.metadata.SQLMetadataConnector - Table[druid_dataSource] already exists 2023-08-30T04:28:35,525 INFO [main] org.apache.druid.metadata.SQLMetadataConnector - Creating table[druid_pendingSegments] Exception in thread "main" java.lang.NoClassDefFoundError: com/mysql/jdbc/exceptions/MySQLTransientException at org.apache.druid.metadata.storage.mysql.MySQLConnector.connectorIsTransientException(MySQLConnector.java:208) at org.apache.druid.metadata.SQLMetadataConnector.isTransientException(SQLMetadataConnector.java:169) at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:135) at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:138) at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:148) at org.apache.druid.metadata.SQLMetadataConnector.createTable(SQLMetadataConnector.java:182) at org.apache.druid.metadata.SQLMetadataConnector.createPendingSegmentsTable(SQLMetadataConnector.java:210) at org.apache.druid.metadata.SQLMetadataConnector.createPendingSegmentsTable(SQLMetadataConnector.java:544) at org.apache.druid.cli.CreateTables.run(CreateTables.java:121) at org.apache.druid.cli.Main.main(Main.java:113) Caused by: java.lang.ClassNotFoundException: com.mysql.jdbc.exceptions.MySQLTransientException at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ... 13 more ``` </details> #### Using MariaDB Docker container instead of On-Prem cluster <details> <summary><i>This produces no issue.</i></summary> ```shell docker pull mariadb:10.5.12 docker run --name mariadbtest -e MYSQL_ROOT_PASSWORD=mypass -p 3306:3306 -d mariadb:10.5.12 docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' mariadbtest docker run --rm -ti mariadb:10.5 mysql -u root -p -h 240.0.0.2 # https://druid.apache.org/docs/0.22.1/development/extensions-core/mysql.html#setting-up-mysql ----- CREATE DATABASE druid102 DEFAULT CHARACTER SET utf8mb4; CREATE USER 'druid102'@'%' IDENTIFIED BY 'diurd'; GRANT ALL PRIVILEGES ON druid102.* TO 'druid102'@'%'; ----- java -classpath "lib/*" -Dlog4j.configurationFile=conf/druid/cluster/_common/log4j2.xml -Ddruid.extensions.directory="extensions" -Ddruid.extensions.loadList=[\"mysql-metadata-storage\"] -Ddruid.metadata.storage.type=mysql -Ddruid.metadata.mysql.driver.driverClassName=org.mariadb.jdbc.Driver org.apache.druid.cli.Main tools metadata-init --connectURI="jdbc:mysql://240.0.0.2:3306/druid102" --user druid102 --password diurd --base druid docker stop mariadbtest docker rm -v mariadbtest ``` </details> #### Alternative attempts and results - Put `mysql-connector-java-5.1.48.jar` instead of `mariadb-java-client-2.7.3.jar` - `java.sql.SQLException: Unknown system variable 'transaction_isolation'` - Change to `-Ddruid.metadata.storage.type=mariadb` - `Unknown provider[mariadb] of Key[type=org.apache.druid.metadata.MetadataStorageConnector, annotation=[none]], known options[[mysql, derby]]` - Change `jdbc:mysql` to `jdbc:mariadb` - The same `java.lang.ClassNotFoundException: com.mysql.jdbc.exceptions.MySQLTransientException` - Put both `mysql-connector-java-5.1.48.jar` and `mariadb-java-client-2.7.3.jar` under `./extensions/mysql-metadata-storage` - `java.io.EOFException: unexpected end of stream, read 0 bytes from 4 (socket was closed by server)` <details> <summary><i>log output including EOFException exception</i></summary> ```shell WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/home/druid/apache-druid-0.22.1/lib/guice-4.1.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1 WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 2023-08-30T05:15:27,521 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.2.5.Final 2023-08-30T05:15:27,806 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [mysql-metadata-storage], jars: mysql-metadata-storage-0.22.1.jar, mysql-connector-java-5.1.48.jar, mariadb-java-client-2.7.3.jar 2023-08-30T05:15:27,914 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [mysql-metadata-storage], jars: mysql-metadata-storage-0.22.1.jar, mysql-connector-java-5.1.48.jar, mariadb-java-client-2.7.3.jar 2023-08-30T05:15:28,393 INFO [main] org.apache.druid.server.emitter.EmitterModule - Using emitter [NoopEmitter{}] for metrics and alerts, with dimensions [{version=0.22.1}]. 2023-08-30T05:15:28,394 INFO [main] org.apache.druid.server.metrics.MetricsModule - Loaded 2 monitors: org.apache.druid.query.ExecutorServiceMonitor, org.apache.druid.server.initialization.jetty.JettyServerModule$JettyMonitor 2023-08-30T05:15:28,402 INFO [main] org.apache.druid.metadata.storage.mysql.MySQLConnector - Loading "MySQL" metadata connector driver org.mariadb.jdbc.Driver 2023-08-30T05:15:28,451 INFO [main] org.apache.druid.metadata.storage.mysql.MySQLConnector - Configured MySQL as metadata storage 2023-08-30T05:15:28,546 INFO [main] org.apache.druid.metadata.SQLMetadataConnector - Table[druid_dataSource] already exists 2023-08-30T05:15:28,553 INFO [main] org.apache.druid.metadata.SQLMetadataConnector - Creating table[druid_pendingSegments] 2023-08-30T05:15:28,562 WARN [main] org.apache.druid.java.util.common.RetryUtils - Retrying (1 of 9) in 1,136ms. org.skife.jdbi.v2.exceptions.CallbackFailedException: org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: java.sql.BatchUpdateException: (conn=1021355) unexpected end of stream, read 0 bytes from 4 (socket was closed by server) [statement:"null", located:"null", rewritten:"null", arguments:null] at org.skife.jdbi.v2.DBI.withHandle(DBI.java:284) ~[jdbi-2.63.1.jar:2.63.1] at org.apache.druid.metadata.SQLMetadataConnector.lambda$retryWithHandle$0(SQLMetadataConnector.java:138) ~[druid-server-0.22.1.jar:0.22.1] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) [druid-core-0.22.1.jar:0.22.1] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) [druid-core-0.22.1.jar:0.22.1] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) [druid-core-0.22.1.jar:0.22.1] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) [druid-core-0.22.1.jar:0.22.1] at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:138) [druid-server-0.22.1.jar:0.22.1] at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:148) [druid-server-0.22.1.jar:0.22.1] at org.apache.druid.metadata.SQLMetadataConnector.createTable(SQLMetadataConnector.java:182) [druid-server-0.22.1.jar:0.22.1] at org.apache.druid.metadata.SQLMetadataConnector.createPendingSegmentsTable(SQLMetadataConnector.java:210) [druid-server-0.22.1.jar:0.22.1] at org.apache.druid.metadata.SQLMetadataConnector.createPendingSegmentsTable(SQLMetadataConnector.java:544) [druid-server-0.22.1.jar:0.22.1] at org.apache.druid.cli.CreateTables.run(CreateTables.java:121) [druid-services-0.22.1.jar:0.22.1] at org.apache.druid.cli.Main.main(Main.java:113) [druid-services-0.22.1.jar:0.22.1] Caused by: org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: java.sql.BatchUpdateException: (conn=1021355) unexpected end of stream, read 0 bytes from 4 (socket was closed by server) [statement:"null", located:"null", rewritten:"null", arguments:null] at org.skife.jdbi.v2.Batch.execute(Batch.java:131) ~[jdbi-2.63.1.jar:2.63.1] at org.apache.druid.metadata.SQLMetadataConnector$1.withHandle(SQLMetadataConnector.java:194) ~[druid-server-0.22.1.jar:0.22.1] at org.apache.druid.metadata.SQLMetadataConnector$1.withHandle(SQLMetadataConnector.java:184) ~[druid-server-0.22.1.jar:0.22.1] at org.skife.jdbi.v2.DBI.withHandle(DBI.java:281) ~[jdbi-2.63.1.jar:2.63.1] ... 12 more Caused by: java.sql.BatchUpdateException: (conn=1021355) unexpected end of stream, read 0 bytes from 4 (socket was closed by server) at org.mariadb.jdbc.MariaDbStatement.executeBatchExceptionEpilogue(MariaDbStatement.java:324) ~[mariadb-java-client-2.7.3.jar:?] at org.mariadb.jdbc.MariaDbStatement.executeBatch(MariaDbStatement.java:1398) ~[mariadb-java-client-2.7.3.jar:?] at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:345) ~[commons-dbcp2-2.0.1.jar:2.0.1] at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:345) ~[commons-dbcp2-2.0.1.jar:2.0.1] at org.skife.jdbi.v2.Batch.execute(Batch.java:121) ~[jdbi-2.63.1.jar:2.63.1] at org.apache.druid.metadata.SQLMetadataConnector$1.withHandle(SQLMetadataConnector.java:194) ~[druid-server-0.22.1.jar:0.22.1] at org.apache.druid.metadata.SQLMetadataConnector$1.withHandle(SQLMetadataConnector.java:184) ~[druid-server-0.22.1.jar:0.22.1] at org.skife.jdbi.v2.DBI.withHandle(DBI.java:281) ~[jdbi-2.63.1.jar:2.63.1] ... 12 more Caused by: java.sql.SQLNonTransientConnectionException: (conn=1021355) unexpected end of stream, read 0 bytes from 4 (socket was closed by server) at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:73) ~[mariadb-java-client-2.7.3.jar:?] at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:155) ~[mariadb-java-client-2.7.3.jar:?] at org.mariadb.jdbc.MariaDbStatement.executeBatchExceptionEpilogue(MariaDbStatement.java:320) ~[mariadb-java-client-2.7.3.jar:?] at org.mariadb.jdbc.MariaDbStatement.executeBatch(MariaDbStatement.java:1398) ~[mariadb-java-client-2.7.3.jar:?] at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:345) ~[commons-dbcp2-2.0.1.jar:2.0.1] at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:345) ~[commons-dbcp2-2.0.1.jar:2.0.1] at org.skife.jdbi.v2.Batch.execute(Batch.java:121) ~[jdbi-2.63.1.jar:2.63.1] at org.apache.druid.metadata.SQLMetadataConnector$1.withHandle(SQLMetadataConnector.java:194) ~[druid-server-0.22.1.jar:0.22.1] at org.apache.druid.metadata.SQLMetadataConnector$1.withHandle(SQLMetadataConnector.java:184) ~[druid-server-0.22.1.jar:0.22.1] at org.skife.jdbi.v2.DBI.withHandle(DBI.java:281) ~[jdbi-2.63.1.jar:2.63.1] ... 12 more Caused by: java.sql.SQLNonTransientConnectionException: unexpected end of stream, read 0 bytes from 4 (socket was closed by server) at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.handleIoException(AbstractQueryProtocol.java:2089) ~[mariadb-java-client-2.7.3.jar:?] at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:1540) ~[mariadb-java-client-2.7.3.jar:?] at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1519) ~[mariadb-java-client-2.7.3.jar:?] at org.mariadb.jdbc.internal.protocol.AsyncMultiRead.call(AsyncMultiRead.java:132) ~[mariadb-java-client-2.7.3.jar:?] at org.mariadb.jdbc.internal.protocol.AsyncMultiRead.call(AsyncMultiRead.java:67) ~[mariadb-java-client-2.7.3.jar:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:829) ~[?:?] Caused by: java.io.EOFException: unexpected end of stream, read 0 bytes from 4 (socket was closed by server) at org.mariadb.jdbc.internal.io.input.StandardPacketInputStream.getPacketArray(StandardPacketInputStream.java:249) ~[mariadb-java-client-2.7.3.jar:?] at org.mariadb.jdbc.internal.io.input.StandardPacketInputStream.getPacket(StandardPacketInputStream.java:218) ~[mariadb-java-client-2.7.3.jar:?] at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:1538) ~[mariadb-java-client-2.7.3.jar:?] at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1519) ~[mariadb-java-client-2.7.3.jar:?] at org.mariadb.jdbc.internal.protocol.AsyncMultiRead.call(AsyncMultiRead.java:132) ~[mariadb-java-client-2.7.3.jar:?] at org.mariadb.jdbc.internal.protocol.AsyncMultiRead.call(AsyncMultiRead.java:67) ~[mariadb-java-client-2.7.3.jar:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:829) ~[?:?] ``` </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
