Hi all,
I'm writing this mail to let you know that there's a bug
<https://github.com/docker/for-mac/issues/5527> in Docker Desktop for Mac
which could lead to failure of bringing up MySQL service when running
Druid's integration-test.
I experienced this bug last week and spent days to dig out what happened.
Fortunately, this bug has been fixed in the latest release of Docker
Desktop 3.3.1 released in 2021-04-15.
It looks like this bug is introduced in 3.3.0, *So if your Docker Desktop
is that version, an upgrade of Docker Desktop to the latest is highly
recommended.*
Below is some information when this bug comes out during integration-test.
1. From console, exception log of mysql startup could be observed:
*druid-metadata-storage | 2021-04-16 11:39:29,469 INFO exited: mysql
(exit status 0; not expected)*
* druid-metadata-storage | 2021-04-16 11:39:30,471 INFO gave up:
mysql entered FATAL state, too many start retries too quickly*
2. Inside the druid-metadata-storage container, an exception log is
generated by MySQL to /tmp directory saying that it fails to create
/dev/null directory because of denied permission.
3. Since MySQL fails to start, Druid Coordinator prints lots of exception
message related to MySQL, and due to that, Coordinator exits unexpectedly
soon.
The last packet sent successfully to the server was 0 milliseconds ago. The
driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) ~[?:1.8.0_282]
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
~[?:1.8.0_282]
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
~[?:1.8.0_282]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
~[?:1.8.0_282]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
~[mysql-connector-java.jar:5.1.48]
at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:990)
~[mysql-connector-java.jar:5.1.48]
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:335)
~[mysql-connector-java.jar:5.1.48]
at
com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2187)
~[mysql-connector-java.jar:5.1.48]
at
com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2220)
~[mysql-connector-java.jar:5.1.48]
at
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2015)
~[mysql-connector-java.jar:5.1.48]
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:768)
~[mysql-connector-java.jar:5.1.48]
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
~[mysql-connector-java.jar:5.1.48]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) ~[?:1.8.0_282]
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
~[?:1.8.0_282]
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
~[?:1.8.0_282]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
~[?:1.8.0_282]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
~[mysql-connector-java.jar:5.1.48]
at
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:385)
~[mysql-connector-java.jar:5.1.48]
at
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:323)
~[mysql-connector-java.jar:5.1.48]
at
org.apache.druid.metadata.BasicDataSourceExt.lambda$createConnectionFactory$0(BasicDataSourceExt.java:185)
~[druid-server-0.22.0-SNAPSHOT.jar:0.22.0-SNAPSHOT]
at
org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:205)
~[commons-dbcp2-2.0.1.jar:2.0.1]
at
org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:2162)
~[commons-dbcp2-2.0.1.jar:2.0.1]
at
org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2148)
~[commons-dbcp2-2.0.1.jar:2.0.1]
at
org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:1903)
~[commons-dbcp2-2.0.1.jar:2.0.1]
at
org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1413)
~[commons-dbcp2-2.0.1.jar:2.0.1]
at
org.skife.jdbi.v2.DataSourceConnectionFactory.openConnection(DataSourceConnectionFactory.java:36)
~[jdbi-2.63.1.jar:2.63.1]
at org.skife.jdbi.v2.DBI.open(DBI.java:212)
~[jdbi-2.63.1.jar:2.63.1]
at org.skife.jdbi.v2.DBI.withHandle(DBI.java:279)
~[jdbi-2.63.1.jar:2.63.1]
at
org.apache.druid.metadata.SQLMetadataConnector.lookup(SQLMetadataConnector.java:600)
~[druid-server-0.22.0-SNAPSHOT.jar:0.22.0-SNAPSHOT]
at
org.apache.druid.security.basic.authentication.db.updater.CoordinatorBasicAuthenticatorMetadataStorageUpdater.getCurrentUserMapBytes(CoordinatorBasicAuthenticatorMetadataStorageUpdater.java:271)
~[druid-basic-security-0.22.0-SNAPSHOT.jar:0.22.0-SNAPSHOT]
at
org.apache.druid.security.basic.authentication.db.updater.CoordinatorBasicAuthenticatorMetadataStorageUpdater.lambda$start$0(CoordinatorBasicAuthenticatorMetadataStorageUpdater.java:131)
~[druid-basic-security-0.22.0-SNAPSHOT.jar:0.22.0-SNAPSHOT]
at
org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87)
~[druid-core-0.22.0-SNAPSHOT.jar:0.22.0-SNAPSHOT]
at
org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115)
~[druid-core-0.22.0-SNAPSHOT.jar:0.22.0-SNAPSHOT]
at
org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105)
~[druid-core-0.22.0-SNAPSHOT.jar:0.22.0-SNAPSHOT]
at
org.apache.druid.security.basic.BasicAuthUtils.maybeInitialize(BasicAuthUtils.java:289)
~[druid-basic-security-0.22.0-SNAPSHOT.jar:0.22.0-SNAPSHOT]
at
org.apache.druid.security.basic.authentication.db.updater.CoordinatorBasicAuthenticatorMetadataStorageUpdater.start(CoordinatorBasicAuthenticatorMetadataStorageUpdater.java:122)
~[druid-basic-security-0.22.0-SNAPSHOT.jar:0.22.0-SNAPSHOT]
... 10 more
Caused by: java.net.ConnectException: Connection refused (Connection
refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
~[?:1.8.0_282]
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
~[?:1.8.0_282]
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
~[?:1.8.0_282]
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
~[?:1.8.0_282]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
~[?:1.8.0_282]
at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_282]
at
com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:211)
~[mysql-connector-java.jar:5.1.48]
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:299)
~[mysql-connector-java.jar:5.1.48]
at
com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2187)
~[mysql-connector-java.jar:5.1.48]
at
com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2220)
~[mysql-connector-java.jar:5.1.48]
at
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2015)
~[mysql-connector-java.jar:5.1.48]
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:768)
~[mysql-connector-java.jar:5.1.48]
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
~[mysql-connector-java.jar:5.1.48]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) ~[?:1.8.0_282]
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
~[?:1.8.0_282]
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
~[?:1.8.0_282]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
~[?:1.8.0_282]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
~[mysql-connector-java.jar:5.1.48]
at
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:385)
~[mysql-connector-java.jar:5.1.48]
at
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:323)
~[mysql-connector-java.jar:5.1.48]
at
org.apache.druid.metadata.BasicDataSourceExt.lambda$createConnectionFactory$0(BasicDataSourceExt.java:185)
~[druid-server-0.22.0-SNAPSHOT.jar:0.22.0-SNAPSHOT]
at
org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:205)
~[commons-dbcp2-2.0.1.jar:2.0.1]
at
org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:2162)
~[commons-dbcp2-2.0.1.jar:2.0.1]
at
org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2148)
~[commons-dbcp2-2.0.1.jar:2.0.1]
at
org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:1903)
~[commons-dbcp2-2.0.1.jar:2.0.1]
at
org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1413)
~[commons-dbcp2-2.0.1.jar:2.0.1]
at
org.skife.jdbi.v2.DataSourceConnectionFactory.openConnection(DataSourceConnectionFactory.java:36)
~[jdbi-2.63.1.jar:2.63.1]
at org.skife.jdbi.v2.DBI.open(DBI.java:212)
~[jdbi-2.63.1.jar:2.63.1]
at org.skife.jdbi.v2.DBI.withHandle(DBI.java:279)
~[jdbi-2.63.1.jar:2.63.1]
at
org.apache.druid.metadata.SQLMetadataConnector.lookup(SQLMetadataConnector.java:600)
~[druid-server-0.22.0-SNAPSHOT.jar:0.22.0-SNAPSHOT]
at
org.apache.druid.security.basic.authentication.db.updater.CoordinatorBasicAuthenticatorMetadataStorageUpdater.getCurrentUserMapBytes(CoordinatorBasicAuthenticatorMetadataStorageUpdater.java:271)
~[druid-basic-security-0.22.0-SNAPSHOT.jar:0.22.0-SNAPSHOT]
at
org.apache.druid.security.basic.authentication.db.updater.CoordinatorBasicAuthenticatorMetadataStorageUpdater.lambda$start$0(CoordinatorBasicAuthenticatorMetadataStorageUpdater.java:131)
~[druid-basic-security-0.22.0-SNAPSHOT.jar:0.22.0-SNAPSHOT]
at
org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87)
~[druid-core-0.22.0-SNAPSHOT.jar:0.22.0-SNAPSHOT]
at
org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115)
~[druid-core-0.22.0-SNAPSHOT.jar:0.22.0-SNAPSHOT]
at
org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105)
~[druid-core-0.22.0-SNAPSHOT.jar:0.22.0-SNAPSHOT]
at
org.apache.druid.security.basic.BasicAuthUtils.maybeInitialize(BasicAuthUtils.java:289)
~[druid-basic-security-0.22.0-SNAPSHOT.jar:0.22.0-SNAPSHOT]
at
org.apache.druid.security.basic.authentication.db.updater.CoordinatorBasicAuthenticatorMetadataStorageUpdater.start(CoordinatorBasicAuthenticatorMetadataStorageUpdater.java:122)
~[druid-basic-security-0.22.0-SNAPSHOT.jar:0.22.0-SNAPSHOT]
... 10 more
