[
https://issues.apache.org/jira/browse/SPARK-12426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15083482#comment-15083482
]
Mark Grover commented on SPARK-12426:
-------------------------------------
Sean and Josh,
I got to the bottom of this. This is because docker sucks when bubbling up the
error that docker engine is not running on the machine running the unit tests.
The instructions for installing docker engine on various OSs are at
https://docs.docker.com/engine/installation/
Once installed the docker service needs to be started, if it's not already
running. On Linux, this is simply {{sudo service docker start}} and then our
docker integration tests pass.
Sorry that I didn't get a chance to look into it around 1.6 rc time, holidays
got in the way.
I am thinking of adding this info on [this wiki
page|https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-IntelliJ].
Please let me know if you think there is a better place, that's the best I
could find. I don't seem to have access to edit that page, can one of you
please give me access?
Also, I was trying to search in the code for any puppet recipes we maintain for
the setting up build slaves. In order, if our Jenkins infra were wiped out, how
do we make sure docker-engine is installed and running? How do we maintain keep
track of build dependencies? Thanks in advance!
> Docker JDBC integration tests are failing again
> -----------------------------------------------
>
> Key: SPARK-12426
> URL: https://issues.apache.org/jira/browse/SPARK-12426
> Project: Spark
> Issue Type: Bug
> Components: SQL, Tests
> Affects Versions: 1.6.0
> Reporter: Mark Grover
>
> The Docker JDBC integration tests were fixed in SPARK-11796 but they seem to
> be failing again on my machine (Ubuntu Precise). This was the same box that I
> tested my previous commit on. Also, I am not confident this failure has much
> to do with Spark, since a well known commit where the tests were passing,
> fails now, in the same environment.
> [~sowen] mentioned on the Spark 1.6 voting thread that the tests were failing
> on his Ubuntu 15 box as well.
> Here's the error, fyi:
> {code}
> 15/12/18 10:12:50 INFO SparkContext: Successfully stopped SparkContext
> 15/12/18 10:12:50 INFO RemoteActorRefProvider$RemotingTerminator: Shutting
> down remote daemon.
> 15/12/18 10:12:50 INFO RemoteActorRefProvider$RemotingTerminator: Remote
> daemon shut down; proceeding with flushing remote transports.
> *** RUN ABORTED ***
> com.spotify.docker.client.DockerException:
> java.util.concurrent.ExecutionException:
> com.spotify.docker.client.shaded.javax.ws.rs.ProcessingException:
> java.io.IOException: No such file or directory
> at
> com.spotify.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:1141)
> at
> com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:1082)
> at
> com.spotify.docker.client.DefaultDockerClient.ping(DefaultDockerClient.java:281)
> at
> org.apache.spark.sql.jdbc.DockerJDBCIntegrationSuite.beforeAll(DockerJDBCIntegrationSuite.scala:76)
> at
> org.scalatest.BeforeAndAfterAll$class.beforeAll(BeforeAndAfterAll.scala:187)
> at
> org.apache.spark.sql.jdbc.DockerJDBCIntegrationSuite.beforeAll(DockerJDBCIntegrationSuite.scala:58)
> at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:253)
> at
> org.apache.spark.sql.jdbc.DockerJDBCIntegrationSuite.run(DockerJDBCIntegrationSuite.scala:58)
> at org.scalatest.Suite$class.callExecuteOnSuite$1(Suite.scala:1492)
> at org.scalatest.Suite$$anonfun$runNestedSuites$1.apply(Suite.scala:1528)
> ...
> Cause: java.util.concurrent.ExecutionException:
> com.spotify.docker.client.shaded.javax.ws.rs.ProcessingException:
> java.io.IOException: No such file or directory
> at
> jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> at
> jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> at
> jersey.repackaged.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> at
> com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:1080)
> at
> com.spotify.docker.client.DefaultDockerClient.ping(DefaultDockerClient.java:281)
> at
> org.apache.spark.sql.jdbc.DockerJDBCIntegrationSuite.beforeAll(DockerJDBCIntegrationSuite.scala:76)
> at
> org.scalatest.BeforeAndAfterAll$class.beforeAll(BeforeAndAfterAll.scala:187)
> at
> org.apache.spark.sql.jdbc.DockerJDBCIntegrationSuite.beforeAll(DockerJDBCIntegrationSuite.scala:58)
> at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:253)
> at
> org.apache.spark.sql.jdbc.DockerJDBCIntegrationSuite.run(DockerJDBCIntegrationSuite.scala:58)
> ...
> Cause: com.spotify.docker.client.shaded.javax.ws.rs.ProcessingException:
> java.io.IOException: No such file or directory
> at
> org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:481)
> at
> org.glassfish.jersey.apache.connector.ApacheConnector$1.run(ApacheConnector.java:491)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> jersey.repackaged.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
> at
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
> at
> jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:50)
> at
> jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:37)
> at
> org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:487)
> 15/12/18 10:12:50 INFO RemoteActorRefProvider$RemotingTerminator: Remoting
> shut down.
> at org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:177)
> ...
> Cause: java.io.IOException: No such file or directory
> at jnr.unixsocket.UnixSocketChannel.doConnect(UnixSocketChannel.java:94)
> at jnr.unixsocket.UnixSocketChannel.connect(UnixSocketChannel.java:102)
> at
> com.spotify.docker.client.ApacheUnixSocket.connect(ApacheUnixSocket.java:73)
> at
> com.spotify.docker.client.UnixConnectionSocketFactory.connectSocket(UnixConnectionSocketFactory.java:74)
> at
> org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
> at
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
> at
> org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
> at
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
> at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
> ...
> 15/12/18 10:12:50 INFO ShutdownHookManager: Shutdown hook called
> 15/12/18 10:12:50 INFO ShutdownHookManager: Deleting directory
> /root/spark/docker-integration-tests/target/tmp/spark-43a76284-b0c5-4570-9576-d65b21f45058
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 26.787 s
> [INFO] Finished at: 2015-12-18T10:12:50-08:00
> [INFO] Final Memory: 57M/579M
> [INFO]
> ------------------------------------------------------------------------
> [ERROR] Failed to execute goal org.scalatest:scalatest-maven-plugin:1.0:test
> (test) on project spark-docker-integration-tests_2.10: There are test
> failures -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> {code}
> And, here's the way to reproduce:
> {code}
> build/mvn -pl docker-integration-tests package
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
