Hi everyone,I am using livy (livy-0.4.0-incubating-bin), and a remote spark
(spark-2.2.0-bin-hadoop2.7) stand alone cluster. livy's setting is default
except setting livy.spark.master = spark://10.128.1.1:6066 and
livy.spark.deploy-mode = cluster (have tried with default deploy-mode but then
it throws a different error)when i try to create an interactive spark session on
remote cluster it creates the driver and immediately sends it to error state.
livy console error 17/10/15 20:18:08 INFO LineBufferedStream: stdout: 17/10/15 20:18:08 INFO RestSubmissionClient: Submitting a request to launch an application in spark://10.128.1.1:6066.17/10/15 20:18:09 INFO LineBufferedStream: stdout: 17/10/15 20:18:09 INFO RestSubmissionClient: Submission successfully created as driver-20171015201808-0002. Polling submission state...17/10/15 20:18:09 INFO LineBufferedStream: stdout: 17/10/15 20:18:09 INFO RestSubmissionClient: Submitting a request for the status of submission driver-20171015201808-0002 in spark://10.128.1.1:6066.17/10/15 20:18:09 INFO LineBufferedStream: stdout: 17/10/15 20:18:09 INFO RestSubmissionClient: State of driver driver-20171015201808-0002 is now ERROR.17/10/15 20:18:09 INFO LineBufferedStream: stdout: 17/10/15 20:18:09 INFO RestSubmissionClient: Driver is running on worker worker-20171015195836-10.128.1.1-38097 at 10.128.1.1:38097.17/10/15 20:18:09 INFO LineBufferedStream: stdout: 17/10/15 20:18:09 ERROR RestSubmissionClient: Exception from the cluster:17/10/15 20:18:09 INFO LineBufferedStream: stdout: java.nio.file.NoSuchFileException: spark-internal17/10/15 20:18:09 INFO LineBufferedStream: stdout: sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)17/10/15 20:18:09 INFO LineBufferedStream: stdout: sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)17/10/15 20:18:09 INFO LineBufferedStream: stdout: sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)17/10/15 20:18:09 INFO LineBufferedStream: stdout: sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:526)17/10/15 20:18:09 INFO LineBufferedStream: stdout: sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:253)17/10/15 20:18:09 INFO LineBufferedStream: stdout: java.nio.file.Files.copy(Files.java:1274)17/10/15 20:18:09 INFO LineBufferedStream: stdout: org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$copyRecursive(Utils.scala:625)17/10/15 20:18:09 INFO LineBufferedStream: stdout: org.apache.spark.util.Utils$.copyFile(Utils.scala:596)17/10/15 20:18:09 INFO LineBufferedStream: stdout: org.apache.spark.util.Utils$.doFetchFile(Utils.scala:681)17/10/15 20:18:09 INFO LineBufferedStream: stdout: org.apache.spark.util.Utils$.fetchFile(Utils.scala:480)17/10/15 20:18:09 INFO LineBufferedStream: stdout: org.apache.spark.deploy.worker.DriverRunner.downloadUserJar(DriverRunner.scala:155)17/10/15 20:18:09 INFO LineBufferedStream: stdout: org.apache.spark.deploy.worker.DriverRunner.prepareAndRunDriver(DriverRunner.scala:173)17/10/15 20:18:09 INFO LineBufferedStream: stdout: org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:92)17/10/15 20:18:09 INFO LineBufferedStream: stdout: 17/10/15 20:18:09 INFO RestSubmissionClient: Server responded with CreateSubmissionResponse:17/10/15 20:18:09 INFO LineBufferedStream: stdout: {17/10/15 20:18:09 INFO LineBufferedStream: stdout: "action" : "CreateSubmissionResponse",17/10/15 20:18:09 INFO LineBufferedStream: stdout: "message" : "Driver successfully submitted as driver-20171015201808-0002",17/10/15 20:18:09 INFO LineBufferedStream: stdout: "serverSparkVersion" : "2.2.0",17/10/15 20:18:09 INFO LineBufferedStream: stdout: "submissionId" : "driver-20171015201808-0002",17/10/15 20:18:09 INFO LineBufferedStream: stdout: "success" : true17/10/15 20:18:09 INFO LineBufferedStream: stdout: }
Not using Mixmax yet?
can anyone please guide me how to resolve this?
I have tried setting up spark locally and it works fine. so i guess problem is
with uploading jar files to remote cluster. I am using GCE for both spark
cluster and livy server but they have all ports (on internal network open)
spark worker log show this too
17/10/15 20:18:08 INFO Worker: Asked to launch driver driver-20171015201808-000217/10/15 20:18:08 INFO DriverRunner: Copying user jar spark-internal to /home/junaid.nasir/spark-2.2.0-bin-hadoop2.7/work/driver-20171015201808-0002/spark-internal17/10/15 20:18:08 INFO Utils: Copying /spark-internal to /home/junaid.nasir/spark-2.2.0-bin-hadoop2.7/work/driver-20171015201808-0002/spark-internal17/10/15 20:18:08 INFO DriverRunner: Killing driver process!17/10/15 20:18:08 WARN Worker: Driver driver-20171015201808-0002 failed with unrecoverable exception: java.nio.file.NoSuchFileException: spark-internal
Not using Mixmax yet?
