[
https://issues.apache.org/jira/browse/LIVY-636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16923973#comment-16923973
]
Konstantin commented on LIVY-636:
---------------------------------
Update: the crash in my comment was due to the fact that I've included Scala in
the JAR being added. Once I rebuilt it without Scala libraries, all worked
smoothly.
To understand this, I've checked through the code in the stacktrace. It appears
that if the exception is thrown inside the [{{"run"}} function of
RSCDriver|https://github.com/apache/incubator-livy/blob/master/rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java#L321],
including the initialization process (that was the case here, since
"scala.reflect" will misbehave when there are two Scala libraries on the
classpath). If the error is thrown during initialization, the "jc" variable
remains "null". Then, after the exception is thrown, the "finally" block runs,
calling for shutdown. If there were any queued job, driver will try to cancel
it. But, since there is no context yet for the jobs, job.cancel (called
[here|https://github.com/apache/incubator-livy/blob/master/rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java#L127])
throws the NPE, which interrupts the program, hiding the original exception.
> Unable to create interactive session with additional JAR in
> spark.driver.extraClassPath
> ---------------------------------------------------------------------------------------
>
> Key: LIVY-636
> URL: https://issues.apache.org/jira/browse/LIVY-636
> Project: Livy
> Issue Type: Bug
> Affects Versions: 0.6.0
> Reporter: Ishita Virmani
> Priority: Major
> Attachments: applicationmaster.log, container.log, stacktrace.txt,
> test.png
>
>
> Command Run: c{{url -H "Content-Type: application/json" -X POST -d
> '\{"kind":"pyspark","conf":{"spark.driver.extraClassPath":"/data/XXX-0.0.1-SNAPSHOT.jar"}}'
> -i http://<LIVY_SERVER_IP:PORT>/session}}
> {{The above command fails to create a Spark Session on YARN with Null pointer
> exception. Stack trace for the same has been attached along-with.}}
> The JAR file here is present on local driver Path. Also tried using HDFS path
> in the following manner
> {{hdfs://<NM_IP>:<NM_Port>/data/XXX-0.0.1-SNAPSHOT.jar}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
