shanyu zhao created LIVY-749:
--------------------------------
Summary: Datanucleus jars are uploaded to hdfs unnecessarily when
starting a livy session
Key: LIVY-749
URL: https://issues.apache.org/jira/browse/LIVY-749
Project: Livy
Issue Type: New Feature
Components: Server
Affects Versions: 0.7.0, 0.6.0
Reporter: shanyu zhao
If we start any Livy session with hive support
(livy.repl.enable-hive-context=true), we see that 3 datanucleus jars are
uploaded to HDFS and downloaded to drivers/executors:
Uploading resource file:/opt/spark/jars/datanucleus-api-jdo-3.2.6.jar ->
hdfs://namenode/user/test1/.sparkStaging/application_1581024490249_0002/datanucleus-api-jdo-3.2.6.jar
...
These 3 datanucleus jars are not needed bacause they are already included in
Spark 2.x jars folder.
The reason is because in InteractiveSession.scala, method
mergeHiveSiteAndHiveDeps(), we merged datanucleus jars to spark.jars list with
method datanucleusJars(). We should remove datanucleusJars() function.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
