shanyu zhao created LIVY-749:
--------------------------------

             Summary: Datanucleus jars are uploaded to hdfs unnecessarily when 
starting a livy session
                 Key: LIVY-749
                 URL: https://issues.apache.org/jira/browse/LIVY-749
             Project: Livy
          Issue Type: New Feature
          Components: Server
    Affects Versions: 0.7.0, 0.6.0
            Reporter: shanyu zhao


If we start any Livy session with hive support 
(livy.repl.enable-hive-context=true), we see that 3 datanucleus jars are 
uploaded to HDFS and downloaded to drivers/executors:

Uploading resource file:/opt/spark/jars/datanucleus-api-jdo-3.2.6.jar -> 
hdfs://namenode/user/test1/.sparkStaging/application_1581024490249_0002/datanucleus-api-jdo-3.2.6.jar
...

These 3 datanucleus jars are not needed bacause they are already included in 
Spark 2.x jars folder.

The reason is because in InteractiveSession.scala, method 
mergeHiveSiteAndHiveDeps(), we merged datanucleus jars to spark.jars list with 
method datanucleusJars(). We should remove datanucleusJars() function.






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to