[ https://issues.apache.org/jira/browse/SPARK-11798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15924420#comment-15924420 ]
Luca Menichetti commented on SPARK-11798: ----------------------------------------- I have exactly the same issue, I had to include all datanucleus-* jar manually with the --jars command when submitting job in cluster mode. With --master yarn-client everything is working, so apparently these jars are not automatically imported. Spark is compiled with Hive profile and version is 1.6.2. The logs I see are the following: WARN metastore.HiveMetaStore: Retrying creating default database after error: Class org.datanucleus.api.jdo.JDOPersistenceManagerFactory was not found. javax.jdo.JDOFatalUserException: Class org.datanucleus.api.jdo.JDOPersistenceManagerFactory was not found. > Datanucleus jars is missing under lib_managed/jars > -------------------------------------------------- > > Key: SPARK-11798 > URL: https://issues.apache.org/jira/browse/SPARK-11798 > Project: Spark > Issue Type: Bug > Components: Build, SQL > Reporter: Jeff Zhang > > I notice the comments in https://github.com/apache/spark/pull/9575 said that > Datanucleus related jars will still be copied to lib_managed/jars. But I > don't see any jars under lib_managed/jars. The weird thing is that I see the > jars on another machine, but could not see jars on my laptop even after I > delete the whole spark project and start from scratch. Does it related with > environments ? I try to add the following code in SparkBuild.scala to track > the issue, it shows that the jars is empty. > {code} > deployDatanucleusJars := { > val jars: Seq[File] = (fullClasspath in assembly).value.map(_.data) > .filter(_.getPath.contains("org.datanucleus")) > // this is what I added > println("*********************************************") > println("fullClasspath:"+fullClasspath) > println("assembly:"+assembly) > println("jars:"+jars.map(_.getAbsolutePath()).mkString(",")) > // > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org