[ https://issues.apache.org/jira/browse/TOREE-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15855019#comment-15855019 ]
Jakob Odersky edited comment on TOREE-375 at 2/7/17 12:16 AM: -------------------------------------------------------------- -Yrepl-class-based strikes again. I managed to track this down to the [refreshDefinitions()|https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala-2.11/org/apache/toree/kernel/interpreter/scala/ScalaInterpreterSpecific.scala#L79-L97] function that is called when jars are added dynamically. It appears that the `valueOfTerm` method does not find a value associated to any variables. The following snippet illustrates that this behaviour only happens when the `-Yrepl-class-based` option is set in the repl (the option is required for Spark to correctly serialize objects). {code} import scala.tools.nsc.Settings import scala.tools.nsc.interpreter._ object Main extends App { val settings = new Settings settings.usejavacp.value = true //settings.Yreplclassbased.value = true val iMain: IMain = new IMain(settings) iMain.initializeSynchronous() iMain.interpret("val x = 1") iMain.definedTerms.foreach { name => println("defined term: " + name.toString) iMain.valueOfTerm(name.toString) match { case Some(value) => println("value: " + value) case None => println("no value") } } } {code} Above codes yields: {code} [info] Running foo.Main [info] x: Int = 1 [info] defined term: x [info] value: 1 {code} When setting -Y-repl-class-based by uncommenting above comment: {code} [info] Running foo.Main [info] x: Int = 1 [info] defined term: x [info] no value {code} was (Author: jodersky): -Yrepl-class-based strikes again. I managed to track this down to the [refreshDefinitions()|https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala-2.11/org/apache/toree/kernel/interpreter/scala/ScalaInterpreterSpecific.scala#L79-L97] function that is called when jars are added dynamically. It appears that the `valueOfTerm` method does not find a value associated to any variables. The following snippet illustrates that this behaviour only happens when the `-Yrepl-class-based` option is set in the repl (the option is required for Spark to correctly serialize objects). {code} object Main extends App { val settings = new Settings settings.usejavacp.value = true //settings.Yreplclassbased.value = true val iMain: IMain = new IMain(settings) iMain.initializeSynchronous() iMain.interpret("val x = 1") iMain.definedTerms.foreach { name => println("defined term: " + name.toString) iMain.valueOfTerm(name.toString) match { case Some(value) => println("value: " + value) case None => println("no value") } } } {code} Above codes yields: {code} [info] Running foo.Main [info] x: Int = 1 [info] defined term: x [info] value: 1 {code} When setting -Y-repl-class-based by uncommenting above comment: {code} [info] Running foo.Main [info] x: Int = 1 [info] defined term: x [info] no value {code} > Incorrect fully qualified name for spark context > ------------------------------------------------ > > Key: TOREE-375 > URL: https://issues.apache.org/jira/browse/TOREE-375 > Project: TOREE > Issue Type: Bug > Environment: Jupyter Notebook with Toree latest master > (1a9c11f5f1381c15b691a716acd0e1f0432a9a35) and Spark 2.0.2, Scala 2.11 > Reporter: Felix Schüler > > When running below snippet in a cell I get a compile error for the MLContext > Constructor. Somehow the fully qualified name of the SparkContext gets messed > up. > The same does not happen when I start a Spark shell with the --jars command > and create the MLContext there. > Snippet (the systemml jar is build with the latest master of SystemML): > {code} > %addjar > file:///home/felix/repos/incubator-systemml/target/systemml-0.13.0-incubating-SNAPSHOT.jar > -f > import org.apache.sysml.api.mlcontext._ > import org.apache.sysml.api.mlcontext.ScriptFactory._ > val ml = new MLContext(sc) > Starting download from > file:///home/felix/repos/incubator-systemml/target/systemml-0.13.0-incubating-SNAPSHOT.jar > Finished download of systemml-0.13.0-incubating-SNAPSHOT.jar > Name: Compile Error > Message: <console>:25: error: overloaded method constructor MLContext with > alternatives: > (x$1: > org.apache.spark.api.java.JavaSparkContext)org.apache.sysml.api.mlcontext.MLContext > <and> > (x$1: > org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.SparkContext)org.apache.sysml.api.mlcontext.MLContext > cannot be applied to > (org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.org.apache.spark.SparkContext) > val ml = new MLContext(sc) > ^ > StackTrace: > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)