[ https://issues.apache.org/jira/browse/MAHOUT-1653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14616106#comment-14616106 ]
ASF GitHub Bot commented on MAHOUT-1653: ---------------------------------------- Github user andrewpalumbo commented on a diff in the pull request: https://github.com/apache/mahout/pull/146#discussion_r34004807 --- Diff: spark-shell/src/main/scala/org/apache/mahout/sparkbindings/shell/MahoutSparkILoop.scala --- @@ -48,55 +77,81 @@ class MahoutSparkILoop extends SparkILoop { conf.set("spark.executor.memory", "1g") - sparkContext = mahoutSparkContext( + _interp.sparkContext = mahoutSparkContext( masterUrl = master, appName = "Mahout Spark Shell", customJars = jars, sparkConf = conf ) - echo("Created spark context..") + echoToShell("Created spark context..") sparkContext } + // this is technically not part of Spark's explicitly defined Developer API though + // nothing in the SparkILoopInit.scala file is marked as such. override def initializeSpark() { - intp.beQuietDuring { - command(""" + _interp.beQuietDuring { + + // initalize the Mahout Context as a SparkDistributedContext since so that we can access + // the SparkContext from it. + _interp.interpret(""" - @transient implicit val sdc: org.apache.mahout.math.drm.DistributedContext = + @transient implicit val sdc: org.apache.mahout.sparkbindings.SparkDistributedContext = new org.apache.mahout.sparkbindings.SparkDistributedContext( org.apache.spark.repl.Main.interp.createSparkContext()) + """) + echoToShell("Mahout distributed context is available as \"implicit val sdc\".") + + // get the spark context from the mahout distributed context. + _interp.interpret("import org.apache.spark.SparkContext._") + _interp.interpret("@transient val sc: org.apache.spark.SparkContext = sdc.sc") + echoToShell("Spark context is available as \"val sc\".") --- End diff -- Reset the mahout distributed context to `implicit val sdc`. Set `val sc` to the SparkContext in `sdc`. Created a SqlContext as is done in the overridden method. > Spark 1.3 > --------- > > Key: MAHOUT-1653 > URL: https://issues.apache.org/jira/browse/MAHOUT-1653 > Project: Mahout > Issue Type: Dependency upgrade > Reporter: Andrew Musselman > Assignee: Andrew Palumbo > Fix For: 0.11.0 > > > Support Spark 1.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)