[ https://issues.apache.org/jira/browse/MAHOUT-1653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14582610#comment-14582610 ]
ASF GitHub Bot commented on MAHOUT-1653: ---------------------------------------- Github user pferrel commented on a diff in the pull request: https://github.com/apache/mahout/pull/136#discussion_r32275250 --- Diff: spark/src/main/scala/org/apache/mahout/sparkbindings/drm/CheckpointedDrmSpark.scala --- @@ -165,7 +168,14 @@ class CheckpointedDrmSpark[K: ClassTag]( else if (classOf[Writable].isAssignableFrom(ktag.runtimeClass)) (x: K) => x.asInstanceOf[Writable] else throw new IllegalArgumentException("Do not know how to convert class tag %s to Writable.".format(ktag)) - rdd.saveAsSequenceFile(path) + // rdd.saveAsSequenceFile(path) + // this is a (working) deprecated method used as a stop-gap while we investigate the shell issues + SparkContext.rddToSequenceFileRDDFunctions(rdd.asInstanceOf[RDD[(K, Vector)]]).saveAsSequenceFile(path) + // discussion here: http://search-lucene.com/m/rcu7o1J1Z7mvnGmP1 + // The correct function to use is here: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/RDD.scala#L1671 + // The commit message for this change is here: https://github.com/apache/spark/tree/master/core/src/test/scala/org/apache/sparktest + // The key and value factories will be determined using reflection on the RDD if the right classes are set to null + // see this example: https://github.com/apache/spark/blob/branch-1.4/core/src/main/scala/org/apache/spark/SparkContext.scala#L2236 } --- End diff -- There seems to be a way to use reflection to get the Writable factory from the RDD by setting it to null. not sure exactly how to do that here. > Spark 1.3 > --------- > > Key: MAHOUT-1653 > URL: https://issues.apache.org/jira/browse/MAHOUT-1653 > Project: Mahout > Issue Type: Dependency upgrade > Reporter: Andrew Musselman > Assignee: Andrew Palumbo > Fix For: 0.11.0 > > > Support Spark 1.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)