[ https://issues.apache.org/jira/browse/SPARK-13094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15124278#comment-15124278 ]
Deenar Toraskar commented on SPARK-13094: ----------------------------------------- [~marmbrus] Still the same error on nightly snapshot build of 1.6.0 http://people.apache.org/~pwendell/spark-nightly/spark-branch-1.6-bin/latest/spark-1.6.0-SNAPSHOT-bin-hadoop2.6.tgz > Dataset Aggregators do not work with complex types > -------------------------------------------------- > > Key: SPARK-13094 > URL: https://issues.apache.org/jira/browse/SPARK-13094 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.6.0 > Reporter: Deenar Toraskar > > Dataset aggregators with complex types fail with unable to find encoder for > type stored in a Dataset. Though Datasets with these complex types are > supported. > val arraySum = new Aggregator[Seq[Float], Seq[Float], > Seq[Float]] with Serializable { > def zero: Seq[Float] = Nil > // The initial value. > def reduce(currentSum: Seq[Float], currentRow: Seq[Float]) = > sumArray(currentSum, currentRow) > def merge(sum: Seq[Float], row: Seq[Float]) = sumArray(sum, row) > def finish(b: Seq[Float]) = b // Return the final result. > def sumArray(a: Seq[Float], b: Seq[Float]): Seq[Float] = { > (a, b) match { > case (Nil, Nil) => Nil > case (Nil, row) => row > case (sum, Nil) => sum > case (sum, row) => (a, b).zipped.map { case (a, b) => a + b } > } > } > }.toColumn > <console>:47: error: Unable to find encoder for type stored in a Dataset. > Primitive types (Int, String, etc) and Product types (case classes) are > supported by importing sqlContext.implicits._ Support for serializing other > types will be added in future releases. > }.toColumn -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org