Hi all,
So I've been attempting to reformat a project I'm working on to use the
Dataset API and have been having some issues with encoding errors. From
what I've read, I think that I should be able to store Arrays of primitive
values in a dataset. However, the following class gives me encoding errors:
case class InvertedIndex(partition:Int, docs:Array[Int],
indices:Array[Long], weights:Array[Double])
val inv = RDD[InvertedIndex]
val invertedIndexDataset = sqlContext.createDataset(inv)
invertedIndexDataset.groupBy(x => x.partition).mapGroups {
//...
}
Could someone please help me understand what the issue is here? Can
Datasets not currently handle Arrays of primitives, or is there something
extra that I need to do to make them work?
Thank you
