Hello,Since Dataset has no zip(..) methods, so I wrote following code to zip
two datasets:
1 def zipDatasets[X: Encoder, Y: Encoder](spark: SparkSession, m:
Dataset[X], n: Dataset[Y]) = {
2 val rdd = m.rdd.zip(n.rdd);
3 import spark.implicits._
4 spark.createDataset(rdd);
5 }
However, in the m.rdd.zip(…) call, compile error is reported: No ClassTag
available for Y
I know this error can be corrected when I declare Y as a ClassTag like this:
1 def foo[X: Encoder, Y: ClassTag](spark: SparkSession, …
But this will make line 5 report a new error:
Unable to find encoder for type stored in a Dataset.
Now, I have no idea to solve this problem. How to declared Y as both an Encoder
and a ClassTag?
Many thanks!
Best regards,
bluejoe
---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]
