I'm attempting to perform a map on a Dataset[Row] but getting an error on decode when attempting to pass a custom encoder. My code looks similar to the following:
val source =
spark.read.format("parquet").load("/emrdata/sources/very_large_ds")
source.map{ row => {
val key = row(0)
}
}
