Hi Mohit,
I’m not sure that there is a “correct” answer here, but I tend to use classes whenever the input or output data represents something meaningful (such as a domain model object). I would recommend against creating many temporary classes for each and every transformation step as that may be difficult to maintain over time. Using withColumn statements will continue to work, and you don’t need to cast to your output class until you’ve setup all tranformations. Therefore, you can do things like: case class A (f1, f2, f3) case class B (f1, f2, f3, f4, f5, f6) ds_a = spark.read.csv(“path”).as[A] ds_b = ds_a .withColumn(“f4”, someUdf) .withColumn(“f5”, someUdf) .withColumn(“f6”, someUdf) .as[B] Kevin From: Mohit Jaggi <mohitja...@gmail.com> Sent: Tuesday, January 15, 2019 1:31 PM To: user <user@spark.apache.org> Subject: dataset best practice question Fellow Spark Coders, I am trying to move from using Dataframes to Datasets for a reasonably large code base. Today the code looks like this: df_a= read_csv df_b = df.withColumn ( some_transform_that_adds_more_columns ) //repeat the above several times With datasets, this will require defining case class A { f1, f2, f3 } //fields from csv file case class B { f1, f2, f3, f4 } //union of A and new field added by some_transform_that_adds_more_columns //repeat this 10 times Is there a better way? Mohit.