Is it possible to concisely create a dataset from a dataframe with fewer columns? Specifically, suppose I create a dataframe with: val df: DataFrame = Seq(("v1"),("v2")).toDF("f1")
Then, I have a case class for a dataset defined as: case class CC(f1: String, f2: Option[String] = None) I’d like to use df.as[CC] to get an instance of the case class, but this gives me the following error: org.apache.spark.sql.AnalysisException: cannot resolve '`f2`' given input columns: [f1]; Is there a concise way to use the default values as defined by the case class so that I don't have to explicitly add those columns to the dataframe before converting to a dataset? Warmest Regards, Jason Tokayer, PhD --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org