Hi Jason,
Is there a reason why you are not adding the desired column before mapping
it to a Dataset[CC]?
You could just do something like:
df = df.withColumn('f2', )
then do the:
df.as(CC)
Of course your default value can be null: lit(None).cast(to-some-type)
best,
On Thu, Jun 15, 2017 at
Is it possible to concisely create a dataset from a dataframe with missing
columns? Specifically, suppose I create a dataframe with:
val df: DataFrame = Seq(("v1"),("v2")).toDF("f1")
Then, I have a case class for a dataset defined as:
case class CC(f1: String, f2: Option[String] = None)
I’d
Is it possible to concisely create a dataset from a dataframe with missing
columns? Specifically, suppose I create a dataframe with:
val df: DataFrame = Seq(("v1"),("v2")).toDF("f1")
Then, I have a case class for a dataset defined as:
case class CC(f1: String, f2: Option[String] = None)
I’d
