Re: How to deal Schema Evolution with Dataset API

2020-05-11 Thread Mike Thomsen
This should be posted on the Spark user list, not the NiFi one. On Sat, May 9, 2020 at 3:07 AM Jorge Machado wrote: > Hello everyone, > > One question to the community. > > Imagine I have this > > Case class Person(age: int) > > spark.read.parquet(“inputPath”).as[Person] > > >

How to deal Schema Evolution with Dataset API

2020-05-09 Thread Jorge Machado
Hello everyone, One question to the community. Imagine I have this Case class Person(age: int) spark.read.parquet(“inputPath”).as[Person] After a few weeks of coding I change the class to: Case class Person(age: int, name: Option[String] = None) Then when I run