Re: DataFrame Parquet Writer doesn't keep schema

The same as https://mail.google.com/mail/#label/Spark%2Fuser/14f64c75c15f5ccd Please follow the discussion there. On Tue, Aug 25, 2015 at 5:02 PM, Petr Novak oss.mli...@gmail.com wrote: Hi all, when I read parquet files with required fields aka nullable=false they are read correctly. Then I

DataFrame Parquet Writer doesn't keep schema

Hi all, when I read parquet files with required fields aka nullable=false they are read correctly. Then I save them (df.write.parquet) and read again all my fields are saved and read as optional, aka nullable=true. Which means I suddenly have files with incompatible schemas. This happens on