Hi all, when I read parquet files with "required" fields aka nullable=false they are read correctly. Then I save them (df.write.parquet) and read again all my fields are saved and read as optional, aka nullable=true. Which means I suddenly have files with incompatible schemas. This happens on 1.3.0-1.4.1 and even on 1.5.1-rc1.
Should I set some write option to keep nullability? Is there a specific reason why nullability is always overriden to true? Many thanks, Peter
