Re: Spark 3.0.1 new Proleptic Gregorian calendar

2020-11-19 Thread Maxim Gekk
Hello Saurabh, > What config options should we set, > - if we are always going to read old data written from Spark2.4 using Spark 3.0 You should set *spark.sql.legacy.parquet.datetimeRebaseModeInRead* to *LEGACY *when you read old data*.* You see this exception because Spark 3.0 cannot

Spark 3.0.1 new Proleptic Gregorian calendar

2020-11-19 Thread Saurabh Gulati
Hello, First of all, Thanks to you guys for maintaining and improving Spark. We just updated to Spark 3.0.1 and are facing some issues with the new Proleptic Gregorian calendar. We have data from different sources in our platform and we saw there were some date/timestamp columns that go back