Re: [Spark SQL, intermediate+] possible bug or weird behavior of insertInto

2021-03-04 Thread Oldrich Vlasic
rict schema matching". From: Jeff Evans Sent: Thursday, March 4, 2021 2:55 PM To: Oldrich Vlasic Cc: Russell Spitzer ; Sean Owen ; user ; Ondřej Havlíček Subject: Re: [Spark SQL, intermediate+] possible bug or weird behavior of insertInto Why not perform a df.select(...) before th

Re: [Spark SQL, intermediate+] possible bug or weird behavior of insertInto

2021-03-04 Thread Oldrich Vlasic
Thanks for reply! Is there something to be done, setting a config property for example? I'd like to prevent users (mainly data scientists) from falling victim to this. From: Russell Spitzer Sent: Wednesday, March 3, 2021 3:31 PM To: Sean Owen Cc: Oldrich V

[Spark SQL, intermediate+] possible bug or weird behavior of insertInto

2021-03-02 Thread Oldrich Vlasic
Hi, I have encountered a weird and potentially dangerous behaviour of Spark concerning partial overwrites of partitioned data. Not sure if this is a bug or just abstraction leak. I have checked Spark section of Stack Overflow and haven't found any relevant questions or answers. Full minimal wo