Handle FileAlreadyExistsException for .spark-staging files

Hello all, I met the following exception while writing data after several executors had lost. org.apache.hadoop.fs.FileAlreadyExistsException: File already exists:wasbs://

Re: [SQL][AQE] Advice needed: a trivial code change with a huge reading impact?

Salut Sean ! Merci beaucoup mon ami Sean ! That's exactly an answer I hoped for. Thank you! Pozdrawiam, Jacek Laskowski https://about.me/JacekLaskowski "The Internals Of" Online Books Follow me on https://twitter.com/jaceklaskowski

Re: [SQL][AQE] Advice needed: a trivial code change with a huge reading impact?

That does seem pointless. The body could just be .flatten()-ed to achieve the same result. Maybe it was just written that way for symmetry with the block above. You could open a PR to change it. On Wed, Sep 8, 2021 at 4:31 AM Jacek Laskowski wrote: > Hi Spark Devs, > > I'm curious what your

[SQL][AQE] Advice needed: a trivial code change with a huge reading impact?

Hi Spark Devs, I'm curious what your take on this code [1] would be if you were me trying to understand it: (0 until 1).flatMap { _ => (splitPoints :+ numMappers).sliding(2).map { case Seq(start, end) => CoalescedMapperPartitionSpec(start, end, numReducers) }