Re: [SQL] s.s.a.coalescePartitions.parallelismFirst true but recommends false

Thanks Wenchen. If it's ever asked on SO I'm simply gonna quote you :) Pozdrawiam, Jacek Laskowski https://about.me/JacekLaskowski "The Internals Of" Online Books Follow me on https://twitter.com/jaceklaskowski On Tue, Sep 7,

Re: [SQL] s.s.a.coalescePartitions.parallelismFirst true but recommends false

This is correct. It's true by default so that AQE doesn't have performance regression. If you run a benchmark, larger parallelism usually means better performance. However, it's recommended to set it to false, so that AQE can give better resource utilization, which is good for a busy Spark

[SQL] s.s.a.coalescePartitions.parallelismFirst true but recommends false

Hi, Found this new spark.sql.adaptive.coalescePartitions.parallelismFirst config property [1] with the default value `true` but the descriptions says the opposite: > It's recommended to set this config to false Is this OK and I'm misreading it? [1]