Thanks Wenchen. If it's ever asked on SO I'm simply gonna quote you :) Pozdrawiam, Jacek Laskowski ---- https://about.me/JacekLaskowski "The Internals Of" Online Books <https://books.japila.pl/> Follow me on https://twitter.com/jaceklaskowski
<https://twitter.com/jaceklaskowski> On Tue, Sep 7, 2021 at 6:58 AM Wenchen Fan <cloud0...@gmail.com> wrote: > This is correct. It's true by default so that AQE doesn't have performance > regression. If you run a benchmark, larger parallelism usually means better > performance. However, it's recommended to set it to false, so that AQE can > give better resource utilization, which is good for a busy Spark cluster. > > On Fri, Sep 3, 2021 at 7:33 PM Jacek Laskowski <ja...@japila.pl> wrote: > >> Hi, >> >> Found this new spark.sql.adaptive.coalescePartitions.parallelismFirst >> config property [1] with the default value `true` but the descriptions says >> the opposite: >> >> > It's recommended to set this config to false >> >> Is this OK and I'm misreading it? >> >> [1] >> https://github.com/apache/spark/blob/54cca7f82ecf23e062bb4f6d68697abec2dbcc5b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L519-L530 >> >> Pozdrawiam, >> Jacek Laskowski >> ---- >> https://about.me/JacekLaskowski >> "The Internals Of" Online Books <https://books.japila.pl/> >> Follow me on https://twitter.com/jaceklaskowski >> >> <https://twitter.com/jaceklaskowski> >> >
