Hi
I believe there is a feature in Spark specifically for this purpose. You
can create a new spark session and set those configs.
Note that it's not the same as creating a separate driver processes with
separate sessions, here you will still have the same SparkContext that
works as a backend for b
AM
To: Felipe Pessoto ; user@spark.apache.org
Subject: [EXTERNAL] Re: How to set a config for a single query?
ATTENTION: This email originated from outside of GM.
Hey Felipe,
Since you are collecting the dataframes, you might as well run them separately
with desired configs and store them in
set a config for a single query?
Caution! This email originated outside of FedEx. Please do not open attachments
or click links from an unknown or suspicious origin.
Hi,
In Scala is it possible to set a config value to a single query?
I could set/unset the value, but it won’t work for
Hi,
In Scala is it possible to set a config value to a single query?
I could set/unset the value, but it won't work for multithreading scenarios.
Example:
spark.sql.adaptive.coalescePartitions.enabled = false
queryA_df.collect
spark.sql.adaptive.coalescePartitions.enabled=origin
