Hi,
In Scala is it possible to set a config value to a single query?
I could set/unset the value, but it won't work for multithreading scenarios.
Example:
spark.sql.adaptive.coalescePartitions.enabled = false
queryA_df.collect
spark.sql.adaptive.coalescePartitions.enabled=original value
queryB_df.collect
queryC_df.collect
queryD_df.collect
If I execute that block of code multiple times using multiple thread, I can end
up executing Query A with coalescePartitions.enabled=true, and Queries B, C and
D with the config set to false, because another thread could set it between the
executions.
Is there any good alternative to this?
Thanks.