Ali Afroozeh created SPARK-30798:
------------------------------------
Summary: Scope Session.active in QueryExecution
Key: SPARK-30798
URL: https://issues.apache.org/jira/browse/SPARK-30798
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.0.0
Reporter: Ali Afroozeh
SparkSession.active is a thread local variable that points to the current
thread's spark session. It is important to note that the SQLConf.get method
depends on SparkSession.active. In the current implementation it is possible
that SparkSession.active points to a different session which causes various
problems. Most of these problems arise because part of the query processing is
done using the configurations of a different session. For example, when
creating a data frame using a new session, i.e., session.sql("..."), part of
the data frame is constructed using the currently active spark session, which
can be a different session from the one used later for processing the query.
This PR scopes SparkSession.active to prevent the above-mentioned problems. A
new method, withActive is introduced on SparkSession that restores the previous
spark session after the block of code is executed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
