[
https://issues.apache.org/jira/browse/HIVE-12538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15032988#comment-15032988
]
Nemon Lou commented on HIVE-12538:
----------------------------------
[~jxiang] ,thanks for review.
{quote}
That means we reset the flag once we copy the hiveconf to the operation level.
This may not work in some cases, right? For example, the first query has the
flag set, and the second query has the flag not set, if the second query
happens to create the spark session earlier, it will not create a new spark
session.
{quote}
Aggreed
{quote}
I think it is better to fix SparkUtilities to use the session level conf in
checking if a new spark session should be created,
{quote}
Aggreed
{quote}
and use the operation level conf to create the spark session.
{quote}
Not quite follow.Is there anything special in operation conf for SparkSession?
And when to set "isSparkConfigUpdated =false " ?
Since each hive session only has one related spark session,i think it's should
be fine to use session conf to start spark session.
{quote}Multiple queries can run in a single session asynchronously, right? If
so, each one should has its own copy of conf object. If we are going to fix
SparkUtilities, we may just need to synchronized the specific area of method
SparkUtilities#getSparkSession().
{quote}
This depend on the question above : when to set "isSparkConfigUpdated =false "?
Client may set another spark related parameter during
SparkUtilities#getSparkSession(),that makes session level lock still needed.
> After set spark related config, SparkSession never get reused
> -------------------------------------------------------------
>
> Key: HIVE-12538
> URL: https://issues.apache.org/jira/browse/HIVE-12538
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Affects Versions: 1.3.0
> Reporter: Nemon Lou
> Assignee: Nemon Lou
> Attachments: HIVE-12538.patch
>
>
> Hive on Spark yarn-cluster mode.
> After setting "set spark.yarn.queue=QueueA;" ,
> run the query "select count(*) from test" 3 times and you will find 3
> different yarn applications.
> Two of the yarn applications in FINISHED & SUCCEEDED state,and one in RUNNING
> & UNDEFINED state waiting for next work.
> And if you submit one more "select count(*) from test" ,the third one will be
> in FINISHED & SUCCEEDED state and a new yarn application will start up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
