[
https://issues.apache.org/jira/browse/HIVE-12538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15032702#comment-15032702
]
Jimmy Xiang commented on HIVE-12538:
------------------------------------
bq. 1,Fixing the bug : " isSparkConfigUpdated = isSparkRelatedConfig(name)"
The fix to HiveConf.java is good.
bq. 2,Setting "isSparkConfigUpdated =false " at session level
That means we reset the flag once we copy the hiveconf to the operation level.
This may not work in some cases, right? For example, the first query has the
flag set, and the second query has the flag not set, if the second query
happens to create the spark session earlier, it will not create a new spark
session.
I think it is better to fix SparkUtilities to use the session level conf in
checking if a new spark session should be created, and use the operation level
conf to create the speark session. What do you think?
bq. 3,Taking concurrency into consideration by adding a session level conf
lock.As multiple queries can run in a single session.
Multiple queries can run in a single session asynchronously, right? If so, each
one should has its own copy of conf object. If we are going to fix
SparkUtilities, we may just need to synchronized the specific area of method
SparkUtilities#getSparkSession().
> After set spark related config, SparkSession never get reused
> -------------------------------------------------------------
>
> Key: HIVE-12538
> URL: https://issues.apache.org/jira/browse/HIVE-12538
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Affects Versions: 1.3.0
> Reporter: Nemon Lou
> Assignee: Nemon Lou
> Attachments: HIVE-12538.patch
>
>
> Hive on Spark yarn-cluster mode.
> After setting "set spark.yarn.queue=QueueA;" ,
> run the query "select count(*) from test" 3 times and you will find 3
> different yarn applications.
> Two of the yarn applications in FINISHED & SUCCEEDED state,and one in RUNNING
> & UNDEFINED state waiting for next work.
> And if you submit one more "select count(*) from test" ,the third one will be
> in FINISHED & SUCCEEDED state and a new yarn application will start up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
