[
https://issues.apache.org/jira/browse/HIVE-17111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091045#comment-16091045
]
Sahil Takiar commented on HIVE-17111:
-------------------------------------
I'm good either way. While TestMiniSparkOnYarnCliDriver is closer to what is
used in production, a lot of qtests probably don't need to be run on a
mini-cluster of processes. If everything is run in the same process, the test
coverage is probably similar. I think Hive-on-MR qtests is all in process, and
some of the LLAP qtests are in process too. The main advantage is reduced time
to run the tests.
That being said, I'm good with either approach. Basically, I think there should
just be a way to run a HoS qtest entirely in process (could just be a config
option). It can help a lot with debugging.
> TestSparkCliDriver does not use LocalHiveSparkClient
> ----------------------------------------------------
>
> Key: HIVE-17111
> URL: https://issues.apache.org/jira/browse/HIVE-17111
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
>
> The TestSparkCliDriver sets the spark.master to local-cluster[2,2,1024] but
> the HoS still uses decides to use the RemoteHiveSparkClient rather than the
> LocalHiveSparkClient.
> The issue is with the following check in HiveSparkClientFactory:
> {code}
> if (master.equals("local") || master.startsWith("local[")) {
> // With local spark context, all user sessions share the same spark
> context.
> return LocalHiveSparkClient.getInstance(generateSparkConf(sparkConf));
> } else {
> return new RemoteHiveSparkClient(hiveconf, sparkConf);
> }
> {code}
> When {{master.startsWith("local[")}} it checks the value of spark.master and
> sees that it doesn't start with {{local[}} and then decides to use the
> RemoteHiveSparkClient.
> We should fix this so that the LocalHiveSparkClient is used. It should speed
> up some of the tests, and also makes qtests easier to debug since everything
> will now be run in the same process.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
