[
https://issues.apache.org/jira/browse/HIVE-17111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091008#comment-16091008
]
Rui Li commented on HIVE-17111:
-------------------------------
I agree our tests should cover LocalHiveSparkClient. But since the remote mode
is what we really use in production, shouldn't we focus on this mode in tests?
I'm even wondering if it makes sense to move all the tests of
TestSparkCliDriver to TestMiniSparkOnYarnCliDriver. I believe the local-cluster
mode was used as a "workaround" when we couldn't run HoS tests on a mini
cluster (and didn't want to use pure local mode).
[~xuefuz], [~csun], [~jxiang], what do you think?
> TestSparkCliDriver does not use LocalHiveSparkClient
> ----------------------------------------------------
>
> Key: HIVE-17111
> URL: https://issues.apache.org/jira/browse/HIVE-17111
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
>
> The TestSparkCliDriver sets the spark.master to local-cluster[2,2,1024] but
> the HoS still uses decides to use the RemoteHiveSparkClient rather than the
> LocalHiveSparkClient.
> The issue is with the following check in HiveSparkClientFactory:
> {code}
> if (master.equals("local") || master.startsWith("local[")) {
> // With local spark context, all user sessions share the same spark
> context.
> return LocalHiveSparkClient.getInstance(generateSparkConf(sparkConf));
> } else {
> return new RemoteHiveSparkClient(hiveconf, sparkConf);
> }
> {code}
> When {{master.startsWith("local[")}} it checks the value of spark.master and
> sees that it doesn't start with {{local[}} and then decides to use the
> RemoteHiveSparkClient.
> We should fix this so that the LocalHiveSparkClient is used. It should speed
> up some of the tests, and also makes qtests easier to debug since everything
> will now be run in the same process.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
