[
https://issues.apache.org/jira/browse/HIVE-17291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16126257#comment-16126257
]
Peter Vary commented on HIVE-17291:
-----------------------------------
For testing this is exactly what I did in HIVE-17292.3.patch. I am not sure why
we do not have the flakiness in this case when the configuration is changed,
and the spark session is killed and restarted. Logic suggests, that in this
case we will request new executors based on the new settings, and we are in the
middle of the query test file, so [~lirui]'s magic does not applies here.
For example:
{code:title=spark_dynamic_partition_pruning_2.q}
EXPLAIN SELECT d1.label, count(*), sum(agg.amount)
[..]
set hive.spark.dynamic.partition.pruning.max.data.size=1; <-- I think new
session is started here
EXPLAIN SELECT d1.label, count(*), sum(agg.amount)
[..]
{code}
> Set the number of executors based on config if client does not provide
> information
> ----------------------------------------------------------------------------------
>
> Key: HIVE-17291
> URL: https://issues.apache.org/jira/browse/HIVE-17291
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Affects Versions: 3.0.0
> Reporter: Peter Vary
> Assignee: Peter Vary
> Attachments: HIVE-17291.1.patch
>
>
> When calculating the memory and cores and the client does not provide
> information we should try to use the one provided by default. This can happen
> on startup, when {{spark.dynamicAllocation.enabled}} is not enabled
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
