[jira] [Commented] (HIVE-17362) The MAX_PREWARM_TIME should be configurable on HoS

[Peter Vary (JIRA)]

    [ 
https://issues.apache.org/jira/browse/HIVE-17362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16135615#comment-16135615
 ] 

Peter Vary commented on HIVE-17362:
-----------------------------------

Thanks for the review [~xuefuz]!

According to my tests 30s should be enough. I used 4 mins only because the 
original code in QTestUtil used this specific value:
{code:title=QTestUtil}
  private CliSessionState createSessionState() {
   return new CliSessionState(conf) {
      @Override
      public void setSparkSession(SparkSession sparkSession) {
[..]
            // Wait a little for cluster to init, at most 4 minutes
            long endTime = System.currentTimeMillis() + 240000;
[..]
      }
    };
  }
{code}

Shall I change it to a lower value?

Good catch with the {{hive.prewarm.numcontainers}}. We have to set it in case 
of the standalone configuration, thanks for pointing that out. With the 
yarn-master we do not need it, since we define {{spark.executor.instances}}

Also there was an issue with perwarm - it was only used in case of yarn master. 
We should enable it in local mode as well. See the original code. This caused 
the {{TestSparkNegativeCliDriver}} failures:
{code:title=RemoteHiveSparkClient}
  private void createRemoteClient() throws Exception {
    remoteClient = SparkClientFactory.createClient(conf, hiveConf);

    if (HiveConf.getBoolVar(hiveConf, ConfVars.HIVE_PREWARM_ENABLED) &&
            SparkClientUtilities.isYarnMaster(hiveConf.get("spark.master"))) {
[..]
    }
  }
{code}

> The MAX_PREWARM_TIME should be configurable on HoS
> --------------------------------------------------
>
>                 Key: HIVE-17362
>                 URL: https://issues.apache.org/jira/browse/HIVE-17362
>             Project: Hive
>          Issue Type: Improvement
>          Components: Spark
>    Affects Versions: 3.0.0
>            Reporter: Peter Vary
>            Assignee: Peter Vary
>         Attachments: HIVE-17362.patch
>
>
> When using HIVE_PREWARM_ENABLED, we are waiting MAX_PREWARM_TIME for the 
> containers to warm up. This is currently set to 5s. This is often not enough 
> for a spark session to initialize the executors. We should be able to 
> configure this, so we can set a value which has an effect.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)