Chao Sun created HIVE-16984:
-------------------------------
Summary: HoS: avoid waiting for RemoteSparkJobStatus::getAppID()
when remote driver died
Key: HIVE-16984
URL: https://issues.apache.org/jira/browse/HIVE-16984
Project: Hive
Issue Type: Bug
Components: Spark
Reporter: Chao Sun
Assignee: Chao Sun
In HoS, after a RemoteDriver is launched, it may fail to initialize a Spark
context and thus the ApplicationMaster will die eventually. In this case, there
are two issues related to RemoteSparkJobStatus::getAppID():
1. Currently we call {{getAppID()}} before starting the monitoring job. For the
first, it will wait for {{hive.spark.client.future.timeout}}, and for the
latter, it will wait for {{hive.spark.job.monitor.timeout}}. The error message
for the latter treats the {{hive.spark.job.monitor.timeout}} as the time
waiting for the job submission. However, this is inaccurate as it doesn't
include {{hive.spark.client.future.timeout}}.
2. In case the RemoteDriver suddenly died, currently we still may wait
hopelessly for the timeouts. This should potentially be avoided if we know that
the channel has closed between the client and remote driver.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
