Xuefu Zhang created HIVE-11276:
----------------------------------
Summary: Optimization around job submission and adding jars [Spark
Branch]
Key: HIVE-11276
URL: https://issues.apache.org/jira/browse/HIVE-11276
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: 1.1.0
Reporter: Xuefu Zhang
It seems that Hive on Spark has some room for performance improvement on job
submission. Specifically, we are calling refreshLocalResources() for every job
submission despite there is are no changes in the jar list. Since Hive on Spark
is reusing the containers in the whole user session, we might be able to
optimize that.
We do need to take into consideration the case of dynamic allocation, in which
new executors might be added.
This task is some R&D in this area.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
