Csaba Ringhofer created IMPALA-9609:
---------------------------------------
Summary: Minimize Frontend activity in executor only Impalas
Key: IMPALA-9609
URL: https://issues.apache.org/jira/browse/IMPALA-9609
Project: IMPALA
Issue Type: Improvement
Components: Backend, Frontend
Reporter: Csaba Ringhofer
Currently Impala both starts a JVM and creates a Frontend in executor mode
impalads:
JVM:
https://github.com/apache/impala/blob/04fd9ae268d89b07e2a692a916bf2ddcfb2e351b/be/src/service/impalad-main.cc#L61
The 3rd argument of is init_jvm. This is actually needed for some executor
features, e.g. Hive UDFs and HBase.
Frontend:
https://github.com/apache/impala/blob/d877dbc572e95f086b37142e1d41231d5b6b7f9f/be/src/runtime/exec-env.cc#L250
The frontend code itself doesn't really seem to care about whether it serves as
a coordinator.
It would be great to remove the Frontend object from executors completely, but
this may be tricky as Java features by the executor can have widespread
dependencies.
The critical part is to avoid starting services that are both not needed and
are problematic, e.g. connecting to HMS and Auth services. These can consume
resource and cause errors if not configured well.
The executor only setting is only tested by custom_cluster test, so probably
we'll need to extend those to ensure that HBase/Hive UDFs are not broken.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
