[
https://issues.apache.org/jira/browse/TEZ-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated TEZ-4650:
------------------------------
Description:
Proposal is to remove the method:
[processSchedulerDescriptors|https://github.com/apache/tez/blob/0a5c7142c8ac5dff0fe6b18358bdb9eb99e2a678/tez-dag/src/main/java/org/apache/tez/dag/app/PluginManager.java#L198-L225]
It is for two purposes:
1. If, according to a variable *isLocal*, local mode is enabled (so
*uberEnabled*), checks if the task scheduler plugin descriptors list contain a
corresponding *TezUber* entity. This is unnecessary, because if so, it's a bug
in
[parsePlugin|https://github.com/apache/tez/blob/0a5c7142c8ac5dff0fe6b18358bdb9eb99e2a678/tez-dag/src/main/java/org/apache/tez/dag/app/PluginManager.java#L170-L175]
method, which ran previously and should have taken care about the same.
2. If, according to a variable, containers are enabled (so *tezYarnEnabled*),
but there is no corresponding *TezYarn* entity in the descriptors, it magically
adds it: so this fixes a bug where parsePlugin didn't take care of the same
previously
[here|https://github.com/apache/tez/blob/0a5c7142c8ac5dff0fe6b18358bdb9eb99e2a678/tez-dag/src/main/java/org/apache/tez/dag/app/PluginManager.java#L163-L168]
This is just collection of hacks: in case an upstream application creates the
ServicePluginsDescriptor properly (like hive does
[here|https://github.com/apache/hive/blob/c338904b80d4dc6df08981738b69fa136bf6c624/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java#L330-L350]),
we should never hit this codepath. I'm assuming this is a leftover of early
days of tez and local mode developments.
was:
Proposal is to remove the method:
[processSchedulerDescriptors|https://github.com/apache/tez/blob/0a5c7142c8ac5dff0fe6b18358bdb9eb99e2a678/tez-dag/src/main/java/org/apache/tez/dag/app/PluginManager.java#L198-L225]
It is for the purposes:
1. If, according to a variable isLocal, local mode is enabled (so uberEnabled),
checks if the task scheduler plugin descriptors contain a corresponding
*TezUber* entity: this is useless...if it's otherwise, it's a bug in
[parsePlugin|https://github.com/apache/tez/blob/0a5c7142c8ac5dff0fe6b18358bdb9eb99e2a678/tez-dag/src/main/java/org/apache/tez/dag/app/PluginManager.java#L170-L175]
method, which run previously and takes care about the same
2. If, according to a variable, containers are enabled (so tezYarnEnabled), but
there is no corresponding *TezYarn* entity in the descriptors, it magically
adds it: so this fixes a bug where parsePlugin didn't take care of the same
previously
[here|https://github.com/apache/tez/blob/0a5c7142c8ac5dff0fe6b18358bdb9eb99e2a678/tez-dag/src/main/java/org/apache/tez/dag/app/PluginManager.java#L163-L168]
This is just a hack: in case an upstream application creates the
ServicePluginsDescriptor properly (like hive does
[here|https://github.com/apache/hive/blob/c338904b80d4dc6df08981738b69fa136bf6c624/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java#L330-L350]),
we should never hit this codepath. I'm assuming this is a leftover of early
days of tez and local mode developments.
> Remove useless logic from AM plugin management: processSchedulerDescriptors
> ---------------------------------------------------------------------------
>
> Key: TEZ-4650
> URL: https://issues.apache.org/jira/browse/TEZ-4650
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Proposal is to remove the method:
> [processSchedulerDescriptors|https://github.com/apache/tez/blob/0a5c7142c8ac5dff0fe6b18358bdb9eb99e2a678/tez-dag/src/main/java/org/apache/tez/dag/app/PluginManager.java#L198-L225]
> It is for two purposes:
> 1. If, according to a variable *isLocal*, local mode is enabled (so
> *uberEnabled*), checks if the task scheduler plugin descriptors list contain
> a corresponding *TezUber* entity. This is unnecessary, because if so, it's a
> bug in
> [parsePlugin|https://github.com/apache/tez/blob/0a5c7142c8ac5dff0fe6b18358bdb9eb99e2a678/tez-dag/src/main/java/org/apache/tez/dag/app/PluginManager.java#L170-L175]
> method, which ran previously and should have taken care about the same.
> 2. If, according to a variable, containers are enabled (so *tezYarnEnabled*),
> but there is no corresponding *TezYarn* entity in the descriptors, it
> magically adds it: so this fixes a bug where parsePlugin didn't take care of
> the same previously
> [here|https://github.com/apache/tez/blob/0a5c7142c8ac5dff0fe6b18358bdb9eb99e2a678/tez-dag/src/main/java/org/apache/tez/dag/app/PluginManager.java#L163-L168]
> This is just collection of hacks: in case an upstream application creates the
> ServicePluginsDescriptor properly (like hive does
> [here|https://github.com/apache/hive/blob/c338904b80d4dc6df08981738b69fa136bf6c624/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java#L330-L350]),
> we should never hit this codepath. I'm assuming this is a leftover of early
> days of tez and local mode developments.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
