[
https://issues.apache.org/jira/browse/TEZ-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16811408#comment-16811408
]
Siddharth Seth commented on TEZ-1348:
-------------------------------------
Took a while to refresh my memory about Local mode etc. Local mode allows
running the AM locally, along with all tasks. Also, to some extent, parts of
the code facilitate running tasks in the AM on a regular YARN based AM. The
second code path isn't affected here.
The main issue was that setting configs in TezConfiguration in the TezClient,
which gets propagated to the AM, is not sufficient. Input/Output configs pick
up their configs from tez-site.xml. Setting up file:/// as defaultFS in
tez-site for this to work, would end up affecting the AM.
The patch looks good to me. +1 (Still needs to go through jenkins). I think it
fixes the fs.defaultFS issue. Parameters like tez.runtime.optimize.local.fetch
still need to be set up in tez-site.xml. [~tlipcon] - can you please confirm
that you have the optimize local fetch config in your tez-site. Otherwise want
to go back and read some more code.
> Setup configs required for local mode automatically, instead of relying on
> changes to tez-site
> ----------------------------------------------------------------------------------------------
>
> Key: TEZ-1348
> URL: https://issues.apache.org/jira/browse/TEZ-1348
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Todd Lipcon
> Priority: Critical
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In TEZ-717, I incorrect thought setting fs.defaultFS programmatically in
> tez-site would work for local mode.
> Currently the requirement is that tez-site.xml must have fs.defaultFS set to
> file:///.
> While that works, it doesn't allow for seamless execution in either
> local-mode or on a cluster.
> The main issue here is that when Inputs / Outputs are configured - they use a
> version of configuration which reads tez-site, and do not use the
> configuration from the client itself (which is correct behaviour).
> Not sure what a good way to fix this is
> 1) It may be possible to override this value each time an instance of
> Configuration/TezConfiguration is created. One possible way would be to
> statically add a default resource to Configuration the moment a local client
> is created.
> 2) Provide information in the contexts on whether this is local or not. This
> is fairly ugly, and would get in the way of running mixed mode tasks.
> Anyone have other suggestions ?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
