[
https://issues.apache.org/jira/browse/TEZ-4685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18059262#comment-18059262
]
Ayush Saxena commented on TEZ-4685:
-----------------------------------
You’re right about *precedence* — client-passed configs will override
*tez-site.xml.*
But the subtle problem TEZ-346 was solving is not precedence, it’s *pollution*
+ {*}nondeterminism{*}. These two are different.
Let me unpack it cleanly in YARN terms
The original problem wasn’t: -> “Client config is ignored”
It was: -> “{_}*AM is reading configs it should never even see.*{_}”
if {{tez-site.xml}} on the node contains:
{noformat}
<property>
<name>tez.am.resource.memory.mb</name>
<value>16384</value>
</property>
{noformat}
and the client does not specify this, the AM now silently picks it up. Firstly
this is a behaviour change now and another it means now:
* The DAG behavior depends on what’s installed on the cluster node
* Two clusters with same client config can behave differently
* Even two nodes in the same cluster (misconfigured) could behave differently
--> This is exactly what I believe [~hitesh] in TEZ-346 wanted to eliminate.
*The design intent I believe was:*
AM config = only what client sends + Hadoop defaults NOT “whatever is lying
around in tez-site.xml on the node”
> DagAppMaster is not picking tez-site.xml from classpath
> -------------------------------------------------------
>
> Key: TEZ-4685
> URL: https://issues.apache.org/jira/browse/TEZ-4685
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Raghav Aggarwal
> Assignee: Raghav Aggarwal
> Priority: Major
> Attachments: After.png, Before.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In DAGAppMaster#main()
> [https://github.com/apache/tez/blob/1fc00352ccae70d265649621aca22a5904ba771f/tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java#L2432]
> {code:java}
> Configuration conf = new Configuration(); {code}
> is not picking tez-site.xml from classpath. Even during debugging
> tez-site.xml is in classpath, its not getting picked up
> Moving to TezConfiguration is helping
> {code:java}
> Configuration conf = new TezConfiguration();{code}
> Please check the attached screenshots
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
