[ https://issues.apache.org/jira/browse/TEZ-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17216670#comment-17216670 ]
László Bodor commented on TEZ-4236: ----------------------------------- could you please take a look at this chance [~ashutoshc], [~rajesh.balamohan]? this only affects tez local mode, so it's not supposed to cause problems in prod systems about the patch, feel free to comment on the [latest version|https://github.com/abstractdog/tez/commit/df5b1117e992833016c2b5ec8bc873fe75686b49] 1. [introduced new config|https://github.com/abstractdog/tez/commit/df5b1117e992833016c2b5ec8bc873fe75686b49#diff-bedc4ac5f0df37e202aa39d270d75e32cad8a996b39741c6e267c355ece8738eR1760], tez.local.mode.without.network, which is about to drive whether a dag rpc server is started for DAGAppMaster, or not <-- I was struggling with the proper config name, maybe it should be tez.am.dag.rpc.server=false to reflect the behavior 2. Adapt to the setting in LocalClient: instantiate the new LocalDAGAppMaster, which is a no-op for initClientRpcServer 3. Most of the change is done in TezClient/FrameworkClient: these changes are what I didn't do for the first time, but then I needed and it was worth...basically, TezClient is a base logic for communicating dag events, wrapping a FrameworkClient, which is an abstract for every specific implementation...now as LocalClient started to do more locally (without RPC), I needed to refactor stuff from TezClient -> FrameworkClient, because TezClient relied heavily on the fact that it should communicate with an RPC server...now, all these moved to FrameworkClient/TezYarnClient (getAMStatus, submitDag, shutdownSession) 4. adapting tests: [TestLocalMode|https://github.com/abstractdog/tez/commit/df5b1117e992833016c2b5ec8bc873fe75686b49#diff-d1d1b79a331141bde965de33cf204b77f15094760f7bea52d56abf1fe9fff93eR80] has a full coverage on the new config option [TestTezClient|https://github.com/abstractdog/tez/commit/df5b1117e992833016c2b5ec8bc873fe75686b49#diff-e3ed5501876742e26dfcfffa2a50f39fa22c02e3039ec8a6868022dc837539fbR141] has been changed in a way that it reflects that from new FrameworkClient is the class which needs to be "mocked" -> TezYarnClientForTest > Investigate if DAGAppMaster's DAGClientServer is really need to be > started/used in local mode > --------------------------------------------------------------------------------------------- > > Key: TEZ-4236 > URL: https://issues.apache.org/jira/browse/TEZ-4236 > Project: Apache Tez > Issue Type: Improvement > Reporter: László Bodor > Assignee: László Bodor > Priority: Major > Attachments: TEZ-4236.01.patch, TEZ-4236.02.patch, TEZ-4236.03.patch, > TEZ-4236.04.patch > > > For a customer, we found that they try to run hive on tez local mode tests, > and due to strict firewall rules and security policy, the DagClientServer > struggles to start or respond to connections (think about corporate laptops > here). Instead of investigating the network-related workarounds, let's find > another way and decide if DAGAppMaster really needs this server. > LocalClient has an embedded > [DAGAppMaster|https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/client/LocalClient.java#L65], > so at the first sight, it doesn't make sense to rely on the network > environment in terms of AM usage. Hopefully, AM can become more lightweight > in the long term if we try to eliminate pointless network usage. -- This message was sent by Atlassian Jira (v8.3.4#803005)