For Edges, the approach that you took with edgeBuilder.setAdditionalConfiguration will work to set relevant Tez properties for an edge. You should be able to iterate through properties and set the config on the edge - and the relevant ones will be set. (Compression has a specific API which you could use, but using setAdditionalConfiguration will also work). Typically, additional Hadoop properties are also required for Edges - things like the list of compression codecs. edgeConfigs.setAdditionalConfiguration does take care of allowing these properties through.
The TezClient needs to be provided a config - which is then made available to the AM. There's not much filtering involved here, and you could set tez.* for this configuration instance. An attempt will be made to pick up YarnConfiguration to connect to the cluster. The same applies for InputInitializers and OutputCommitters. Typically (and unfortunately), you'll end up setting all configs. dag.setConf, and vertex.setConf should not be used - I've opened a jira to add docs for these. How do you get the Hadoop configs in this case ? Is that part of the Configuration like object ? On Mon, Sep 14, 2015 at 9:47 AM, Johannes Zillmann <[email protected] > wrote: > Ok, > > found it. The > > edgeBuilder.setAdditionalConfiguration(TezRuntimeConfiguration.TEZ_RUNTIME_COMPRESS, > "true”); > does work for me! > > So let me describe my use case a little bit... > Basically i have one Configuration like object on the client side. This is > assembled by multiple sources and the only way a user can set custom Tez > properties (do not use tez-site.xml in any perspective). > Then i’m building my DAG with its vertices and edges programatically. > Now, do you have any recommendation for me how to route the right Tez > properties effectively to the corresponding Tez components ? (with tez > components i mean like vertex properties, dag properties, AM properties, > edge properties, etc..) > > Should i simply set all tez.* properties to any component or is there a > smarter way ? > And what components/properties might i’m missing ? > > Any help appreciated! > Johannes > > > On 14 Sep 2015, at 16:57, Johannes Zillmann <[email protected]> > wrote: > > Hey guys, > > question. How do i enabled tez.runtime.compress programatically ? > When i set this property in the tez-site.xml it is picket up correctly. > But all other options i tried: > - dag.setConf(TezRuntimeConfiguration.TEZ_RUNTIME_COMPRESS, "true"); > - mapVertex.setConf(TezRuntimeConfiguration.TEZ_RUNTIME_COMPRESS, "true"); > - reduceVertex.setConf(TezRuntimeConfiguration.TEZ_RUNTIME_COMPRESS, "true > ”); > > do not have any effect! (Checking the log output of the Shuffle class) > > Johannes > > >
