[jira] [Created] (TEZ-4074) Tez does not run with Hadoop Trunk (3.3.0-snapshot)
Ahmed Hussein created TEZ-4074: -- Summary: Tez does not run with Hadoop Trunk (3.3.0-snapshot) Key: TEZ-4074 URL: https://issues.apache.org/jira/browse/TEZ-4074 Project: Apache Tez Issue Type: Bug Reporter: Ahmed Hussein Tez throws a runtime exception when compiled against Hadoop-3.3.0. With Tez running Guava (11.0.2) and Hadoop eunning Guava 27.0-jre (see HADOOP-16210), there is an incompatibility of Guava library. {code:java} [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.136 s <<< FAILURE! - in org.apache.tez.dag.app.TestSpeculation [ERROR] org.apache.tez.dag.app.TestSpeculation Time elapsed: 0.136 s <<< ERROR! java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V at org.apache.tez.dag.app.TestSpeculation.setupSpeculation(TestSpeculation.java:86) {code} It looks like guava added single parameter optimizations which breaks compatibility with {{VAR_ARGS}}. So, even though it shows source compatibility it is throwing a runtime error due to binary incompatibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4074) Tez does not run with Hadoop Trunk (3.3.0-snapshot)
[ https://issues.apache.org/jira/browse/TEZ-4074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16853032#comment-16853032 ] Ahmed Hussein commented on TEZ-4074: Guava27 and 11.0.2 are not source compatible. For example, Guava27 removed API methods such as {{Futures.addCallback(ListenableFuture future, FutureCallback callback)}} * Guava11.0.2: [FutureCallback|https://google.github.io/guava/releases/11.0.2/api/docs/com/google/common/util/concurrent/Futures.html#addCallback(com.google.common.util.concurrent.ListenableFuture,%20com.google.common.util.concurrent.FutureCallback)] * Guava27: [FutureCallback|https://static.javadoc.io/com.google.guava/guava/27.0.1-jre/com/google/common/util/concurrent/Futures.html#addCallback-com.google.common.util.concurrent.ListenableFuture-com.google.common.util.concurrent.FutureCallback-java.util.concurrent.Executor-] > Tez does not run with Hadoop Trunk (3.3.0-snapshot) > --- > > Key: TEZ-4074 > URL: https://issues.apache.org/jira/browse/TEZ-4074 > Project: Apache Tez > Issue Type: Bug >Reporter: Ahmed Hussein >Priority: Major > > Tez throws a runtime exception when compiled against Hadoop-3.3.0. > With Tez running Guava (11.0.2) and Hadoop eunning Guava 27.0-jre (see > HADOOP-16210), there is an incompatibility of Guava library. > {code:java} > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.136 > s <<< FAILURE! - in org.apache.tez.dag.app.TestSpeculation > [ERROR] org.apache.tez.dag.app.TestSpeculation Time elapsed: 0.136 s <<< > ERROR! > java.lang.NoSuchMethodError: > com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V > at > org.apache.tez.dag.app.TestSpeculation.setupSpeculation(TestSpeculation.java:86) > {code} > It looks like guava added single parameter optimizations which breaks > compatibility with {{VAR_ARGS}}. So, even though it shows source > compatibility it is throwing a runtime error due to binary incompatibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4073) Configuration: Reduce Vertex and DAG Payload Size
[ https://issues.apache.org/jira/browse/TEZ-4073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16853527#comment-16853527 ] Gopal V commented on TEZ-4073: -- The trivial version of this is to skip all the configs which have *.xml when writing this out, since the all the XML files have already been parsed + sent for the AM init. > Configuration: Reduce Vertex and DAG Payload Size > - > > Key: TEZ-4073 > URL: https://issues.apache.org/jira/browse/TEZ-4073 > Project: Apache Tez > Issue Type: Bug >Reporter: Gopal V >Priority: Major > Attachments: tez-am-protobuf-reading.png, tez-protobuf-writing.png > > > As the total number of vertices go up, the Tez protobuf transport starts to > show up as a potential scalability problem for the task submission and the AM > {code} > public TezTaskRunner2(Configuration tezConf, UserGroupInformation ugi, > String[] localDirs, > ... > this.taskConf = new Configuration(tezConf); > if (taskSpec.getTaskConf() != null) { > Iterator> iter = > taskSpec.getTaskConf().iterator(); > while (iter.hasNext()) { > Entry entry = iter.next(); > taskConf.set(entry.getKey(), entry.getValue()); > } > } > {code} > The TaskSpec getTaskConf() need not include any of the default configs, since > the keys are placed into an existing task conf. > {code} > // Security framework already loaded the tokens into current ugi > DAGProtos.ConfigurationProto confProto = > > TezUtilsInternal.readUserSpecifiedTezConfiguration(System.getenv(Environment.PWD.name())); > TezUtilsInternal.addUserSpecifiedTezConfiguration(defaultConf, > confProto.getConfKeyValuesList()); > UserGroupInformation.setConfiguration(defaultConf); > Credentials credentials = > UserGroupInformation.getCurrentUser().getCredentials(); > {code} > At the very least, the DAG and Vertex do not both need to have the same > configs repeated in them. > !tez-protobuf-writing.png! > + > !tez-am-protobuf-reading.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4073) Configuration: Reduce Vertex and DAG Payload Size
[ https://issues.apache.org/jira/browse/TEZ-4073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16853529#comment-16853529 ] Gopal V commented on TEZ-4073: -- We can also differentiate between the configs set via HiveConf::initialize() && HiveConf::set() using the source variable. > Configuration: Reduce Vertex and DAG Payload Size > - > > Key: TEZ-4073 > URL: https://issues.apache.org/jira/browse/TEZ-4073 > Project: Apache Tez > Issue Type: Bug >Reporter: Gopal V >Priority: Major > Attachments: tez-am-protobuf-reading.png, tez-protobuf-writing.png > > > As the total number of vertices go up, the Tez protobuf transport starts to > show up as a potential scalability problem for the task submission and the AM > {code} > public TezTaskRunner2(Configuration tezConf, UserGroupInformation ugi, > String[] localDirs, > ... > this.taskConf = new Configuration(tezConf); > if (taskSpec.getTaskConf() != null) { > Iterator> iter = > taskSpec.getTaskConf().iterator(); > while (iter.hasNext()) { > Entry entry = iter.next(); > taskConf.set(entry.getKey(), entry.getValue()); > } > } > {code} > The TaskSpec getTaskConf() need not include any of the default configs, since > the keys are placed into an existing task conf. > {code} > // Security framework already loaded the tokens into current ugi > DAGProtos.ConfigurationProto confProto = > > TezUtilsInternal.readUserSpecifiedTezConfiguration(System.getenv(Environment.PWD.name())); > TezUtilsInternal.addUserSpecifiedTezConfiguration(defaultConf, > confProto.getConfKeyValuesList()); > UserGroupInformation.setConfiguration(defaultConf); > Credentials credentials = > UserGroupInformation.getCurrentUser().getCredentials(); > {code} > At the very least, the DAG and Vertex do not both need to have the same > configs repeated in them. > !tez-protobuf-writing.png! > + > !tez-am-protobuf-reading.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4073) Configuration: Reduce Vertex and DAG Payload Size
[ https://issues.apache.org/jira/browse/TEZ-4073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16853602#comment-16853602 ] Gopal V commented on TEZ-4073: -- The HiveConf source annotation, allows for a source filter version of {{public static ByteString createByteStringFromConf(Configuration conf) throws IOException}} > Configuration: Reduce Vertex and DAG Payload Size > - > > Key: TEZ-4073 > URL: https://issues.apache.org/jira/browse/TEZ-4073 > Project: Apache Tez > Issue Type: Bug >Reporter: Gopal V >Priority: Major > Attachments: tez-am-protobuf-reading.png, tez-protobuf-writing.png > > > As the total number of vertices go up, the Tez protobuf transport starts to > show up as a potential scalability problem for the task submission and the AM > {code} > public TezTaskRunner2(Configuration tezConf, UserGroupInformation ugi, > String[] localDirs, > ... > this.taskConf = new Configuration(tezConf); > if (taskSpec.getTaskConf() != null) { > Iterator> iter = > taskSpec.getTaskConf().iterator(); > while (iter.hasNext()) { > Entry entry = iter.next(); > taskConf.set(entry.getKey(), entry.getValue()); > } > } > {code} > The TaskSpec getTaskConf() need not include any of the default configs, since > the keys are placed into an existing task conf. > {code} > // Security framework already loaded the tokens into current ugi > DAGProtos.ConfigurationProto confProto = > > TezUtilsInternal.readUserSpecifiedTezConfiguration(System.getenv(Environment.PWD.name())); > TezUtilsInternal.addUserSpecifiedTezConfiguration(defaultConf, > confProto.getConfKeyValuesList()); > UserGroupInformation.setConfiguration(defaultConf); > Credentials credentials = > UserGroupInformation.getCurrentUser().getCredentials(); > {code} > At the very least, the DAG and Vertex do not both need to have the same > configs repeated in them. > !tez-protobuf-writing.png! > + > !tez-am-protobuf-reading.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)