[jira] [Created] (TEZ-4074) Tez does not run with Hadoop Trunk (3.3.0-snapshot)
Ahmed Hussein created TEZ-4074:
--
Summary: Tez does not run with Hadoop Trunk (3.3.0-snapshot)
Key: TEZ-4074
URL: https://issues.apache.org/jira/browse/TEZ-4074
Project: Apache Tez
Issue Type: Bug
Reporter: Ahmed Hussein
Tez throws a runtime exception when compiled against Hadoop-3.3.0.
With Tez running Guava (11.0.2) and Hadoop eunning Guava 27.0-jre (see
HADOOP-16210), there is an incompatibility of Guava library.
{code:java}
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.136 s
<<< FAILURE! - in org.apache.tez.dag.app.TestSpeculation
[ERROR] org.apache.tez.dag.app.TestSpeculation Time elapsed: 0.136 s <<< ERROR!
java.lang.NoSuchMethodError:
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
at
org.apache.tez.dag.app.TestSpeculation.setupSpeculation(TestSpeculation.java:86)
{code}
It looks like guava added single parameter optimizations which breaks
compatibility with {{VAR_ARGS}}. So, even though it shows source compatibility
it is throwing a runtime error due to binary incompatibility.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-4074) Tez does not run with Hadoop Trunk (3.3.0-snapshot)
[
https://issues.apache.org/jira/browse/TEZ-4074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16853032#comment-16853032
]
Ahmed Hussein commented on TEZ-4074:
Guava27 and 11.0.2 are not source compatible.
For example, Guava27 removed API methods such as
{{Futures.addCallback(ListenableFuture future, FutureCallback
callback)}}
* Guava11.0.2:
[FutureCallback|https://google.github.io/guava/releases/11.0.2/api/docs/com/google/common/util/concurrent/Futures.html#addCallback(com.google.common.util.concurrent.ListenableFuture,%20com.google.common.util.concurrent.FutureCallback)]
* Guava27:
[FutureCallback|https://static.javadoc.io/com.google.guava/guava/27.0.1-jre/com/google/common/util/concurrent/Futures.html#addCallback-com.google.common.util.concurrent.ListenableFuture-com.google.common.util.concurrent.FutureCallback-java.util.concurrent.Executor-]
> Tez does not run with Hadoop Trunk (3.3.0-snapshot)
> ---
>
> Key: TEZ-4074
> URL: https://issues.apache.org/jira/browse/TEZ-4074
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Priority: Major
>
> Tez throws a runtime exception when compiled against Hadoop-3.3.0.
> With Tez running Guava (11.0.2) and Hadoop eunning Guava 27.0-jre (see
> HADOOP-16210), there is an incompatibility of Guava library.
> {code:java}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.136
> s <<< FAILURE! - in org.apache.tez.dag.app.TestSpeculation
> [ERROR] org.apache.tez.dag.app.TestSpeculation Time elapsed: 0.136 s <<<
> ERROR!
> java.lang.NoSuchMethodError:
> com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
> at
> org.apache.tez.dag.app.TestSpeculation.setupSpeculation(TestSpeculation.java:86)
> {code}
> It looks like guava added single parameter optimizations which breaks
> compatibility with {{VAR_ARGS}}. So, even though it shows source
> compatibility it is throwing a runtime error due to binary incompatibility.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-4073) Configuration: Reduce Vertex and DAG Payload Size
[
https://issues.apache.org/jira/browse/TEZ-4073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16853527#comment-16853527
]
Gopal V commented on TEZ-4073:
--
The trivial version of this is to skip all the configs which have
*.xml when writing this out, since the all the XML files have
already been parsed + sent for the AM init.
> Configuration: Reduce Vertex and DAG Payload Size
> -
>
> Key: TEZ-4073
> URL: https://issues.apache.org/jira/browse/TEZ-4073
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Gopal V
>Priority: Major
> Attachments: tez-am-protobuf-reading.png, tez-protobuf-writing.png
>
>
> As the total number of vertices go up, the Tez protobuf transport starts to
> show up as a potential scalability problem for the task submission and the AM
> {code}
> public TezTaskRunner2(Configuration tezConf, UserGroupInformation ugi,
> String[] localDirs,
> ...
> this.taskConf = new Configuration(tezConf);
> if (taskSpec.getTaskConf() != null) {
> Iterator> iter =
> taskSpec.getTaskConf().iterator();
> while (iter.hasNext()) {
> Entry entry = iter.next();
> taskConf.set(entry.getKey(), entry.getValue());
> }
> }
> {code}
> The TaskSpec getTaskConf() need not include any of the default configs, since
> the keys are placed into an existing task conf.
> {code}
> // Security framework already loaded the tokens into current ugi
> DAGProtos.ConfigurationProto confProto =
>
> TezUtilsInternal.readUserSpecifiedTezConfiguration(System.getenv(Environment.PWD.name()));
> TezUtilsInternal.addUserSpecifiedTezConfiguration(defaultConf,
> confProto.getConfKeyValuesList());
> UserGroupInformation.setConfiguration(defaultConf);
> Credentials credentials =
> UserGroupInformation.getCurrentUser().getCredentials();
> {code}
> At the very least, the DAG and Vertex do not both need to have the same
> configs repeated in them.
> !tez-protobuf-writing.png!
> +
> !tez-am-protobuf-reading.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-4073) Configuration: Reduce Vertex and DAG Payload Size
[
https://issues.apache.org/jira/browse/TEZ-4073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16853529#comment-16853529
]
Gopal V commented on TEZ-4073:
--
We can also differentiate between the configs set via HiveConf::initialize() &&
HiveConf::set() using the source variable.
> Configuration: Reduce Vertex and DAG Payload Size
> -
>
> Key: TEZ-4073
> URL: https://issues.apache.org/jira/browse/TEZ-4073
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Gopal V
>Priority: Major
> Attachments: tez-am-protobuf-reading.png, tez-protobuf-writing.png
>
>
> As the total number of vertices go up, the Tez protobuf transport starts to
> show up as a potential scalability problem for the task submission and the AM
> {code}
> public TezTaskRunner2(Configuration tezConf, UserGroupInformation ugi,
> String[] localDirs,
> ...
> this.taskConf = new Configuration(tezConf);
> if (taskSpec.getTaskConf() != null) {
> Iterator> iter =
> taskSpec.getTaskConf().iterator();
> while (iter.hasNext()) {
> Entry entry = iter.next();
> taskConf.set(entry.getKey(), entry.getValue());
> }
> }
> {code}
> The TaskSpec getTaskConf() need not include any of the default configs, since
> the keys are placed into an existing task conf.
> {code}
> // Security framework already loaded the tokens into current ugi
> DAGProtos.ConfigurationProto confProto =
>
> TezUtilsInternal.readUserSpecifiedTezConfiguration(System.getenv(Environment.PWD.name()));
> TezUtilsInternal.addUserSpecifiedTezConfiguration(defaultConf,
> confProto.getConfKeyValuesList());
> UserGroupInformation.setConfiguration(defaultConf);
> Credentials credentials =
> UserGroupInformation.getCurrentUser().getCredentials();
> {code}
> At the very least, the DAG and Vertex do not both need to have the same
> configs repeated in them.
> !tez-protobuf-writing.png!
> +
> !tez-am-protobuf-reading.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-4073) Configuration: Reduce Vertex and DAG Payload Size
[
https://issues.apache.org/jira/browse/TEZ-4073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16853602#comment-16853602
]
Gopal V commented on TEZ-4073:
--
The HiveConf source annotation, allows for a source filter version of {{public
static ByteString createByteStringFromConf(Configuration conf) throws
IOException}}
> Configuration: Reduce Vertex and DAG Payload Size
> -
>
> Key: TEZ-4073
> URL: https://issues.apache.org/jira/browse/TEZ-4073
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Gopal V
>Priority: Major
> Attachments: tez-am-protobuf-reading.png, tez-protobuf-writing.png
>
>
> As the total number of vertices go up, the Tez protobuf transport starts to
> show up as a potential scalability problem for the task submission and the AM
> {code}
> public TezTaskRunner2(Configuration tezConf, UserGroupInformation ugi,
> String[] localDirs,
> ...
> this.taskConf = new Configuration(tezConf);
> if (taskSpec.getTaskConf() != null) {
> Iterator> iter =
> taskSpec.getTaskConf().iterator();
> while (iter.hasNext()) {
> Entry entry = iter.next();
> taskConf.set(entry.getKey(), entry.getValue());
> }
> }
> {code}
> The TaskSpec getTaskConf() need not include any of the default configs, since
> the keys are placed into an existing task conf.
> {code}
> // Security framework already loaded the tokens into current ugi
> DAGProtos.ConfigurationProto confProto =
>
> TezUtilsInternal.readUserSpecifiedTezConfiguration(System.getenv(Environment.PWD.name()));
> TezUtilsInternal.addUserSpecifiedTezConfiguration(defaultConf,
> confProto.getConfKeyValuesList());
> UserGroupInformation.setConfiguration(defaultConf);
> Credentials credentials =
> UserGroupInformation.getCurrentUser().getCredentials();
> {code}
> At the very least, the DAG and Vertex do not both need to have the same
> configs repeated in them.
> !tez-protobuf-writing.png!
> +
> !tez-am-protobuf-reading.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
