[jira] [Commented] (TEZ-2218) Turn on speculation by default
[ https://issues.apache.org/jira/browse/TEZ-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491428#comment-16491428 ] Sergey Shelukhin commented on TEZ-2218: --- Hmm... > Turn on speculation by default > -- > > Key: TEZ-2218 > URL: https://issues.apache.org/jira/browse/TEZ-2218 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Bikas Saha >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-2132) Support fault tolerance & speculation in pipelined data transfer for ordered output
[ https://issues.apache.org/jira/browse/TEZ-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491425#comment-16491425 ] Sergey Shelukhin commented on TEZ-2132: --- hmm... > Support fault tolerance & speculation in pipelined data transfer for ordered > output > --- > > Key: TEZ-2132 > URL: https://issues.apache.org/jira/browse/TEZ-2132 > Project: Apache Tez > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Major > > Follow up of TEZ-2001. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3943) TezClient leaks DAGClient for prewarm
[ https://issues.apache.org/jira/browse/TEZ-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491395#comment-16491395 ] Sergey Shelukhin commented on TEZ-3943: --- Looks like testshiffle is pretty unstable. https://builds.apache.org/job/PreCommit-TEZ-Build/2815/testReport/org.apache.tez.test/TestSecureShuffle/testSecureShuffle_test_sslInCluster_true__resultWithTezSSL_0__resultWithoutTezSSL_1__asyncHttp_false__/history/ The other test also fails often: https://builds.apache.org/job/PreCommit-TEZ-Build/2815/testReport/org.apache.tez.test/TestAMRecovery/testVertexCompletelyFinished_Broadcast/history/ [~ashutoshc] [~hagleitn] can you commit? [~ewohlstadter] reviewed above, but neither of us is a committer > TezClient leaks DAGClient for prewarm > - > > Key: TEZ-3943 > URL: https://issues.apache.org/jira/browse/TEZ-3943 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: TEZ-3943.01.patch, TEZ-3943.02.patch, TEZ-3943.patch > > > This may in turn leak some security related threads. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Failed: TEZ-3943 PreCommit Build #2815
Jira: https://issues.apache.org/jira/browse/TEZ-3943
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2815/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 346.82 KB...]
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :tez-tests
[INFO] Build failures were ignored.
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12925209/TEZ-3943.02.patch
against master revision f6624c1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestSecureShuffle
org.apache.tez.test.TestAMRecovery
The following test timeouts occurred in :
org.apache.tez.tests.TestExternalTezServices
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2815//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2815//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
5 tests failed.
FAILED:
org.apache.tez.test.TestAMRecovery.testVertexCompletelyFinished_Broadcast
Error Message:
test timed out after 12 milliseconds
Stack Trace:
java.lang.Exception: test timed out after 12 milliseconds
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.hadoop.ipc.Client.call(Client.java:1466)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy98.getApplicationReport(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:191)
at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy99.getApplicationReport(Unknown Source)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:430)
at
org.apache.tez.client.TezYarnClient.getApplicationReport(TezYarnClient.java:94)
at
org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.getAppReport(DAGClientRPCImpl.java:245)
at
org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.createAMProxyIfNeeded(DAGClientRPCImpl.java:270)
at
org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.getDAGStatus(DAGClientRPCImpl.java:95)
at
org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatusViaAM(DAGClientImpl.java:371)
at
org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatusInternal(DAGClientImpl.java:221)
at
org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatus(DAGClientImpl.java:208)
at
[jira] [Commented] (TEZ-3943) TezClient leaks DAGClient for prewarm
[
https://issues.apache.org/jira/browse/TEZ-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491382#comment-16491382
]
TezQA commented on TEZ-3943:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12925209/TEZ-3943.02.patch
against master revision f6624c1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestSecureShuffle
org.apache.tez.test.TestAMRecovery
The following test timeouts occurred in :
org.apache.tez.tests.TestExternalTezServices
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2815//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2815//console
This message is automatically generated.
> TezClient leaks DAGClient for prewarm
> -
>
> Key: TEZ-3943
> URL: https://issues.apache.org/jira/browse/TEZ-3943
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: TEZ-3943.01.patch, TEZ-3943.02.patch, TEZ-3943.patch
>
>
> This may in turn leak some security related threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-3899) Update Tez build instructions for Apache Hadoop 2.8 and later
[
https://issues.apache.org/jira/browse/TEZ-3899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Dash updated TEZ-3899:
--
Attachment: TEZ-3899.001.patch
> Update Tez build instructions for Apache Hadoop 2.8 and later
> -
>
> Key: TEZ-3899
> URL: https://issues.apache.org/jira/browse/TEZ-3899
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Arpit Agarwal
>Assignee: Nishant Dash
>Priority: Major
> Attachments: TEZ-3899.001.patch
>
>
> Building Tez with Apache Hadoop 2.8 or later requires the new {{hadoop28}}
> profile. e.g.
> {code}
> mvn clean package -DskipTests=true -Dmaven.javadoc.skip=true -Phadoop28
> {code}
> The Tez install guide should be updated to mention this.
> Also INSTALL.md should be updated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-3943) TezClient leaks DAGClient for prewarm
[ https://issues.apache.org/jira/browse/TEZ-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated TEZ-3943: -- Attachment: TEZ-3943.02.patch > TezClient leaks DAGClient for prewarm > - > > Key: TEZ-3943 > URL: https://issues.apache.org/jira/browse/TEZ-3943 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: TEZ-3943.01.patch, TEZ-3943.02.patch, TEZ-3943.patch > > > This may in turn leak some security related threads. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3943) TezClient leaks DAGClient for prewarm
[
https://issues.apache.org/jira/browse/TEZ-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491260#comment-16491260
]
TezQA commented on TEZ-3943:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12925207/TEZ-3943.01.patch
against master revision f6624c1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to cause Findbugs
(version ) to fail.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The test build failed in
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2814//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2814//console
This message is automatically generated.
> TezClient leaks DAGClient for prewarm
> -
>
> Key: TEZ-3943
> URL: https://issues.apache.org/jira/browse/TEZ-3943
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: TEZ-3943.01.patch, TEZ-3943.patch
>
>
> This may in turn leak some security related threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Failed: TEZ-3943 PreCommit Build #2814
Jira: https://issues.apache.org/jira/browse/TEZ-3943
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2814/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 115.39 KB...]
==
Running tests
/home/jenkins/tools/maven/latest/bin/mvn clean install -fn -DTezPatchProcess
/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build@2/build-tools/test-patch.sh:
line 611:
/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build@2/../patchprocess/testrun.txt:
No such file or directory
cat:
/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build@2/../patchprocess/testrun.txt:
No such file or directory
awk: cannot open
/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build@2/../patchprocess/testrun.txt
(No such file or directory)
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12925207/TEZ-3943.01.patch
against master revision f6624c1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to cause Findbugs
(version ) to fail.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The test build failed in
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2814//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2814//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files
were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
No tests ran.
Failed: TEZ-3923 PreCommit Build #2813
Jira: https://issues.apache.org/jira/browse/TEZ-3923
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2813/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 325.13 KB...]
[ERROR] [Help 2]
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :tez-dag
[INFO] Build failures were ignored.
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12925194/TEZ-3923.002.patch
against master revision f6624c1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color}. The applied patch generated 174 javac
compiler warnings (more than the master's current 26 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to cause Findbugs
(version 3.0.1) to fail.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in :
org.apache.tez.dag.app.rm.TestTaskScheduler
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2813//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/2813//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2813//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-3923) Move master to Hadoop 3+ and create separate 0.9.x line
[
https://issues.apache.org/jira/browse/TEZ-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491257#comment-16491257
]
TezQA commented on TEZ-3923:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12925194/TEZ-3923.002.patch
against master revision f6624c1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color}. The applied patch generated 174 javac
compiler warnings (more than the master's current 26 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to cause Findbugs
(version 3.0.1) to fail.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in :
org.apache.tez.dag.app.rm.TestTaskScheduler
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/2813//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/2813//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2813//console
This message is automatically generated.
> Move master to Hadoop 3+ and create separate 0.9.x line
> ---
>
> Key: TEZ-3923
> URL: https://issues.apache.org/jira/browse/TEZ-3923
> Project: Apache Tez
> Issue Type: Task
>Reporter: Eric Wohlstadter
>Assignee: Jonathan Eagles
>Priority: Major
> Attachments: TEZ-3923.001.patch, TEZ-3923.002.patch
>
>
> Move master to support minimum Hadoop 3+ (0.10.x line) and create separate
> branch for Hadoop 2 (0.9.x line)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3943) TezClient leaks DAGClient for prewarm
[ https://issues.apache.org/jira/browse/TEZ-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491243#comment-16491243 ] Sergey Shelukhin commented on TEZ-3943: --- Fixed some client test. The other ones pass for me. > TezClient leaks DAGClient for prewarm > - > > Key: TEZ-3943 > URL: https://issues.apache.org/jira/browse/TEZ-3943 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: TEZ-3943.01.patch, TEZ-3943.patch > > > This may in turn leak some security related threads. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3943) TezClient leaks DAGClient for prewarm
[ https://issues.apache.org/jira/browse/TEZ-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated TEZ-3943: -- Attachment: TEZ-3943.01.patch > TezClient leaks DAGClient for prewarm > - > > Key: TEZ-3943 > URL: https://issues.apache.org/jira/browse/TEZ-3943 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: TEZ-3943.01.patch, TEZ-3943.patch > > > This may in turn leak some security related threads. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3942) RPC getTask writable optimization invalid in hadoop 2.8+
[
https://issues.apache.org/jira/browse/TEZ-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated TEZ-3942:
-
Attachment: TEZ-3942.001.patch
> RPC getTask writable optimization invalid in hadoop 2.8+
>
>
> Key: TEZ-3942
> URL: https://issues.apache.org/jira/browse/TEZ-3942
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Priority: Major
> Attachments: TEZ-3942.001.patch, TEZ-3942.test.patch
>
>
> TEZ-3140 added an optimization to improve performance of RPC writable.
> HADOOP-13426 added in hadoop 2.8 has invalidated the assumption of the added
> optimization by changing the underlying output buffer.
> {noformat}
> "IPC Server handler 25 on 35274" #85 daemon prio=5 os_prio=0
> tid=0x022c nid=0x1b40f runnable [0x2ba1a6627000]
>java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:3236)
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
> at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
> - locked <0x00072fe9ac68> (a
> org.apache.hadoop.ipc.ResponseBuffer$FramedBuffer)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> - locked <0x00072fe9ac48> (a org.apache.hadoop.ipc.ResponseBuffer)
> at
> org.apache.tez.dag.api.EntityDescriptor.write(EntityDescriptor.java:121)
> at org.apache.tez.runtime.api.impl.InputSpec.write(InputSpec.java:66)
> at org.apache.tez.runtime.api.impl.TaskSpec.write(TaskSpec.java:174)
> at org.apache.tez.common.ContainerTask.write(ContainerTask.java:77)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:202)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:128)
> at org.apache.hadoop.io.ObjectWritable.write(ObjectWritable.java:82)
> at
> org.apache.hadoop.ipc.RpcWritable$WritableWrapper.writeTo(RpcWritable.java:75)
> at
> org.apache.hadoop.ipc.Server.setupResponseForWritable(Server.java:2807)
> at org.apache.hadoop.ipc.Server.setupResponse(Server.java:2792)
> at org.apache.hadoop.ipc.Server.setupResponse(Server.java:2766)
> at org.apache.hadoop.ipc.Server.access$100(Server.java:138)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:905)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:810)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1949)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2523)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-3942) RPC getTask writable optimization invalid in hadoop 2.8+
[
https://issues.apache.org/jira/browse/TEZ-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated TEZ-3942:
-
Attachment: TEZ-3942.test.patch
> RPC getTask writable optimization invalid in hadoop 2.8+
>
>
> Key: TEZ-3942
> URL: https://issues.apache.org/jira/browse/TEZ-3942
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Priority: Major
> Attachments: TEZ-3942.001.patch, TEZ-3942.test.patch
>
>
> TEZ-3140 added an optimization to improve performance of RPC writable.
> HADOOP-13426 added in hadoop 2.8 has invalidated the assumption of the added
> optimization by changing the underlying output buffer.
> {noformat}
> "IPC Server handler 25 on 35274" #85 daemon prio=5 os_prio=0
> tid=0x022c nid=0x1b40f runnable [0x2ba1a6627000]
>java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:3236)
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
> at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
> - locked <0x00072fe9ac68> (a
> org.apache.hadoop.ipc.ResponseBuffer$FramedBuffer)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> - locked <0x00072fe9ac48> (a org.apache.hadoop.ipc.ResponseBuffer)
> at
> org.apache.tez.dag.api.EntityDescriptor.write(EntityDescriptor.java:121)
> at org.apache.tez.runtime.api.impl.InputSpec.write(InputSpec.java:66)
> at org.apache.tez.runtime.api.impl.TaskSpec.write(TaskSpec.java:174)
> at org.apache.tez.common.ContainerTask.write(ContainerTask.java:77)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:202)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:128)
> at org.apache.hadoop.io.ObjectWritable.write(ObjectWritable.java:82)
> at
> org.apache.hadoop.ipc.RpcWritable$WritableWrapper.writeTo(RpcWritable.java:75)
> at
> org.apache.hadoop.ipc.Server.setupResponseForWritable(Server.java:2807)
> at org.apache.hadoop.ipc.Server.setupResponse(Server.java:2792)
> at org.apache.hadoop.ipc.Server.setupResponse(Server.java:2766)
> at org.apache.hadoop.ipc.Server.access$100(Server.java:138)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:905)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:810)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1949)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2523)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-3923) Move master to Hadoop 3+ and create separate 0.9.x line
[ https://issues.apache.org/jira/browse/TEZ-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-3923: -- Attachment: TEZ-3923.002.patch > Move master to Hadoop 3+ and create separate 0.9.x line > --- > > Key: TEZ-3923 > URL: https://issues.apache.org/jira/browse/TEZ-3923 > Project: Apache Tez > Issue Type: Task >Reporter: Eric Wohlstadter >Assignee: Jonathan Eagles >Priority: Major > Attachments: TEZ-3923.001.patch, TEZ-3923.002.patch > > > Move master to support minimum Hadoop 3+ (0.10.x line) and create separate > branch for Hadoop 2 (0.9.x line) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3923) Move master to Hadoop 3+ and create separate 0.9.x line
[
https://issues.apache.org/jira/browse/TEZ-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491162#comment-16491162
]
Eric Wohlstadter commented on TEZ-3923:
---
[~jeagles] [~kshukla] [~jlowe]
I'm only seeing one test failure now against Hadoop3.
It seems like it may just be the environment where I'm running the tests. I'm
going to try resubmitting [~jeagles] patch on this Jira to get new test results
on the Apache infra.
Does anyone know what might be causing this one for me? I will look into it,
but I wanted to see if someone might already have context about it.
{code:java}
Tests run: 13, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.758 sec <<<
FAILURE!
testMapFileAccess(org.apache.tez.auxservices.TestShuffleHandler) Time elapsed:
0.064 sec <<< ERROR!
java.lang.IllegalArgumentException: Can't get Kerberos realm
at sun.security.krb5.Config.getDefaultRealm(Config.java:1029)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:110)
at
org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:63)
at
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:332)
at
org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:387)
at
org.apache.tez.auxservices.TestShuffleHandler.testMapFileAccess(TestShuffleHandler.java:675){code}
> Move master to Hadoop 3+ and create separate 0.9.x line
> ---
>
> Key: TEZ-3923
> URL: https://issues.apache.org/jira/browse/TEZ-3923
> Project: Apache Tez
> Issue Type: Task
>Reporter: Eric Wohlstadter
>Assignee: Jonathan Eagles
>Priority: Major
> Attachments: TEZ-3923.001.patch
>
>
> Move master to support minimum Hadoop 3+ (0.10.x line) and create separate
> branch for Hadoop 2 (0.9.x line)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
