[jira] [Updated] (TEZ-3144) Evaluate a lower overhead collection to store large collections
[ https://issues.apache.org/jira/browse/TEZ-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-3144: - Attachment: TEZ-3144.1-branch-0.7.patch > Evaluate a lower overhead collection to store large collections > --- > > Key: TEZ-3144 > URL: https://issues.apache.org/jira/browse/TEZ-3144 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: TEZ-3144.1-branch-0.7.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3151) expose DAG credentials to plugins
[
https://issues.apache.org/jira/browse/TEZ-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173037#comment-15173037
]
TezQA commented on TEZ-3151:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12790603/TEZ-3151.01.patch
against master revision e9e9db1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.app.rm.TestContainerReuse
org.apache.tez.test.TestFaultTolerance
The following test timeouts occurred in :
org.apache.tez.test.TestRecovery
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1534//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1534//console
This message is automatically generated.
> expose DAG credentials to plugins
> -
>
> Key: TEZ-3151
> URL: https://issues.apache.org/jira/browse/TEZ-3151
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.8.3
>
> Attachments: TEZ-3151.01.patch, TEZ-3151.patch
>
>
> Tez plugins need to pass credentials (e.g. HBase tokens, etc.) to tasks.
> Right now they only have access to AM credentials.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Failed: TEZ-3151 PreCommit Build #1534
Jira: https://issues.apache.org/jira/browse/TEZ-3151
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1534/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4633 lines...]
[ERROR] mvn -rf :tez-dag
[INFO] Build failures were ignored.
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12790603/TEZ-3151.01.patch
against master revision e9e9db1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.app.rm.TestContainerReuse
org.apache.tez.test.TestFaultTolerance
The following test timeouts occurred in :
org.apache.tez.test.TestRecovery
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1534//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1534//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
1508aca706ee96c878e02c06b510bdd8aafc416d logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
8 tests failed.
FAILED:
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources
Error Message:
Wanted but not invoked:
taskSchedulerManagerForTest.taskAllocated(
0,
Mock for TA attempt_0_0001_0_01_04_1,
,
Container: [ContainerId: container_1_0001_01_01, NodeId: host1:0,
NodeHttpAddress: host1:0, Resource: , Priority: 1,
Token: null, ]
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1272)
However, there were other interactions with this mock:
taskSchedulerManagerForTest.init(
Configuration: core-default.xml, core-site.xml, yarn-default.xml,
yarn-site.xml
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1143)
taskSchedulerManagerForTest.setConfig(
Configuration: core-default.xml, core-site.xml, yarn-default.xml,
yarn-site.xml
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1143)
taskSchedulerManagerForTest.serviceInit(
Configuration: core-default.xml, core-site.xml, yarn-default.xml,
yarn-site.xml
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1143)
taskSchedulerManagerForTest.start();
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1144)
taskSchedulerManagerForTest.serviceStart();
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1144)
taskSchedulerManagerForTest.instantiateSchedulers(
"host",
0,
"",
Mock for AppContext, hashCode: 469698423
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1144)
taskSchedulerManagerForTest.getContainerSignatureMatcher();
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1144)
taskSchedulerManagerForTest.getConfig();
-> at
[jira] [Commented] (TEZ-2863) Container, node, and logs not available in UI for tasks that fail to launch
[
https://issues.apache.org/jira/browse/TEZ-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173002#comment-15173002
]
Jeff Zhang commented on TEZ-2863:
-
[~jeagles] Just quick go through your new patch, seems you miss the change on
HistoryEvents.proto. You need to add new fields containerId, nodeHttpAddress
and containerNodeId in HistoryEvents.proto. And also adding code for
serialization/deserialization in TaskAttemptFinishedEvent#toProto and fromProto.
> Container, node, and logs not available in UI for tasks that fail to launch
> ---
>
> Key: TEZ-2863
> URL: https://issues.apache.org/jira/browse/TEZ-2863
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-2863.1.patch, TEZ-2863.2-branch-0.7.patch,
> TEZ-2863.2.patch, TEZ-2863.3-branch-0.7.patch,
> TEZ-2863.3-branch-0.7.patch.addendum, TEZ-2863.3.patch,
> TEZ-2863.3.patch.addendum, TEZ-2863.4-branch-0.7.patch, TEZ-2863.4.patch
>
>
> While running a sample tez job
> {noformat}
> tez-examples-*.jar orderedwordcount -Dtez.task.resource.memory.mb=1
> -Dtez.task.launch.cmd-opts="-Xmx1m" input output
> {noformat}
> It was noticed that the Tez UI task attempt
> http://timelineserverhost:port/ws/v1/timeline/TEZ_TASK_ATTEMPT_ID/attempt_id
> was missing the TEZ_ATTEMPT_STARTED event
> {noformat}
> 2015-10-01 10:03:55,344 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1443711816411_0001_1][Event:TASK_STARTED]:
> vertexName=Tokenizer, taskId=task_1443711816411_0001_1_00_00,
> scheduledTime=1443711835342, launchTime=1443711835342
> 2015-10-01 10:03:55,346 [INFO] [Dispatcher thread {Central}]
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:55,356 [INFO] [TaskSchedulerEventHandlerThread]
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:55,364 [INFO] [TaskSchedulerEventHandlerThread]
> |rm.YarnTaskSchedulerService|: Allocation request for task:
> attempt_1443711816411_0001_1_00_00_0 with request: Capability[ vCores:1>]Priority[2] host: localhost rack: null
> 2015-10-01 10:03:56,639 [INFO] [AMRM Heartbeater thread]
> |impl.AMRMClientImpl|: Received new token for : localhost:57381
> 2015-10-01 10:03:56,646 [INFO] [AMRM Callback Handler Thread]
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:56,648 [INFO] [DelayedContainerManager]
> |rm.YarnTaskSchedulerService|: Assigning container to task:
> containerId=container_1443711816411_0001_01_02,
> task=attempt_1443711816411_0001_1_00_00_0, containerHost=localhost:57381,
> containerPriority= 2, containerResources=,
> localityMatchType=NodeLocal, matchedLocation=localhost,
> honorLocalityFlags=true, reusedContainer=false, delayedContainers=0
> 2015-10-01 10:03:56,649 [INFO] [DelayedContainerManager] |util.RackResolver|:
> Resolved localhost to /default-rack
> 2015-10-01 10:03:56,649 [INFO] [DelayedContainerManager] |util.RackResolver|:
> Resolved localhost to /default-rack
> 2015-10-01 10:03:56,686 [INFO] [TaskSchedulerAppCaller #0]
> |node.AMNodeTracker|: Adding new node: localhost:57381
> 2015-10-01 10:03:56,700 [INFO] [ContainerLauncher #0]
> |launcher.ContainerLauncherImpl|: Launching
> container_1443711816411_0001_01_02
> 2015-10-01 10:03:56,700 [INFO] [ContainerLauncher #0]
> |impl.ContainerManagementProtocolProxy|: Opening proxy : localhost:57381
> 2015-10-01 10:03:56,741 [INFO] [ContainerLauncher #0]
> |history.HistoryEventHandler|: [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]:
> containerId=container_1443711816411_0001_01_02, launchTime=1443711836741
> 2015-10-01 10:03:57,647 [INFO] [AMRM Callback Handler Thread]
> |rm.YarnTaskSchedulerService|: Allocated container
> completed:container_1443711816411_0001_01_02 last allocated to task:
> attempt_1443711816411_0001_1_00_00_0
> 2015-10-01 10:03:57,648 [INFO] [Dispatcher thread {Central}]
> |container.AMContainerImpl|: Container container_1443711816411_0001_01_02
> exited with diagnostics set to Container failed, exitCode=1. Exception from
> container-launch.
> Container id: container_1443711816411_0001_01_02
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at
Failed: TEZ-3151 PreCommit Build #1533
Jira: https://issues.apache.org/jira/browse/TEZ-3151
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1533/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4387 lines...]
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :tez-tests
[INFO] Build failures were ignored.
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12790566/TEZ-3151.patch
against master revision e9e9db1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1533//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1533//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
6fd3e19b008a95f2cd43ef92f6129e052de8300c logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
6 tests failed.
FAILED: org.apache.tez.test.TestFaultTolerance.testBasicInputFailureWithExit
Error Message:
TezSession has already shutdown. No cluster diagnostics found.
Stack Trace:
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No
cluster diagnostics found.
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:784)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:129)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:124)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:120)
at
org.apache.tez.test.TestFaultTolerance.testBasicInputFailureWithExit(TestFaultTolerance.java:261)
FAILED:
org.apache.tez.test.TestFaultTolerance.testInputFailureRerunCanSendOutputToTwoDownstreamVertices
Error Message:
TezSession has already shutdown. No cluster diagnostics found.
Stack Trace:
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No
cluster diagnostics found.
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:784)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:129)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:124)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:120)
at
org.apache.tez.test.TestFaultTolerance.testInputFailureRerunCanSendOutputToTwoDownstreamVertices(TestFaultTolerance.java:703)
FAILED:
org.apache.tez.test.TestFaultTolerance.testMultipleInputFailureWithoutExit
Error Message:
TezSession has already shutdown. No cluster diagnostics found.
Stack Trace:
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No
cluster diagnostics found.
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:784)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:129)
at
[jira] [Commented] (TEZ-3151) expose DAG credentials to plugins
[
https://issues.apache.org/jira/browse/TEZ-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172920#comment-15172920
]
TezQA commented on TEZ-3151:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12790566/TEZ-3151.patch
against master revision e9e9db1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1533//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1533//console
This message is automatically generated.
> expose DAG credentials to plugins
> -
>
> Key: TEZ-3151
> URL: https://issues.apache.org/jira/browse/TEZ-3151
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: TEZ-3151.01.patch, TEZ-3151.patch
>
>
> Tez plugins need to pass credentials (e.g. HBase tokens, etc.) to tasks.
> Right now they only have access to AM credentials.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3151) expose DAG credentials to plugins
[ https://issues.apache.org/jira/browse/TEZ-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172858#comment-15172858 ] Sergey Shelukhin commented on TEZ-3151: --- Yeah, I put the updated context in the description; there was no description before that. > expose DAG credentials to plugins > - > > Key: TEZ-3151 > URL: https://issues.apache.org/jira/browse/TEZ-3151 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: TEZ-3151.01.patch, TEZ-3151.patch > > > Tez plugins need to pass credentials (e.g. HBase tokens, etc.) to tasks. > Right now they only have access to AM credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3151) expose DAG credentials to plugins
[ https://issues.apache.org/jira/browse/TEZ-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172855#comment-15172855 ] Siddharth Seth commented on TEZ-3151: - +1. Looks good. Waiting for jenkins before committing. [~hitesh] - more context around this. The TaskCommunicator / Launcher needs access to tokens for the specific dag. However, the only thing exposed at the moment is the AM credentials. > expose DAG credentials to plugins > - > > Key: TEZ-3151 > URL: https://issues.apache.org/jira/browse/TEZ-3151 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: TEZ-3151.01.patch, TEZ-3151.patch > > > Tez plugins need to pass credentials (e.g. HBase tokens, etc.) to tasks. > Right now they only have access to AM credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3151) expose DAG credentials to plugins
[ https://issues.apache.org/jira/browse/TEZ-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated TEZ-3151: -- Attachment: TEZ-3151.01.patch Updated. > expose DAG credentials to plugins > - > > Key: TEZ-3151 > URL: https://issues.apache.org/jira/browse/TEZ-3151 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: TEZ-3151.01.patch, TEZ-3151.patch > > > Tez plugins need to pass credentials (e.g. HBase tokens, etc.) to tasks. > Right now they only have access to AM credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2863) Container, node, and logs not available in UI for tasks that fail to launch
[
https://issues.apache.org/jira/browse/TEZ-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172828#comment-15172828
]
TezQA commented on TEZ-2863:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12790565/TEZ-2863.4.patch
against master revision dd79190.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 33 javac
compiler warnings (more than the master's current 31 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in :
org.apache.tez.test.TestRecovery
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1532//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1532//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1532//console
This message is automatically generated.
> Container, node, and logs not available in UI for tasks that fail to launch
> ---
>
> Key: TEZ-2863
> URL: https://issues.apache.org/jira/browse/TEZ-2863
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-2863.1.patch, TEZ-2863.2-branch-0.7.patch,
> TEZ-2863.2.patch, TEZ-2863.3-branch-0.7.patch,
> TEZ-2863.3-branch-0.7.patch.addendum, TEZ-2863.3.patch,
> TEZ-2863.3.patch.addendum, TEZ-2863.4-branch-0.7.patch, TEZ-2863.4.patch
>
>
> While running a sample tez job
> {noformat}
> tez-examples-*.jar orderedwordcount -Dtez.task.resource.memory.mb=1
> -Dtez.task.launch.cmd-opts="-Xmx1m" input output
> {noformat}
> It was noticed that the Tez UI task attempt
> http://timelineserverhost:port/ws/v1/timeline/TEZ_TASK_ATTEMPT_ID/attempt_id
> was missing the TEZ_ATTEMPT_STARTED event
> {noformat}
> 2015-10-01 10:03:55,344 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1443711816411_0001_1][Event:TASK_STARTED]:
> vertexName=Tokenizer, taskId=task_1443711816411_0001_1_00_00,
> scheduledTime=1443711835342, launchTime=1443711835342
> 2015-10-01 10:03:55,346 [INFO] [Dispatcher thread {Central}]
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:55,356 [INFO] [TaskSchedulerEventHandlerThread]
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:55,364 [INFO] [TaskSchedulerEventHandlerThread]
> |rm.YarnTaskSchedulerService|: Allocation request for task:
> attempt_1443711816411_0001_1_00_00_0 with request: Capability[ vCores:1>]Priority[2] host: localhost rack: null
> 2015-10-01 10:03:56,639 [INFO] [AMRM Heartbeater thread]
> |impl.AMRMClientImpl|: Received new token for : localhost:57381
> 2015-10-01 10:03:56,646 [INFO] [AMRM Callback Handler Thread]
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:56,648 [INFO] [DelayedContainerManager]
> |rm.YarnTaskSchedulerService|: Assigning container to task:
> containerId=container_1443711816411_0001_01_02,
> task=attempt_1443711816411_0001_1_00_00_0, containerHost=localhost:57381,
> containerPriority= 2, containerResources=,
> localityMatchType=NodeLocal, matchedLocation=localhost,
> honorLocalityFlags=true, reusedContainer=false, delayedContainers=0
> 2015-10-01 10:03:56,649 [INFO] [DelayedContainerManager] |util.RackResolver|:
> Resolved localhost to /default-rack
> 2015-10-01 10:03:56,649 [INFO] [DelayedContainerManager] |util.RackResolver|:
> Resolved localhost to /default-rack
> 2015-10-01 10:03:56,686 [INFO] [TaskSchedulerAppCaller #0]
> |node.AMNodeTracker|: Adding new node: localhost:57381
> 2015-10-01 10:03:56,700 [INFO] [ContainerLauncher #0]
> |launcher.ContainerLauncherImpl|: Launching
> container_1443711816411_0001_01_02
> 2015-10-01 10:03:56,700 [INFO] [ContainerLauncher #0]
> |impl.ContainerManagementProtocolProxy|: Opening proxy : localhost:57381
> 2015-10-01 10:03:56,741 [INFO] [ContainerLauncher #0]
> |history.HistoryEventHandler|: [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]:
> containerId=container_1443711816411_0001_01_02, launchTime=1443711836741
> 2015-10-01 10:03:57,647 [INFO] [AMRM Callback Handler Thread]
> |rm.YarnTaskSchedulerService|: Allocated container
>
Failed: TEZ-2863 PreCommit Build #1532
Jira: https://issues.apache.org/jira/browse/TEZ-2863
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1532/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4337 lines...]
[ERROR] [Help 2]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :tez-tests
[INFO] Build failures were ignored.
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12790565/TEZ-2863.4.patch
against master revision dd79190.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 33 javac
compiler warnings (more than the master's current 31 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in :
org.apache.tez.test.TestRecovery
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1532//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1532//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1532//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
5f12cfcc4fec03e59ffebacbc48546920037188e logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Updated] (TEZ-3149) Tez-tools: Add username in DagInfo
[ https://issues.apache.org/jira/browse/TEZ-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-3149: -- Fix Version/s: 0.8.3 > Tez-tools: Add username in DagInfo > -- > > Key: TEZ-3149 > URL: https://issues.apache.org/jira/browse/TEZ-3149 > Project: Apache Tez > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan > Fix For: 0.7.1, 0.8.3 > > Attachments: TEZ-3149.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3151) expose DAG credentials to plugins
[ https://issues.apache.org/jira/browse/TEZ-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172714#comment-15172714 ] Siddharth Seth commented on TEZ-3151: - [~sershe] - can we add a getCredentials api into DagInfo itself. DagInfo is meant to provide information for the currently running DAG. getCredentials on the object itself can be used for AM credentials. > expose DAG credentials to plugins > - > > Key: TEZ-3151 > URL: https://issues.apache.org/jira/browse/TEZ-3151 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: TEZ-3151.patch > > > Tez plugins need to pass credentials (e.g. HBase tokens, etc.) to tasks. > Right now they only have access to AM credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-3151) expose DAG credentials to plugins
[ https://issues.apache.org/jira/browse/TEZ-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172641#comment-15172641 ] Sergey Shelukhin edited comment on TEZ-3151 at 2/29/16 9:56 PM: [~sseth] does this make sense? was (Author: sershe): [~sseth] does it make sense? > expose DAG credentials to plugins > - > > Key: TEZ-3151 > URL: https://issues.apache.org/jira/browse/TEZ-3151 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: TEZ-3151.patch > > > Tez plugins need to pass credentials (e.g. HBase tokens, etc.) to tasks. > Right now they only have access to AM credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3149) Tez-tools: Add username in DagInfo
[ https://issues.apache.org/jira/browse/TEZ-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172648#comment-15172648 ] Bikas Saha commented on TEZ-3149: - lgtm . backport to 0.7 would be good. thanks! > Tez-tools: Add username in DagInfo > -- > > Key: TEZ-3149 > URL: https://issues.apache.org/jira/browse/TEZ-3149 > Project: Apache Tez > Issue Type: Improvement >Reporter: Rajesh Balamohan > Attachments: TEZ-3149.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3151) expose DAG credentials to plugins
[ https://issues.apache.org/jira/browse/TEZ-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated TEZ-3151: -- Attachment: TEZ-3151.patch [~sseth] does it make sense? > expose DAG credentials to plugins > - > > Key: TEZ-3151 > URL: https://issues.apache.org/jira/browse/TEZ-3151 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: TEZ-3151.patch > > > Tez plugins need to pass credentials (e.g. HBase tokens, etc.) to tasks. > Right now they only have access to AM credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2863) Container, node, and logs not available in UI for tasks that fail to launch
[
https://issues.apache.org/jira/browse/TEZ-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated TEZ-2863:
-
Attachment: TEZ-2863.4.patch
TEZ-2863.4-branch-0.7.patch
Thanks, [~hitesh]. Posted a new a new corrected patch.
> Container, node, and logs not available in UI for tasks that fail to launch
> ---
>
> Key: TEZ-2863
> URL: https://issues.apache.org/jira/browse/TEZ-2863
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-2863.1.patch, TEZ-2863.2-branch-0.7.patch,
> TEZ-2863.2.patch, TEZ-2863.3-branch-0.7.patch,
> TEZ-2863.3-branch-0.7.patch.addendum, TEZ-2863.3.patch,
> TEZ-2863.3.patch.addendum, TEZ-2863.4-branch-0.7.patch, TEZ-2863.4.patch
>
>
> While running a sample tez job
> {noformat}
> tez-examples-*.jar orderedwordcount -Dtez.task.resource.memory.mb=1
> -Dtez.task.launch.cmd-opts="-Xmx1m" input output
> {noformat}
> It was noticed that the Tez UI task attempt
> http://timelineserverhost:port/ws/v1/timeline/TEZ_TASK_ATTEMPT_ID/attempt_id
> was missing the TEZ_ATTEMPT_STARTED event
> {noformat}
> 2015-10-01 10:03:55,344 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1443711816411_0001_1][Event:TASK_STARTED]:
> vertexName=Tokenizer, taskId=task_1443711816411_0001_1_00_00,
> scheduledTime=1443711835342, launchTime=1443711835342
> 2015-10-01 10:03:55,346 [INFO] [Dispatcher thread {Central}]
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:55,356 [INFO] [TaskSchedulerEventHandlerThread]
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:55,364 [INFO] [TaskSchedulerEventHandlerThread]
> |rm.YarnTaskSchedulerService|: Allocation request for task:
> attempt_1443711816411_0001_1_00_00_0 with request: Capability[ vCores:1>]Priority[2] host: localhost rack: null
> 2015-10-01 10:03:56,639 [INFO] [AMRM Heartbeater thread]
> |impl.AMRMClientImpl|: Received new token for : localhost:57381
> 2015-10-01 10:03:56,646 [INFO] [AMRM Callback Handler Thread]
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:56,648 [INFO] [DelayedContainerManager]
> |rm.YarnTaskSchedulerService|: Assigning container to task:
> containerId=container_1443711816411_0001_01_02,
> task=attempt_1443711816411_0001_1_00_00_0, containerHost=localhost:57381,
> containerPriority= 2, containerResources=,
> localityMatchType=NodeLocal, matchedLocation=localhost,
> honorLocalityFlags=true, reusedContainer=false, delayedContainers=0
> 2015-10-01 10:03:56,649 [INFO] [DelayedContainerManager] |util.RackResolver|:
> Resolved localhost to /default-rack
> 2015-10-01 10:03:56,649 [INFO] [DelayedContainerManager] |util.RackResolver|:
> Resolved localhost to /default-rack
> 2015-10-01 10:03:56,686 [INFO] [TaskSchedulerAppCaller #0]
> |node.AMNodeTracker|: Adding new node: localhost:57381
> 2015-10-01 10:03:56,700 [INFO] [ContainerLauncher #0]
> |launcher.ContainerLauncherImpl|: Launching
> container_1443711816411_0001_01_02
> 2015-10-01 10:03:56,700 [INFO] [ContainerLauncher #0]
> |impl.ContainerManagementProtocolProxy|: Opening proxy : localhost:57381
> 2015-10-01 10:03:56,741 [INFO] [ContainerLauncher #0]
> |history.HistoryEventHandler|: [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]:
> containerId=container_1443711816411_0001_01_02, launchTime=1443711836741
> 2015-10-01 10:03:57,647 [INFO] [AMRM Callback Handler Thread]
> |rm.YarnTaskSchedulerService|: Allocated container
> completed:container_1443711816411_0001_01_02 last allocated to task:
> attempt_1443711816411_0001_1_00_00_0
> 2015-10-01 10:03:57,648 [INFO] [Dispatcher thread {Central}]
> |container.AMContainerImpl|: Container container_1443711816411_0001_01_02
> exited with diagnostics set to Container failed, exitCode=1. Exception from
> container-launch.
> Container id: container_1443711816411_0001_01_02
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
>
[jira] [Updated] (TEZ-3151) expose DAG credentials to plugins
[ https://issues.apache.org/jira/browse/TEZ-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated TEZ-3151: -- Description: Tez plugins need to pass credentials (e.g. HBase tokens, etc.) to tasks. Right now they only have access to AM credentials. > expose DAG credentials to plugins > - > > Key: TEZ-3151 > URL: https://issues.apache.org/jira/browse/TEZ-3151 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > Tez plugins need to pass credentials (e.g. HBase tokens, etc.) to tasks. > Right now they only have access to AM credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3151) expose DAG credentials to plugins
[ https://issues.apache.org/jira/browse/TEZ-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172610#comment-15172610 ] Hitesh Shah commented on TEZ-3151: -- Please add more details on what this jira is meant to address. > expose DAG credentials to plugins > - > > Key: TEZ-3151 > URL: https://issues.apache.org/jira/browse/TEZ-3151 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3120) Remove TaskCommContext.getCurrentDagName, Identifier
[ https://issues.apache.org/jira/browse/TEZ-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172596#comment-15172596 ] Siddharth Seth commented on TEZ-3120: - [~ozawa] - from a very quick look, the patch looks good. I did not put up a patch earlier though, since I want to commit this just before a 0.8.3 release - so that downstream projects (Hive) keep working against Tez master till then. > Remove TaskCommContext.getCurrentDagName, Identifier > > > Key: TEZ-3120 > URL: https://issues.apache.org/jira/browse/TEZ-3120 > Project: Apache Tez > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-3120.001.patch > > > Incompatible change. Remove prior to the 0.8.3 release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-3151) expose DAG credentials to plugins
Sergey Shelukhin created TEZ-3151: - Summary: expose DAG credentials to plugins Key: TEZ-3151 URL: https://issues.apache.org/jira/browse/TEZ-3151 Project: Apache Tez Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2863) Container, node, and logs not available in UI for tasks that fail to launch
[
https://issues.apache.org/jira/browse/TEZ-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172407#comment-15172407
]
Hitesh Shah commented on TEZ-2863:
--
bq. nodeHttpAddress is not not needed for completedLogsUrl
This may be so for an incomplete task but the UI still uses it for an
inprogress url. Also, the code change done via [~jeagles]'s patch seems to use
it for completed log info:
{code}
private String getCompletedLogsUrl() {
String completedLogsUrl = null;
1112if (containerId != null && nodeHttpAddress != null) {
{code}
+1 for the general fix but needs nodeHttpAddress and containerNodeId need to be
recovered too.
[~jeagles], just noticed that the patch needs a minor fix for the above code -
it should be checking if containerNodeId is not null for completed logs.
> Container, node, and logs not available in UI for tasks that fail to launch
> ---
>
> Key: TEZ-2863
> URL: https://issues.apache.org/jira/browse/TEZ-2863
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-2863.1.patch, TEZ-2863.2-branch-0.7.patch,
> TEZ-2863.2.patch, TEZ-2863.3-branch-0.7.patch,
> TEZ-2863.3-branch-0.7.patch.addendum, TEZ-2863.3.patch,
> TEZ-2863.3.patch.addendum
>
>
> While running a sample tez job
> {noformat}
> tez-examples-*.jar orderedwordcount -Dtez.task.resource.memory.mb=1
> -Dtez.task.launch.cmd-opts="-Xmx1m" input output
> {noformat}
> It was noticed that the Tez UI task attempt
> http://timelineserverhost:port/ws/v1/timeline/TEZ_TASK_ATTEMPT_ID/attempt_id
> was missing the TEZ_ATTEMPT_STARTED event
> {noformat}
> 2015-10-01 10:03:55,344 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1443711816411_0001_1][Event:TASK_STARTED]:
> vertexName=Tokenizer, taskId=task_1443711816411_0001_1_00_00,
> scheduledTime=1443711835342, launchTime=1443711835342
> 2015-10-01 10:03:55,346 [INFO] [Dispatcher thread {Central}]
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:55,356 [INFO] [TaskSchedulerEventHandlerThread]
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:55,364 [INFO] [TaskSchedulerEventHandlerThread]
> |rm.YarnTaskSchedulerService|: Allocation request for task:
> attempt_1443711816411_0001_1_00_00_0 with request: Capability[ vCores:1>]Priority[2] host: localhost rack: null
> 2015-10-01 10:03:56,639 [INFO] [AMRM Heartbeater thread]
> |impl.AMRMClientImpl|: Received new token for : localhost:57381
> 2015-10-01 10:03:56,646 [INFO] [AMRM Callback Handler Thread]
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:56,648 [INFO] [DelayedContainerManager]
> |rm.YarnTaskSchedulerService|: Assigning container to task:
> containerId=container_1443711816411_0001_01_02,
> task=attempt_1443711816411_0001_1_00_00_0, containerHost=localhost:57381,
> containerPriority= 2, containerResources=,
> localityMatchType=NodeLocal, matchedLocation=localhost,
> honorLocalityFlags=true, reusedContainer=false, delayedContainers=0
> 2015-10-01 10:03:56,649 [INFO] [DelayedContainerManager] |util.RackResolver|:
> Resolved localhost to /default-rack
> 2015-10-01 10:03:56,649 [INFO] [DelayedContainerManager] |util.RackResolver|:
> Resolved localhost to /default-rack
> 2015-10-01 10:03:56,686 [INFO] [TaskSchedulerAppCaller #0]
> |node.AMNodeTracker|: Adding new node: localhost:57381
> 2015-10-01 10:03:56,700 [INFO] [ContainerLauncher #0]
> |launcher.ContainerLauncherImpl|: Launching
> container_1443711816411_0001_01_02
> 2015-10-01 10:03:56,700 [INFO] [ContainerLauncher #0]
> |impl.ContainerManagementProtocolProxy|: Opening proxy : localhost:57381
> 2015-10-01 10:03:56,741 [INFO] [ContainerLauncher #0]
> |history.HistoryEventHandler|: [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]:
> containerId=container_1443711816411_0001_01_02, launchTime=1443711836741
> 2015-10-01 10:03:57,647 [INFO] [AMRM Callback Handler Thread]
> |rm.YarnTaskSchedulerService|: Allocated container
> completed:container_1443711816411_0001_01_02 last allocated to task:
> attempt_1443711816411_0001_1_00_00_0
> 2015-10-01 10:03:57,648 [INFO] [Dispatcher thread {Central}]
> |container.AMContainerImpl|: Container container_1443711816411_0001_01_02
> exited with diagnostics set to Container failed, exitCode=1. Exception from
> container-launch.
> Container id: container_1443711816411_0001_01_02
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
>
[jira] [Commented] (TEZ-2988) DAGAppMaster::shutdownTezAM should return with a no-op if it has been invoked earlier
[ https://issues.apache.org/jira/browse/TEZ-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172365#comment-15172365 ] Hitesh Shah commented on TEZ-2988: -- +1 committing shortly. > DAGAppMaster::shutdownTezAM should return with a no-op if it has been invoked > earlier > - > > Key: TEZ-2988 > URL: https://issues.apache.org/jira/browse/TEZ-2988 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-2988.001.patch, TEZ-2988.002.patch > > > If the session is already shutting down, DAGAppMaster::shutdownTezAM() should > not block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3115) Shuffle string handling adds significant memory overhead
[
https://issues.apache.org/jira/browse/TEZ-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172321#comment-15172321
]
TezQA commented on TEZ-3115:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12790511/TEZ-3115.3.patch
against master revision 18398c8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.app.rm.TestContainerReuse
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1531//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1531//console
This message is automatically generated.
> Shuffle string handling adds significant memory overhead
>
>
> Key: TEZ-3115
> URL: https://issues.apache.org/jira/browse/TEZ-3115
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Jason Lowe
>Assignee: Jonathan Eagles
> Attachments: TEZ-3115.1.patch, TEZ-3115.2.patch,
> TEZ-3115.3-branch-0.7.patch, TEZ-3115.3.patch
>
>
> While investigating the OOM heap dump from TEZ-3114 I noticed that the
> ShuffleManager and other shuffle-related objects were holding onto many
> strings that added up to over a hundred megabytes of memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Failed: TEZ-3115 PreCommit Build #1531
Jira: https://issues.apache.org/jira/browse/TEZ-3115
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1531/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4572 lines...]
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] [Help 2]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :tez-dag
[INFO] Build failures were ignored.
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12790511/TEZ-3115.3.patch
against master revision 18398c8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.app.rm.TestContainerReuse
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1531//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1531//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
0e8de45db80ba3798293c3b623eb6e8e26575f25 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
1 tests failed.
FAILED:
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources
Error Message:
Wanted but not invoked:
taskSchedulerManagerForTest.taskAllocated(
0,
Mock for TA attempt_0_0001_0_01_04_1,
,
Container: [ContainerId: container_1_0001_01_01, NodeId: host1:0,
NodeHttpAddress: host1:0, Resource: , Priority: 1,
Token: null, ]
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1272)
However, there were other interactions with this mock:
taskSchedulerManagerForTest.init(
Configuration: core-default.xml, core-site.xml, yarn-default.xml,
yarn-site.xml
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1143)
taskSchedulerManagerForTest.setConfig(
Configuration: core-default.xml, core-site.xml, yarn-default.xml,
yarn-site.xml
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1143)
taskSchedulerManagerForTest.serviceInit(
Configuration: core-default.xml, core-site.xml, yarn-default.xml,
yarn-site.xml
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1143)
taskSchedulerManagerForTest.start();
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1144)
taskSchedulerManagerForTest.serviceStart();
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1144)
taskSchedulerManagerForTest.instantiateSchedulers(
"host",
0,
"",
Mock for AppContext, hashCode: 897988698
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1144)
taskSchedulerManagerForTest.getContainerSignatureMatcher();
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1144)
taskSchedulerManagerForTest.getConfig();
-> at
[jira] [Commented] (TEZ-3140) Reduce AM memory usage while serialization
[ https://issues.apache.org/jira/browse/TEZ-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172213#comment-15172213 ] Rohini Palaniswamy commented on TEZ-3140: - The new patch only has a test added and variables declared as final. Failure of https://builds.apache.org/job/PreCommit-TEZ-Build/1530/testReport/junit/org.apache.tez.dag.app.rm/TestContainerReuse/testReuseConflictLocalResources/ is not related. Seems to be flaky. Runs fine in laptop. > Reduce AM memory usage while serialization > -- > > Key: TEZ-3140 > URL: https://issues.apache.org/jira/browse/TEZ-3140 > Project: Apache Tez > Issue Type: Improvement >Reporter: Rohini Palaniswamy >Assignee: Rohini Palaniswamy > Fix For: 0.7.1, 0.8.3 > > Attachments: TEZ-3140-1.patch, TEZ-3140-2.patch > > >There is an unnecessary copy of userpayload byte array during > serialization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3140) Reduce AM memory usage while serialization
[
https://issues.apache.org/jira/browse/TEZ-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172167#comment-15172167
]
TezQA commented on TEZ-3140:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12790501/TEZ-3140-2.patch
against master revision 18398c8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.app.rm.TestContainerReuse
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1530//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1530//console
This message is automatically generated.
> Reduce AM memory usage while serialization
> --
>
> Key: TEZ-3140
> URL: https://issues.apache.org/jira/browse/TEZ-3140
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rohini Palaniswamy
>Assignee: Rohini Palaniswamy
> Fix For: 0.7.1, 0.8.3
>
> Attachments: TEZ-3140-1.patch, TEZ-3140-2.patch
>
>
>There is an unnecessary copy of userpayload byte array during
> serialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Failed: TEZ-3140 PreCommit Build #1530
Jira: https://issues.apache.org/jira/browse/TEZ-3140
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1530/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4530 lines...]
[ERROR] [Help 2]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :tez-dag
[INFO] Build failures were ignored.
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12790501/TEZ-3140-2.patch
against master revision 18398c8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.dag.app.rm.TestContainerReuse
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1530//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1530//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
b8c4c235fa44b6a7b9729b6d8d21c3e9eccf5220 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Compressed 3.36 MB of artifacts by 28.9% relative to #1524
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
1 tests failed.
FAILED:
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources
Error Message:
Wanted but not invoked:
taskSchedulerManagerForTest.taskAllocated(
0,
Mock for TA attempt_0_0001_0_01_03_1,
,
Container: [ContainerId: container_1_0001_01_01, NodeId: host1:0,
NodeHttpAddress: host1:0, Resource: , Priority: 1,
Token: null, ]
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1254)
However, there were other interactions with this mock:
taskSchedulerManagerForTest.init(
Configuration: core-default.xml, core-site.xml, yarn-default.xml,
yarn-site.xml
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1143)
taskSchedulerManagerForTest.setConfig(
Configuration: core-default.xml, core-site.xml, yarn-default.xml,
yarn-site.xml
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1143)
taskSchedulerManagerForTest.serviceInit(
Configuration: core-default.xml, core-site.xml, yarn-default.xml,
yarn-site.xml
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1143)
taskSchedulerManagerForTest.start();
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1144)
taskSchedulerManagerForTest.serviceStart();
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1144)
taskSchedulerManagerForTest.instantiateSchedulers(
"host",
0,
"",
Mock for AppContext, hashCode: 981423673
);
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1144)
taskSchedulerManagerForTest.getContainerSignatureMatcher();
-> at
org.apache.tez.dag.app.rm.TestContainerReuse.testReuseConflictLocalResources(TestContainerReuse.java:1144)
taskSchedulerManagerForTest.getConfig();
-> at
[jira] [Updated] (TEZ-3115) Shuffle string handling adds significant memory overhead
[ https://issues.apache.org/jira/browse/TEZ-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-3115: - Attachment: TEZ-3115.3.patch TEZ-3115.3-branch-0.7.patch [~rajesh.balamohan], addressed the extra string construction. Also, refactored out the no longer used constructBaseURIForShuffleHandler to remove an additional string constriction for host:port. Let me know if this is sufficient. I don't think this introduces a backwards compatibility issue. > Shuffle string handling adds significant memory overhead > > > Key: TEZ-3115 > URL: https://issues.apache.org/jira/browse/TEZ-3115 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe >Assignee: Jonathan Eagles > Attachments: TEZ-3115.1.patch, TEZ-3115.2.patch, > TEZ-3115.3-branch-0.7.patch, TEZ-3115.3.patch > > > While investigating the OOM heap dump from TEZ-3114 I noticed that the > ShuffleManager and other shuffle-related objects were holding onto many > strings that added up to over a hundred megabytes of memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3147) Intermediate mem-to-mem: Fix early exit when only one segment can fit into memory
[ https://issues.apache.org/jira/browse/TEZ-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated TEZ-3147: Fix Version/s: 0.7.1 Thanks, Rajesh! I committed this to branch-0.7. > Intermediate mem-to-mem: Fix early exit when only one segment can fit into > memory > - > > Key: TEZ-3147 > URL: https://issues.apache.org/jira/browse/TEZ-3147 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan > Fix For: 0.7.1, 0.8.3 > > Attachments: TEZ-3147.1.patch > > > Related to TEZ-1911 (since it got committed already, created this jira) > If there is only one segment left out in intermediate mem-to-mem merging, it > does an early exit. But before that it should add the segment back to > inMemoryMapOutput so that it can be considered for merging later. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-1911) MergeManager's unconditionalReserve() should check for memory limits before allocating memory to IntermediateMemoryToMemoryMerger
[ https://issues.apache.org/jira/browse/TEZ-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated TEZ-1911: Fix Version/s: 0.7.1 Thanks, Rajesh! I committed this and the addendum patch to branch-0.7. > MergeManager's unconditionalReserve() should check for memory limits before > allocating memory to IntermediateMemoryToMemoryMerger > - > > Key: TEZ-1911 > URL: https://issues.apache.org/jira/browse/TEZ-1911 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan > Fix For: 0.7.1, 0.8.3 > > Attachments: TEZ-1911.1.patch, TEZ-1911.2.patch, TEZ-1911.3.patch, > TEZ-1911.addendum.findbugs.patch > > > Currently, IntermediateMemoryToMemoryMerger invokes unconditionalReserve() to > get the memory needed for intermediate mem-to-mem merging. It could > potentially cause issue in the following scenario > 1. tez.runtime.io.sort.factor set to 100 and assume shuffled data (e.g 60% > memory occupied) haven't reached TEZ_RUNTIME_SHUFFLE_MERGE_PERCENT_DEFAULT > 2. Assume that it reaches the sort.factor threshold before reaching merge > threshold. This would kick in IntermediateMemoryToMemoryMerger. > In IntermediateMemoryToMemoryMerger, it would try to allocate additional 60% > without any boundary checks. This could lead to OOM depending on the > tez.runtime.io.sort.mb setting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3140) Reduce AM memory usage while serialization
[ https://issues.apache.org/jira/browse/TEZ-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohini Palaniswamy updated TEZ-3140: Attachment: TEZ-3140-2.patch Updated patch addressing review comments from [~sseth]. Did not switch to DataInputByteBuffer to avoid the unnecessary objects it creates. There are couple more in DataInputByteBuffer.Buffer class (scratch, buffers, bidx, pos, length) apart from ones in DataInputStream mentioned by Jon. Though the sizes are negligible, decided to stick with the wrapper class in the patch as that has zero object and size additional overhead. > Reduce AM memory usage while serialization > -- > > Key: TEZ-3140 > URL: https://issues.apache.org/jira/browse/TEZ-3140 > Project: Apache Tez > Issue Type: Improvement >Reporter: Rohini Palaniswamy >Assignee: Rohini Palaniswamy > Fix For: 0.7.1, 0.8.3 > > Attachments: TEZ-3140-1.patch, TEZ-3140-2.patch > > >There is an unnecessary copy of userpayload byte array during > serialization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-3149 PreCommit Build #1529
Jira: https://issues.apache.org/jira/browse/TEZ-3149
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1529/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4401 lines...]
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] [Help 2]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :tez-tests
[INFO] Build failures were ignored.
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12790455/TEZ-3149.1.patch
against master revision 18398c8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1529//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1529//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
e4a6cd63d2970ffe1f94c616eb78e887ea518855 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
7 tests failed.
FAILED: org.apache.tez.test.TestFaultTolerance.testRandomFailingInputs
Error Message:
expected: but was:
Stack Trace:
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:141)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:124)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:120)
at
org.apache.tez.test.TestFaultTolerance.testRandomFailingInputs(TestFaultTolerance.java:763)
FAILED: org.apache.tez.test.TestFaultTolerance.testBasicInputFailureWithExit
Error Message:
TezSession has already shutdown. No cluster diagnostics found.
Stack Trace:
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No
cluster diagnostics found.
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:784)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:129)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:124)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:120)
at
org.apache.tez.test.TestFaultTolerance.testBasicInputFailureWithExit(TestFaultTolerance.java:261)
FAILED:
org.apache.tez.test.TestFaultTolerance.testInputFailureRerunCanSendOutputToTwoDownstreamVertices
Error Message:
TezSession has already shutdown. No cluster diagnostics found.
Stack Trace:
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No
cluster diagnostics found.
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:784)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:129)
at
[jira] [Commented] (TEZ-3149) Tez-tools: Add username in DagInfo
[
https://issues.apache.org/jira/browse/TEZ-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171831#comment-15171831
]
TezQA commented on TEZ-3149:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12790455/TEZ-3149.1.patch
against master revision 18398c8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1529//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1529//console
This message is automatically generated.
> Tez-tools: Add username in DagInfo
> --
>
> Key: TEZ-3149
> URL: https://issues.apache.org/jira/browse/TEZ-3149
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
> Attachments: TEZ-3149.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3149) Tez-tools: Add username in DagInfo
[ https://issues.apache.org/jira/browse/TEZ-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-3149: -- Attachment: TEZ-3149.1.patch [~bikassaha] - Please review when you find time. > Tez-tools: Add username in DagInfo > -- > > Key: TEZ-3149 > URL: https://issues.apache.org/jira/browse/TEZ-3149 > Project: Apache Tez > Issue Type: Improvement >Reporter: Rajesh Balamohan > Attachments: TEZ-3149.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3115) Shuffle string handling adds significant memory overhead
[ https://issues.apache.org/jira/browse/TEZ-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171663#comment-15171663 ] Rajesh Balamohan commented on TEZ-3115: --- Minor: -Should FetcherOrderedGrouped directly call ShuffleUtils.constructBaseURIForShuffleHandler instead of going via another redirection (to avoid additional string creation in for host + ":" + port) > Shuffle string handling adds significant memory overhead > > > Key: TEZ-3115 > URL: https://issues.apache.org/jira/browse/TEZ-3115 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe >Assignee: Jonathan Eagles > Attachments: TEZ-3115.1.patch, TEZ-3115.2.patch > > > While investigating the OOM heap dump from TEZ-3114 I noticed that the > ShuffleManager and other shuffle-related objects were holding onto many > strings that added up to over a hundred megabytes of memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-3150) Tez UI 2: Details displayed need to be pruned
Sreenath Somarajapuram created TEZ-3150: --- Summary: Tez UI 2: Details displayed need to be pruned Key: TEZ-3150 URL: https://issues.apache.org/jira/browse/TEZ-3150 Project: Apache Tez Issue Type: Sub-task Reporter: Sreenath Somarajapuram Following are suggestions from [~gopalv] Tables must display only the following columns by default: - If needed other columns can be enabled from column selector. - DAGs - DAG Name, Submitter, Start time, Duration, Status, Queue, AppID - Vertices - Vertex Name, Status, Start time, Duration, Tasks, Progress - All Tasks - Vertex name, Task Index, Status, Progress, Start time, Duration -- Vertex > Tasks - Task Index, Status, Progress, Start Time, Duration - All Attempts - Vertex Name, Task Index, Attempt No, Status, Start Time, Duration, Node, Progress -- Vertex > Attempts - Task Index, Attempt No, Status, Start Time, Duration, Node, Progress -- Tasks > Attempts - Attempt No, Status, Start time, Duration, Node, Progress Details Page: - Remove end time in all details page. - Vertex details: -- After durations and start/finish times just show, task index and not the complete task id. -- Display duration from Vertex start time to first task start time (Val must be >=0). -- Display duration from last task finish time to vertex end time (Val must be >=0). - Task Details: Move task attempts table into the task details page. \cc [~venkateshrin] [~hitesh] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
