[jira] [Updated] (TEZ-3149) Tez-tools: Add username in DagInfo
[ https://issues.apache.org/jira/browse/TEZ-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-3149: -- Summary: Tez-tools: Add username in DagInfo (was: Tez-toos: Add username in DagInfo) > Tez-tools: Add username in DagInfo > -- > > Key: TEZ-3149 > URL: https://issues.apache.org/jira/browse/TEZ-3149 > Project: Apache Tez > Issue Type: Improvement >Reporter: Rajesh Balamohan > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-3149) Tez-toos: Add username in DagInfo
Rajesh Balamohan created TEZ-3149: - Summary: Tez-toos: Add username in DagInfo Key: TEZ-3149 URL: https://issues.apache.org/jira/browse/TEZ-3149 Project: Apache Tez Issue Type: Improvement Reporter: Rajesh Balamohan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2863) Container, node, and logs not available in UI for tasks that fail to launch
[
https://issues.apache.org/jira/browse/TEZ-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171388#comment-15171388
]
Jeff Zhang commented on TEZ-2863:
-
[~jeagles] I attached 2 addendum patch based on your patch. [~hitesh] Please
help review the addendum patch.
Recovery event handling is in a different thread as ATS event handling. It is
possible that TaskAttemptFinshedEvent in logged as recovery event but has not
been logged as ATS event. So we need to recover TaskAttemptFinished properly
to ensure we send the correct ATS event in recovering.
* add containerId in TaskAttemptFinishedEvent as recovery part
* nodeHttpAddress is not not needed for completedLogsUrl
> Container, node, and logs not available in UI for tasks that fail to launch
> ---
>
> Key: TEZ-2863
> URL: https://issues.apache.org/jira/browse/TEZ-2863
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-2863.1.patch, TEZ-2863.2-branch-0.7.patch,
> TEZ-2863.2.patch, TEZ-2863.3-branch-0.7.patch,
> TEZ-2863.3-branch-0.7.patch.addendum, TEZ-2863.3.patch,
> TEZ-2863.3.patch.addendum
>
>
> While running a sample tez job
> {noformat}
> tez-examples-*.jar orderedwordcount -Dtez.task.resource.memory.mb=1
> -Dtez.task.launch.cmd-opts="-Xmx1m" input output
> {noformat}
> It was noticed that the Tez UI task attempt
> http://timelineserverhost:port/ws/v1/timeline/TEZ_TASK_ATTEMPT_ID/attempt_id
> was missing the TEZ_ATTEMPT_STARTED event
> {noformat}
> 2015-10-01 10:03:55,344 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1443711816411_0001_1][Event:TASK_STARTED]:
> vertexName=Tokenizer, taskId=task_1443711816411_0001_1_00_00,
> scheduledTime=1443711835342, launchTime=1443711835342
> 2015-10-01 10:03:55,346 [INFO] [Dispatcher thread {Central}]
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:55,356 [INFO] [TaskSchedulerEventHandlerThread]
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:55,364 [INFO] [TaskSchedulerEventHandlerThread]
> |rm.YarnTaskSchedulerService|: Allocation request for task:
> attempt_1443711816411_0001_1_00_00_0 with request: Capability[ vCores:1>]Priority[2] host: localhost rack: null
> 2015-10-01 10:03:56,639 [INFO] [AMRM Heartbeater thread]
> |impl.AMRMClientImpl|: Received new token for : localhost:57381
> 2015-10-01 10:03:56,646 [INFO] [AMRM Callback Handler Thread]
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:56,648 [INFO] [DelayedContainerManager]
> |rm.YarnTaskSchedulerService|: Assigning container to task:
> containerId=container_1443711816411_0001_01_02,
> task=attempt_1443711816411_0001_1_00_00_0, containerHost=localhost:57381,
> containerPriority= 2, containerResources=,
> localityMatchType=NodeLocal, matchedLocation=localhost,
> honorLocalityFlags=true, reusedContainer=false, delayedContainers=0
> 2015-10-01 10:03:56,649 [INFO] [DelayedContainerManager] |util.RackResolver|:
> Resolved localhost to /default-rack
> 2015-10-01 10:03:56,649 [INFO] [DelayedContainerManager] |util.RackResolver|:
> Resolved localhost to /default-rack
> 2015-10-01 10:03:56,686 [INFO] [TaskSchedulerAppCaller #0]
> |node.AMNodeTracker|: Adding new node: localhost:57381
> 2015-10-01 10:03:56,700 [INFO] [ContainerLauncher #0]
> |launcher.ContainerLauncherImpl|: Launching
> container_1443711816411_0001_01_02
> 2015-10-01 10:03:56,700 [INFO] [ContainerLauncher #0]
> |impl.ContainerManagementProtocolProxy|: Opening proxy : localhost:57381
> 2015-10-01 10:03:56,741 [INFO] [ContainerLauncher #0]
> |history.HistoryEventHandler|: [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]:
> containerId=container_1443711816411_0001_01_02, launchTime=1443711836741
> 2015-10-01 10:03:57,647 [INFO] [AMRM Callback Handler Thread]
> |rm.YarnTaskSchedulerService|: Allocated container
> completed:container_1443711816411_0001_01_02 last allocated to task:
> attempt_1443711816411_0001_1_00_00_0
> 2015-10-01 10:03:57,648 [INFO] [Dispatcher thread {Central}]
> |container.AMContainerImpl|: Container container_1443711816411_0001_01_02
> exited with diagnostics set to Container failed, exitCode=1. Exception from
> container-launch.
> Container id: container_1443711816411_0001_01_02
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> at
>
[jira] [Commented] (TEZ-2863) Container, node, and logs not available in UI for tasks that fail to launch
[
https://issues.apache.org/jira/browse/TEZ-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171384#comment-15171384
]
TezQA commented on TEZ-2863:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12790402/TEZ-2863.3.patch.addendum
against master revision 18398c8.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1528//console
This message is automatically generated.
> Container, node, and logs not available in UI for tasks that fail to launch
> ---
>
> Key: TEZ-2863
> URL: https://issues.apache.org/jira/browse/TEZ-2863
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-2863.1.patch, TEZ-2863.2-branch-0.7.patch,
> TEZ-2863.2.patch, TEZ-2863.3-branch-0.7.patch,
> TEZ-2863.3-branch-0.7.patch.addendum, TEZ-2863.3.patch,
> TEZ-2863.3.patch.addendum
>
>
> While running a sample tez job
> {noformat}
> tez-examples-*.jar orderedwordcount -Dtez.task.resource.memory.mb=1
> -Dtez.task.launch.cmd-opts="-Xmx1m" input output
> {noformat}
> It was noticed that the Tez UI task attempt
> http://timelineserverhost:port/ws/v1/timeline/TEZ_TASK_ATTEMPT_ID/attempt_id
> was missing the TEZ_ATTEMPT_STARTED event
> {noformat}
> 2015-10-01 10:03:55,344 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1443711816411_0001_1][Event:TASK_STARTED]:
> vertexName=Tokenizer, taskId=task_1443711816411_0001_1_00_00,
> scheduledTime=1443711835342, launchTime=1443711835342
> 2015-10-01 10:03:55,346 [INFO] [Dispatcher thread {Central}]
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:55,356 [INFO] [TaskSchedulerEventHandlerThread]
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:55,364 [INFO] [TaskSchedulerEventHandlerThread]
> |rm.YarnTaskSchedulerService|: Allocation request for task:
> attempt_1443711816411_0001_1_00_00_0 with request: Capability[ vCores:1>]Priority[2] host: localhost rack: null
> 2015-10-01 10:03:56,639 [INFO] [AMRM Heartbeater thread]
> |impl.AMRMClientImpl|: Received new token for : localhost:57381
> 2015-10-01 10:03:56,646 [INFO] [AMRM Callback Handler Thread]
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:56,648 [INFO] [DelayedContainerManager]
> |rm.YarnTaskSchedulerService|: Assigning container to task:
> containerId=container_1443711816411_0001_01_02,
> task=attempt_1443711816411_0001_1_00_00_0, containerHost=localhost:57381,
> containerPriority= 2, containerResources=,
> localityMatchType=NodeLocal, matchedLocation=localhost,
> honorLocalityFlags=true, reusedContainer=false, delayedContainers=0
> 2015-10-01 10:03:56,649 [INFO] [DelayedContainerManager] |util.RackResolver|:
> Resolved localhost to /default-rack
> 2015-10-01 10:03:56,649 [INFO] [DelayedContainerManager] |util.RackResolver|:
> Resolved localhost to /default-rack
> 2015-10-01 10:03:56,686 [INFO] [TaskSchedulerAppCaller #0]
> |node.AMNodeTracker|: Adding new node: localhost:57381
> 2015-10-01 10:03:56,700 [INFO] [ContainerLauncher #0]
> |launcher.ContainerLauncherImpl|: Launching
> container_1443711816411_0001_01_02
> 2015-10-01 10:03:56,700 [INFO] [ContainerLauncher #0]
> |impl.ContainerManagementProtocolProxy|: Opening proxy : localhost:57381
> 2015-10-01 10:03:56,741 [INFO] [ContainerLauncher #0]
> |history.HistoryEventHandler|: [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]:
> containerId=container_1443711816411_0001_01_02, launchTime=1443711836741
> 2015-10-01 10:03:57,647 [INFO] [AMRM Callback Handler Thread]
> |rm.YarnTaskSchedulerService|: Allocated container
> completed:container_1443711816411_0001_01_02 last allocated to task:
> attempt_1443711816411_0001_1_00_00_0
> 2015-10-01 10:03:57,648 [INFO] [Dispatcher thread {Central}]
> |container.AMContainerImpl|: Container container_1443711816411_0001_01_02
> exited with diagnostics set to Container failed, exitCode=1. Exception from
> container-launch.
> Container id: container_1443711816411_0001_01_02
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> at
>
Failed: TEZ-2863 PreCommit Build #1528
Jira: https://issues.apache.org/jira/browse/TEZ-2863
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1528/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 21 lines...]
[PreCommit-TEZ-Build] $ /bin/bash /tmp/hudson5537669531841241805.sh
Running in Jenkins mode
==
==
Testing patch for TEZ-2863.
==
==
HEAD is now at 18398c8 TEZ-3143. Tez UI 2: Make the build faster (sree)
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
Current branch master is up to date.
TEZ-2863 patch is being downloaded at Mon Feb 29 04:18:41 UTC 2016 from
http://issues.apache.org/jira/secure/attachment/12790402/TEZ-2863.3.patch.addendum
The patch does not appear to apply with p0 to p2
PATCH APPLICATION FAILED
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12790402/TEZ-2863.3.patch.addendum
against master revision 18398c8.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1528//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
52833be7d0d7d8141b2f3258f5065129a4e6a3e3 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files
were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
No tests ran.
[jira] [Updated] (TEZ-2863) Container, node, and logs not available in UI for tasks that fail to launch
[
https://issues.apache.org/jira/browse/TEZ-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Zhang updated TEZ-2863:
Attachment: TEZ-2863.3.patch.addendum
TEZ-2863.3-branch-0.7.patch.addendum
> Container, node, and logs not available in UI for tasks that fail to launch
> ---
>
> Key: TEZ-2863
> URL: https://issues.apache.org/jira/browse/TEZ-2863
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-2863.1.patch, TEZ-2863.2-branch-0.7.patch,
> TEZ-2863.2.patch, TEZ-2863.3-branch-0.7.patch,
> TEZ-2863.3-branch-0.7.patch.addendum, TEZ-2863.3.patch,
> TEZ-2863.3.patch.addendum
>
>
> While running a sample tez job
> {noformat}
> tez-examples-*.jar orderedwordcount -Dtez.task.resource.memory.mb=1
> -Dtez.task.launch.cmd-opts="-Xmx1m" input output
> {noformat}
> It was noticed that the Tez UI task attempt
> http://timelineserverhost:port/ws/v1/timeline/TEZ_TASK_ATTEMPT_ID/attempt_id
> was missing the TEZ_ATTEMPT_STARTED event
> {noformat}
> 2015-10-01 10:03:55,344 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1443711816411_0001_1][Event:TASK_STARTED]:
> vertexName=Tokenizer, taskId=task_1443711816411_0001_1_00_00,
> scheduledTime=1443711835342, launchTime=1443711835342
> 2015-10-01 10:03:55,346 [INFO] [Dispatcher thread {Central}]
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:55,356 [INFO] [TaskSchedulerEventHandlerThread]
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:55,364 [INFO] [TaskSchedulerEventHandlerThread]
> |rm.YarnTaskSchedulerService|: Allocation request for task:
> attempt_1443711816411_0001_1_00_00_0 with request: Capability[ vCores:1>]Priority[2] host: localhost rack: null
> 2015-10-01 10:03:56,639 [INFO] [AMRM Heartbeater thread]
> |impl.AMRMClientImpl|: Received new token for : localhost:57381
> 2015-10-01 10:03:56,646 [INFO] [AMRM Callback Handler Thread]
> |util.RackResolver|: Resolved localhost to /default-rack
> 2015-10-01 10:03:56,648 [INFO] [DelayedContainerManager]
> |rm.YarnTaskSchedulerService|: Assigning container to task:
> containerId=container_1443711816411_0001_01_02,
> task=attempt_1443711816411_0001_1_00_00_0, containerHost=localhost:57381,
> containerPriority= 2, containerResources=,
> localityMatchType=NodeLocal, matchedLocation=localhost,
> honorLocalityFlags=true, reusedContainer=false, delayedContainers=0
> 2015-10-01 10:03:56,649 [INFO] [DelayedContainerManager] |util.RackResolver|:
> Resolved localhost to /default-rack
> 2015-10-01 10:03:56,649 [INFO] [DelayedContainerManager] |util.RackResolver|:
> Resolved localhost to /default-rack
> 2015-10-01 10:03:56,686 [INFO] [TaskSchedulerAppCaller #0]
> |node.AMNodeTracker|: Adding new node: localhost:57381
> 2015-10-01 10:03:56,700 [INFO] [ContainerLauncher #0]
> |launcher.ContainerLauncherImpl|: Launching
> container_1443711816411_0001_01_02
> 2015-10-01 10:03:56,700 [INFO] [ContainerLauncher #0]
> |impl.ContainerManagementProtocolProxy|: Opening proxy : localhost:57381
> 2015-10-01 10:03:56,741 [INFO] [ContainerLauncher #0]
> |history.HistoryEventHandler|: [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]:
> containerId=container_1443711816411_0001_01_02, launchTime=1443711836741
> 2015-10-01 10:03:57,647 [INFO] [AMRM Callback Handler Thread]
> |rm.YarnTaskSchedulerService|: Allocated container
> completed:container_1443711816411_0001_01_02 last allocated to task:
> attempt_1443711816411_0001_1_00_00_0
> 2015-10-01 10:03:57,648 [INFO] [Dispatcher thread {Central}]
> |container.AMContainerImpl|: Container container_1443711816411_0001_01_02
> exited with diagnostics set to Container failed, exitCode=1. Exception from
> container-launch.
> Container id: container_1443711816411_0001_01_02
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
Failed: TEZ-2988 PreCommit Build #1527
Jira: https://issues.apache.org/jira/browse/TEZ-2988
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1527/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4400 lines...]
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :tez-tests
[INFO] Build failures were ignored.
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12790388/TEZ-2988.002.patch
against master revision 18398c8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1527//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1527//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
f3dc02b4e527d95fc27d8354de6f1d646be6713e logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
7 tests failed.
FAILED: org.apache.tez.test.TestFaultTolerance.testRandomFailingInputs
Error Message:
expected: but was:
Stack Trace:
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:141)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:124)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:120)
at
org.apache.tez.test.TestFaultTolerance.testRandomFailingInputs(TestFaultTolerance.java:763)
FAILED: org.apache.tez.test.TestFaultTolerance.testBasicInputFailureWithExit
Error Message:
TezSession has already shutdown. No cluster diagnostics found.
Stack Trace:
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No
cluster diagnostics found.
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:784)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:129)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:124)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:120)
at
org.apache.tez.test.TestFaultTolerance.testBasicInputFailureWithExit(TestFaultTolerance.java:261)
FAILED:
org.apache.tez.test.TestFaultTolerance.testInputFailureRerunCanSendOutputToTwoDownstreamVertices
Error Message:
TezSession has already shutdown. No cluster diagnostics found.
Stack Trace:
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No
cluster diagnostics found.
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:784)
at
org.apache.tez.test.TestFaultTolerance.runDAGAndVerify(TestFaultTolerance.java:129)
at
[jira] [Commented] (TEZ-2988) DAGAppMaster::shutdownTezAM should return with a no-op if it has been invoked earlier
[
https://issues.apache.org/jira/browse/TEZ-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171333#comment-15171333
]
TezQA commented on TEZ-2988:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12790388/TEZ-2988.002.patch
against master revision 18398c8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1527//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1527//console
This message is automatically generated.
> DAGAppMaster::shutdownTezAM should return with a no-op if it has been invoked
> earlier
> -
>
> Key: TEZ-2988
> URL: https://issues.apache.org/jira/browse/TEZ-2988
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Hitesh Shah
>Assignee: Tsuyoshi Ozawa
> Attachments: TEZ-2988.001.patch, TEZ-2988.002.patch
>
>
> If the session is already shutting down, DAGAppMaster::shutdownTezAM() should
> not block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2988) DAGAppMaster::shutdownTezAM should return with a no-op if it has been invoked earlier
[ https://issues.apache.org/jira/browse/TEZ-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-2988: Attachment: TEZ-2988.002.patch [~hitesh] thanks for your review. Completely agree, updating a patch. > DAGAppMaster::shutdownTezAM should return with a no-op if it has been invoked > earlier > - > > Key: TEZ-2988 > URL: https://issues.apache.org/jira/browse/TEZ-2988 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-2988.001.patch, TEZ-2988.002.patch > > > If the session is already shutting down, DAGAppMaster::shutdownTezAM() should > not block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3014) OOM during Shuffle in JDK 8
[ https://issues.apache.org/jira/browse/TEZ-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171245#comment-15171245 ] Bikas Saha commented on TEZ-3014: - [~jeagles] [~jlowe] Is this still an issue? OOM + JDK 8. If not, then we could close this. > OOM during Shuffle in JDK 8 > --- > > Key: TEZ-3014 > URL: https://issues.apache.org/jira/browse/TEZ-3014 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2580) Remove VertexManagerPlugin#setVertexParallelism with VertexManagerPlugin#reconfigureVertex
[ https://issues.apache.org/jira/browse/TEZ-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171239#comment-15171239 ] Bikas Saha commented on TEZ-2580: - We can only change this if dependent projects like Hive stop using it or else they will fail to compile. Not sure if they have done that. > Remove VertexManagerPlugin#setVertexParallelism with > VertexManagerPlugin#reconfigureVertex > -- > > Key: TEZ-2580 > URL: https://issues.apache.org/jira/browse/TEZ-2580 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha >Assignee: Tsuyoshi Ozawa >Priority: Blocker > Attachments: TEZ-2580.001.patch > > > This was deprecated in 0.7. Should be replaced with reconfigureVertex() - > change of name - to make it consistent with other reconfigureVertex() API's. > Should be done just close to release to enabled Hive to continue to build/use > master of Tez. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-2988 PreCommit Build #1526
Jira: https://issues.apache.org/jira/browse/TEZ-2988
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1526/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4556 lines...]
[INFO]
[INFO] Total time: 55:42 min
[INFO] Finished at: 2016-02-28T22:58:05+00:00
[INFO] Final Memory: 75M/1479M
[INFO]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12790379/TEZ-2988.001.patch
against master revision 18398c8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1526//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1526//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
af1f2213a1dfea04691570874ee57a98ecf664a8 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2988) DAGAppMaster::shutdownTezAM should return with a no-op if it has been invoked earlier
[
https://issues.apache.org/jira/browse/TEZ-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171227#comment-15171227
]
TezQA commented on TEZ-2988:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12790379/TEZ-2988.001.patch
against master revision 18398c8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1526//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1526//console
This message is automatically generated.
> DAGAppMaster::shutdownTezAM should return with a no-op if it has been invoked
> earlier
> -
>
> Key: TEZ-2988
> URL: https://issues.apache.org/jira/browse/TEZ-2988
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Hitesh Shah
>Assignee: Tsuyoshi Ozawa
> Attachments: TEZ-2988.001.patch
>
>
> If the session is already shutting down, DAGAppMaster::shutdownTezAM() should
> not block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2988) DAGAppMaster::shutdownTezAM should return with a no-op if it has been invoked earlier
[ https://issues.apache.org/jira/browse/TEZ-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171203#comment-15171203 ] Hitesh Shah commented on TEZ-2988: -- Generally looks fine. Minor nit: Instead of putting the full block of code into the if section, maybe do a return if sessionStopped.compareAndSet(false, true) returns false with a comment on why the return is being done? > DAGAppMaster::shutdownTezAM should return with a no-op if it has been invoked > earlier > - > > Key: TEZ-2988 > URL: https://issues.apache.org/jira/browse/TEZ-2988 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-2988.001.patch > > > If the session is already shutting down, DAGAppMaster::shutdownTezAM() should > not block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2988) DAGAppMaster::shutdownTezAM should return with a no-op if it has been invoked earlier
[ https://issues.apache.org/jira/browse/TEZ-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-2988: Attachment: TEZ-2988.001.patch Attaching v1 patch. [~hitesh], could you take a look? > DAGAppMaster::shutdownTezAM should return with a no-op if it has been invoked > earlier > - > > Key: TEZ-2988 > URL: https://issues.apache.org/jira/browse/TEZ-2988 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah > Attachments: TEZ-2988.001.patch > > > If the session is already shutting down, DAGAppMaster::shutdownTezAM() should > not block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3148) Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
[
https://issues.apache.org/jira/browse/TEZ-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171193#comment-15171193
]
Hitesh Shah commented on TEZ-3148:
--
Not related to the state machine error though.
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
>
>
> Key: TEZ-3148
> URL: https://issues.apache.org/jira/browse/TEZ-3148
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
> Attachments: am.log.gz, dag.dot
>
>
> Got the following when executing one of the DAG.
> Tez details:
> versionInfo=[ component=tez-dag, version=0.8.3-SNAPSHOT,
> revision=3e409ae0ee7233b4cf631cac1bc366679a08b7d1,
> SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git,
> buildTime=20160227-1912]
> {noformat}
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
> attempt_1455662455106_2317_27_02_000339_0
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Invalid event TA_TEZ_EVENT_UPDATE
> on TaskAttempt attempt_1455662455106_2317_27_02_000339_0
> Exception in thread "75b0f971-7f89-461a-b432-45e1ac6e374b main"
> java.lang.AbstractMethodError:
> org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager.close()V
> at org.apache.tez.client.TezClient.stop(TezClient.java:562)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.closeClient(TezSessionState.java:474)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.close(TezSessionState.java:436)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeIfNotDefault(TezSessionPoolManager.java:338)
> at
> org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1469)
> at
> org.apache.hadoop.hive.cli.CliSessionState.close(CliSessionState.java:66)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:719)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:645)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (TEZ-3148) Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
[
https://issues.apache.org/jira/browse/TEZ-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171192#comment-15171192
]
Hitesh Shah edited comment on TEZ-3148 at 2/28/16 9:22 PM:
---
[~rajesh.balamohan] Are you using some history service impl which has not
implemented the close function for the acl policy?
{code}
Exception in thread "75b0f971-7f89-461a-b432-45e1ac6e374b main"
java.lang.AbstractMethodError:
org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager.close()V
{code}
was (Author: hitesh):
[~rajesh.balamohan] Are you using some history service impl which has not
implemented the close function for the acl policy?
{code}
Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
attempt_1455662455106_2317_27_02_000339_0
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.tez.TezTask. Invalid event TA_TEZ_EVENT_UPDATE
on TaskAttempt attempt_1455662455106_2317_27_02_000339_0
Exception in thread "75b0f971-7f89-461a-b432-45e1ac6e374b main"
java.lang.AbstractMethodError:
org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager.close()V
{code}
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
>
>
> Key: TEZ-3148
> URL: https://issues.apache.org/jira/browse/TEZ-3148
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
> Attachments: am.log.gz, dag.dot
>
>
> Got the following when executing one of the DAG.
> Tez details:
> versionInfo=[ component=tez-dag, version=0.8.3-SNAPSHOT,
> revision=3e409ae0ee7233b4cf631cac1bc366679a08b7d1,
> SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git,
> buildTime=20160227-1912]
> {noformat}
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
> attempt_1455662455106_2317_27_02_000339_0
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Invalid event TA_TEZ_EVENT_UPDATE
> on TaskAttempt attempt_1455662455106_2317_27_02_000339_0
> Exception in thread "75b0f971-7f89-461a-b432-45e1ac6e374b main"
> java.lang.AbstractMethodError:
> org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager.close()V
> at org.apache.tez.client.TezClient.stop(TezClient.java:562)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.closeClient(TezSessionState.java:474)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.close(TezSessionState.java:436)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeIfNotDefault(TezSessionPoolManager.java:338)
> at
> org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1469)
> at
> org.apache.hadoop.hive.cli.CliSessionState.close(CliSessionState.java:66)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:719)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:645)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3148) Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
[
https://issues.apache.org/jira/browse/TEZ-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171192#comment-15171192
]
Hitesh Shah commented on TEZ-3148:
--
[~rajesh.balamohan] Are you using some history service impl which has not
implemented the close function for the acl policy?
{code}
Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
attempt_1455662455106_2317_27_02_000339_0
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.tez.TezTask. Invalid event TA_TEZ_EVENT_UPDATE
on TaskAttempt attempt_1455662455106_2317_27_02_000339_0
Exception in thread "75b0f971-7f89-461a-b432-45e1ac6e374b main"
java.lang.AbstractMethodError:
org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager.close()V
{code}
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
>
>
> Key: TEZ-3148
> URL: https://issues.apache.org/jira/browse/TEZ-3148
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
> Attachments: am.log.gz, dag.dot
>
>
> Got the following when executing one of the DAG.
> Tez details:
> versionInfo=[ component=tez-dag, version=0.8.3-SNAPSHOT,
> revision=3e409ae0ee7233b4cf631cac1bc366679a08b7d1,
> SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git,
> buildTime=20160227-1912]
> {noformat}
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
> attempt_1455662455106_2317_27_02_000339_0
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Invalid event TA_TEZ_EVENT_UPDATE
> on TaskAttempt attempt_1455662455106_2317_27_02_000339_0
> Exception in thread "75b0f971-7f89-461a-b432-45e1ac6e374b main"
> java.lang.AbstractMethodError:
> org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager.close()V
> at org.apache.tez.client.TezClient.stop(TezClient.java:562)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.closeClient(TezSessionState.java:474)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.close(TezSessionState.java:436)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeIfNotDefault(TezSessionPoolManager.java:338)
> at
> org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1469)
> at
> org.apache.hadoop.hive.cli.CliSessionState.close(CliSessionState.java:66)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:719)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:645)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3120) Remove TaskCommContext.getCurrentDagName, Identifier
[ https://issues.apache.org/jira/browse/TEZ-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3120: Attachment: (was: TEZ-3120.001.patch) > Remove TaskCommContext.getCurrentDagName, Identifier > > > Key: TEZ-3120 > URL: https://issues.apache.org/jira/browse/TEZ-3120 > Project: Apache Tez > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-3120.001.patch > > > Incompatible change. Remove prior to the 0.8.3 release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (TEZ-3148) Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
Rajesh Balamohan created TEZ-3148:
-
Summary: Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
Key: TEZ-3148
URL: https://issues.apache.org/jira/browse/TEZ-3148
Project: Apache Tez
Issue Type: Bug
Reporter: Rajesh Balamohan
Attachments: am.log.gz, dag.dot
Got the following when executing one of the DAG.
Tez details:
versionInfo=[ component=tez-dag, version=0.8.3-SNAPSHOT,
revision=3e409ae0ee7233b4cf631cac1bc366679a08b7d1,
SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git,
buildTime=20160227-1912]
{noformat}
Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
attempt_1455662455106_2317_27_02_000339_0
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.tez.TezTask. Invalid event TA_TEZ_EVENT_UPDATE
on TaskAttempt attempt_1455662455106_2317_27_02_000339_0
Exception in thread "75b0f971-7f89-461a-b432-45e1ac6e374b main"
java.lang.AbstractMethodError:
org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager.close()V
at org.apache.tez.client.TezClient.stop(TezClient.java:562)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.closeClient(TezSessionState.java:474)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.close(TezSessionState.java:436)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeIfNotDefault(TezSessionPoolManager.java:338)
at
org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1469)
at
org.apache.hadoop.hive.cli.CliSessionState.close(CliSessionState.java:66)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:719)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:645)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-3148) Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
[
https://issues.apache.org/jira/browse/TEZ-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated TEZ-3148:
--
Attachment: am.log.gz
dag.dot
Attaching dag.dot and am log for this dag.
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
>
>
> Key: TEZ-3148
> URL: https://issues.apache.org/jira/browse/TEZ-3148
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
> Attachments: am.log.gz, dag.dot
>
>
> Got the following when executing one of the DAG.
> Tez details:
> versionInfo=[ component=tez-dag, version=0.8.3-SNAPSHOT,
> revision=3e409ae0ee7233b4cf631cac1bc366679a08b7d1,
> SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git,
> buildTime=20160227-1912]
> {noformat}
> Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt
> attempt_1455662455106_2317_27_02_000339_0
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Invalid event TA_TEZ_EVENT_UPDATE
> on TaskAttempt attempt_1455662455106_2317_27_02_000339_0
> Exception in thread "75b0f971-7f89-461a-b432-45e1ac6e374b main"
> java.lang.AbstractMethodError:
> org.apache.tez.dag.history.ats.acls.ATSHistoryACLPolicyManager.close()V
> at org.apache.tez.client.TezClient.stop(TezClient.java:562)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.closeClient(TezSessionState.java:474)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.close(TezSessionState.java:436)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeIfNotDefault(TezSessionPoolManager.java:338)
> at
> org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1469)
> at
> org.apache.hadoop.hive.cli.CliSessionState.close(CliSessionState.java:66)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:719)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:645)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2580) Remove VertexManagerPlugin#setVertexParallelism with VertexManagerPlugin#reconfigureVertex
[
https://issues.apache.org/jira/browse/TEZ-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171064#comment-15171064
]
TezQA commented on TEZ-2580:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12790360/TEZ-2580.001.patch
against master revision 18398c8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1525//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1525//console
This message is automatically generated.
> Remove VertexManagerPlugin#setVertexParallelism with
> VertexManagerPlugin#reconfigureVertex
> --
>
> Key: TEZ-2580
> URL: https://issues.apache.org/jira/browse/TEZ-2580
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Bikas Saha
>Assignee: Tsuyoshi Ozawa
>Priority: Blocker
> Attachments: TEZ-2580.001.patch
>
>
> This was deprecated in 0.7. Should be replaced with reconfigureVertex() -
> change of name - to make it consistent with other reconfigureVertex() API's.
> Should be done just close to release to enabled Hive to continue to build/use
> master of Tez.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Success: TEZ-2756 PreCommit Build #1524
Jira: https://issues.apache.org/jira/browse/TEZ-2756
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/1524/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 4558 lines...]
[INFO] Tez ... SUCCESS [ 0.026 s]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 56:43 min
[INFO] Finished at: 2016-02-28T14:48:12+00:00
[INFO] Final Memory: 70M/1161M
[INFO]
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12790359/TEZ-2756.001.patch
against master revision 18398c8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1524//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1524//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
b0ad63c195aba4d3282449c87be21d7fb826c4df logged out
==
==
Finished build.
==
==
Archiving artifacts
[description-setter] Description set: TEZ-2756
Recording test results
Email was triggered for: Success
Sending email for trigger: Success
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2756) MergeManager close should not try merging files on close if invoked after a shuffle exception
[
https://issues.apache.org/jira/browse/TEZ-2756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171060#comment-15171060
]
TezQA commented on TEZ-2756:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12790359/TEZ-2756.001.patch
against master revision 18398c8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1524//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1524//console
This message is automatically generated.
> MergeManager close should not try merging files on close if invoked after a
> shuffle exception
> -
>
> Key: TEZ-2756
> URL: https://issues.apache.org/jira/browse/TEZ-2756
> Project: Apache Tez
> Issue Type: Improvement
>Affects Versions: 0.5.0
>Reporter: Siddharth Seth
>Assignee: Tsuyoshi Ozawa
> Attachments: TEZ-2756.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (TEZ-1944) OOM when using tez.runtime.shuffle.memory-to-memory.enable=true
[
https://issues.apache.org/jira/browse/TEZ-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan resolved TEZ-1944.
---
Resolution: Duplicate
Thanks [~ozawa]. Checked this usecase after TEZ-1911 and it works fine.
Marking this as duplicate of TEZ-1911.
> OOM when using tez.runtime.shuffle.memory-to-memory.enable=true
> ---
>
> Key: TEZ-1944
> URL: https://issues.apache.org/jira/browse/TEZ-1944
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>
> ETL workload at 10 TB scale
> {code}
> $HIVE_HOME/bin/hive -hiveconf tez.runtime.io.sort.factor=200 --hiveconf
> hive.tez.auto.reducer.parallelism=true --hiveconf
> tez.am.heartbeat.interval-ms.max=20 --hiveconf tez.runtime.io.sort.mb=1200
> --hiveconf tez.runtime.sort.threads=2 --hiveconf hive.tez.container.size=4096
> --hiveconf tez.runtime.shuffle.memory-to-memory.enable=true --hiveconf
> tez.runtime.shuffle.memory-to-memory.segments=4
> create table testData as select
> ss_sold_date_sk,ss_sold_time_sk,ss_item_sk,ss_customer_sk,ss_quantity,ss_sold_date
> from store_sales distribute by ss_sold_date;
> {code}
> Tasks starts throwing OOM with the following exception.
> {code}
> aused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.hadoop.io.BoundedByteArrayOutputStream.(BoundedByteArrayOutputStream.java:56)
> at
> org.apache.hadoop.io.BoundedByteArrayOutputStream.(BoundedByteArrayOutputStream.java:46)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput.(MapOutput.java:81)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput.createMemoryMapOutput(MapOutput.java:133)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.unconditionalReserve(MergeManager.java:397)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.access$300(MergeManager.java:74)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager$IntermediateMemoryToMemoryMerger.merge(MergeManager.java:530)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeThread.run(MergeThread.java:89)
> , errorMessage=Shuffle Runner
> Failed:org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError:
> error in shuffle in MemToMemMerger [Map_1]
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.call(Shuffle.java:349)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.call(Shuffle.java:328)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Created this to track issues related to memory to memory merging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-2580) Remove VertexManagerPlugin#setVertexParallelism with VertexManagerPlugin#reconfigureVertex
[ https://issues.apache.org/jira/browse/TEZ-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-2580: Attachment: TEZ-2580.001.patch Attaching a patch to remove setVertexParallelism API. > Remove VertexManagerPlugin#setVertexParallelism with > VertexManagerPlugin#reconfigureVertex > -- > > Key: TEZ-2580 > URL: https://issues.apache.org/jira/browse/TEZ-2580 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha >Priority: Blocker > Attachments: TEZ-2580.001.patch > > > This was deprecated in 0.7. Should be replaced with reconfigureVertex() - > change of name - to make it consistent with other reconfigureVertex() API's. > Should be done just close to release to enabled Hive to continue to build/use > master of Tez. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2756) MergeManager close should not try merging files on close if invoked after a shuffle exception
[ https://issues.apache.org/jira/browse/TEZ-2756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-2756: Attachment: TEZ-2756.001.patch Attaching v1 patch. > MergeManager close should not try merging files on close if invoked after a > shuffle exception > - > > Key: TEZ-2756 > URL: https://issues.apache.org/jira/browse/TEZ-2756 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.5.0 >Reporter: Siddharth Seth > Attachments: TEZ-2756.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1944) OOM when using tez.runtime.shuffle.memory-to-memory.enable=true
[
https://issues.apache.org/jira/browse/TEZ-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171028#comment-15171028
]
Tsuyoshi Ozawa commented on TEZ-1944:
-
Can we close this as duplicated problem of TEZ-1911?
> OOM when using tez.runtime.shuffle.memory-to-memory.enable=true
> ---
>
> Key: TEZ-1944
> URL: https://issues.apache.org/jira/browse/TEZ-1944
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>
> ETL workload at 10 TB scale
> {code}
> $HIVE_HOME/bin/hive -hiveconf tez.runtime.io.sort.factor=200 --hiveconf
> hive.tez.auto.reducer.parallelism=true --hiveconf
> tez.am.heartbeat.interval-ms.max=20 --hiveconf tez.runtime.io.sort.mb=1200
> --hiveconf tez.runtime.sort.threads=2 --hiveconf hive.tez.container.size=4096
> --hiveconf tez.runtime.shuffle.memory-to-memory.enable=true --hiveconf
> tez.runtime.shuffle.memory-to-memory.segments=4
> create table testData as select
> ss_sold_date_sk,ss_sold_time_sk,ss_item_sk,ss_customer_sk,ss_quantity,ss_sold_date
> from store_sales distribute by ss_sold_date;
> {code}
> Tasks starts throwing OOM with the following exception.
> {code}
> aused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.hadoop.io.BoundedByteArrayOutputStream.(BoundedByteArrayOutputStream.java:56)
> at
> org.apache.hadoop.io.BoundedByteArrayOutputStream.(BoundedByteArrayOutputStream.java:46)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput.(MapOutput.java:81)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MapOutput.createMemoryMapOutput(MapOutput.java:133)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.unconditionalReserve(MergeManager.java:397)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.access$300(MergeManager.java:74)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager$IntermediateMemoryToMemoryMerger.merge(MergeManager.java:530)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeThread.run(MergeThread.java:89)
> , errorMessage=Shuffle Runner
> Failed:org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError:
> error in shuffle in MemToMemMerger [Map_1]
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.call(Shuffle.java:349)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.call(Shuffle.java:328)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Created this to track issues related to memory to memory merging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3009) Errors that occur during container task acquisition are not logged
[ https://issues.apache.org/jira/browse/TEZ-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171026#comment-15171026 ] Tsuyoshi Ozawa commented on TEZ-3009: - Kicking CI again. [~sseth], the patch looks good to me(non-binding). Could you check it? > Errors that occur during container task acquisition are not logged > -- > > Key: TEZ-3009 > URL: https://issues.apache.org/jira/browse/TEZ-3009 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: TEZ-3009.001.patch, TEZ-3009.002.patch > > > If TezChild encounters an error while trying to obtain a task the error will > be silently handled. This results in a mysterious shutdown of containers > with no cause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3014) OOM during Shuffle in JDK 8
[ https://issues.apache.org/jira/browse/TEZ-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171024#comment-15171024 ] Tsuyoshi Ozawa commented on TEZ-3014: - Linking to TEZ-3114 and TEZ-3115 since they seems to be related. > OOM during Shuffle in JDK 8 > --- > > Key: TEZ-3014 > URL: https://issues.apache.org/jira/browse/TEZ-3014 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3120) Remove TaskCommContext.getCurrentDagName, Identifier
[ https://issues.apache.org/jira/browse/TEZ-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171023#comment-15171023 ] Tsuyoshi Ozawa commented on TEZ-3120: - Opps, overlooking that assignee is Sidd. Feel free to use the patch. Thanks > Remove TaskCommContext.getCurrentDagName, Identifier > > > Key: TEZ-3120 > URL: https://issues.apache.org/jira/browse/TEZ-3120 > Project: Apache Tez > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-3120.001.patch > > > Incompatible change. Remove prior to the 0.8.3 release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-3120) Remove TaskCommContext.getCurrentDagName, Identifier
[ https://issues.apache.org/jira/browse/TEZ-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated TEZ-3120: Attachment: TEZ-3120.001.patch Attaching a patch to remove the deprecated method. > Remove TaskCommContext.getCurrentDagName, Identifier > > > Key: TEZ-3120 > URL: https://issues.apache.org/jira/browse/TEZ-3120 > Project: Apache Tez > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: TEZ-3120.001.patch > > > Incompatible change. Remove prior to the 0.8.3 release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
