[jira] [Commented] (TEZ-145) Support a combiner processor that can run non-local to map/reduce nodes
[ https://issues.apache.org/jira/browse/TEZ-145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491620#comment-14491620 ] Bikas Saha commented on TEZ-145: Right, like I said in a previous comment, the transducer needs to maintain partition boundaries while doing its work for this to be useful. This would need a single vertex with its vertex manager (to do the rack aware grouping) and a single EdgeManager that does the custom routing from grouped maps to their transducer. This would be a fairly asymmetric edge because of arbitrary groupings. Not sure why pipelining is required for this? Essentially we are introducing another vertex that is doing some partial grouping. In fact, it could be done today in user land without Tez changes and we should be able to accomplish that in this jira. The completed map outputs are being aggregated transparently for the next stage. Where Tez support could be needed for efficiency is to be able to short circuit this stage. Lets say, the vertex manager figures out that the transducer stage is going to be useless (given data distribution, size and latency). Then Tez could allow removing this stage from the DAG so that the real consumer stage can be started with no overhead. > Support a combiner processor that can run non-local to map/reduce nodes > --- > > Key: TEZ-145 > URL: https://issues.apache.org/jira/browse/TEZ-145 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-145.2.patch, WIP-TEZ-145-001.patch > > > For aggregate operators that can benefit by running in multi-level trees, > support of being able to run a combiner in a non-local mode would allow > performance efficiencies to be gained by running a combiner at a rack-level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-145) Support a combiner processor that can run non-local to map/reduce nodes
[ https://issues.apache.org/jira/browse/TEZ-145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491654#comment-14491654 ] Gopal V commented on TEZ-145: - bq. Not sure why pipelining is required for this? Essentially we are introducing another vertex that is doing some partial grouping. Pipelining is perhaps the wrong word, but generating multiple partial events from a single task is the performance improvement. That allows a locality missed task to go from start to finish without moving any data across. Reducers tend to be spread across racks (which might be another thing to tune for when using a small queue on a big cluster). Transducers don't have that problem, they're optional - to avoid producing a cross-rack fetches for an intermediate stage, the AM would have to excise any task that runs with sub-optimal locality after the runtime-expansion & scheduling of the downstream with TaskSpecs. The addition of the state machine to handle partial/final events in order for Inputs, effectively allows forwarding of the entire "split" as-is with a final=false flag, making that an entirely local decision in the Task rather than re-adjusting vertex fan-out during runtime. > Support a combiner processor that can run non-local to map/reduce nodes > --- > > Key: TEZ-145 > URL: https://issues.apache.org/jira/browse/TEZ-145 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-145.2.patch, WIP-TEZ-145-001.patch > > > For aggregate operators that can benefit by running in multi-level trees, > support of being able to run a combiner in a non-local mode would allow > performance efficiencies to be gained by running a combiner at a rack-level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2275) TEZ UI: Make data loading faster and caching better
[ https://issues.apache.org/jira/browse/TEZ-2275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-2275: Attachment: TEZ-2275.2.patch Thanks [~pramachandran]. Please find a fresh patch without duplicate code. > TEZ UI: Make data loading faster and caching better > --- > > Key: TEZ-2275 > URL: https://issues.apache.org/jira/browse/TEZ-2275 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-2275.1.patch, TEZ-2275.2.patch > > > Loading: > # Remove counter serialization for all entities to make loading faster. > # Ensure that counter tables works with unserialized data. > # Ensure display of counter data in entity tables. > Caching: > # Separate cache for each table: Refreshing vertex->tasks shouldn't partially > update values displayed in All Tasks table. > # Refreshing an entity, shouldn't change the respective record in any tables. > # Ensure all data displayed in a table are from a single time frame. (No part > of the table should be partially updated by actions elsewhere) > # Data cached would be limited to one DAG at a time. (To limit the values > stores in the browser side) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2252) Tez UI Graphical view looks wrong in some cases
[ https://issues.apache.org/jira/browse/TEZ-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-2252: Attachment: TEZ-2252.1.patch Overlapping of outputs Graph 1: graph-view.js : Issue was with dummy replace logic, fixed. More symmetric display Graph 2: data-processor.js : Instead of the linear map, we use centric/radial map to plot a more symmetric and readable graph. [~jeagles] [~pramachandran] Please review and help to get the patch in. > Tez UI Graphical view looks wrong in some cases > --- > > Key: TEZ-2252 > URL: https://issues.apache.org/jira/browse/TEZ-2252 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Rohini Palaniswamy >Assignee: Sreenath Somarajapuram > Attachments: Graph1.dot, Graph1.png, Graph2.dot, Graph2.png, > TEZ-2252.1.patch > > > The information in the .dot file is correct and script runs fine. But the > Tez UI Graphical view shows that output is being written from multiple > vertices into one sink. Actually it is writing to multiple sinks (checking > the html elements in Firebug), but the sink circles all overlap one another > with exact coordinates and the tool tip only shows for the top one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2275) TEZ UI: Make data loading faster and caching better
[
https://issues.apache.org/jira/browse/TEZ-2275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491734#comment-14491734
]
TezQA commented on TEZ-2275:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12724842/TEZ-2275.2.patch
against master revision bd9b8d9.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/445//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/445//console
This message is automatically generated.
> TEZ UI: Make data loading faster and caching better
> ---
>
> Key: TEZ-2275
> URL: https://issues.apache.org/jira/browse/TEZ-2275
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Sreenath Somarajapuram
>Assignee: Sreenath Somarajapuram
> Attachments: TEZ-2275.1.patch, TEZ-2275.2.patch
>
>
> Loading:
> # Remove counter serialization for all entities to make loading faster.
> # Ensure that counter tables works with unserialized data.
> # Ensure display of counter data in entity tables.
> Caching:
> # Separate cache for each table: Refreshing vertex->tasks shouldn't partially
> update values displayed in All Tasks table.
> # Refreshing an entity, shouldn't change the respective record in any tables.
> # Ensure all data displayed in a table are from a single time frame. (No part
> of the table should be partially updated by actions elsewhere)
> # Data cached would be limited to one DAG at a time. (To limit the values
> stores in the browser side)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Failed: TEZ-2275 PreCommit Build #445
Jira: https://issues.apache.org/jira/browse/TEZ-2275
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/445/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2781 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12724842/TEZ-2275.2.patch
against master revision bd9b8d9.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/445//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/445//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
1a3756fb1670b1059e6ccd3668b180e03035ee95 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #441
Archived 44 artifacts
Archive block size is 32768
Received 8 blocks and 2483472 bytes
Compression is 9.5%
Took 0.97 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2234) Add API for statistics information - allow vertex managers to get output size per source vertex
[ https://issues.apache.org/jira/browse/TEZ-2234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491748#comment-14491748 ] Rohini Palaniswamy commented on TEZ-2234: - Thanks [~bikassaha] > Add API for statistics information - allow vertex managers to get output size > per source vertex > --- > > Key: TEZ-2234 > URL: https://issues.apache.org/jira/browse/TEZ-2234 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha >Assignee: Bikas Saha > Fix For: 0.7.0 > > Attachments: TEZ-2234.1.patch, TEZ-2234.2.patch, TEZ-2234.3.patch, > TEZ-2234.4.patch > > > Vertex managers may need per source vertex output stats to make > reconfiguration decisions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2300) TezClient.stop() takes a lot of time or does not work sometimes
[
https://issues.apache.org/jira/browse/TEZ-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491781#comment-14491781
]
Rohini Palaniswamy commented on TEZ-2300:
-
This one was a case of AM lingering long due to posting to ATS. [~jlowe], did
configure the tez.yarn.ats.max.events.per.batch to 500 in tez-site.xml to make
the problem better.
2015-04-12 23:24:42,040 [Timer-0] INFO
org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status:
status=RUNNING, progress=TotalTasks: 51890 Succeeded: 0 Running: 50 Failed: 0
Killed: 0, diagnostics=, counters=null
Did Ctlr+C here on the pig client.
AM DAG log:
2015-04-12 23:24:52,015 INFO [AsyncDispatcher event handler] app.DAGAppMaster:
DAG completed, dagId=dag_1428329756093_325099_1, dagState=KILLED
2015-04-12 23:24:52,015 INFO [AsyncDispatcher event handler]
common.TezUtilsInternal: Redirecting log file based on addend:
dag_1428329756093_325099_1_post
In the dag_1428329756093_325099_1_post log (attached to the jira as well)
{code}
2015-04-12 23:24:57,029 INFO [AMShutdownThread] ats.ATSHistoryLoggingService:
Stopping ATSService, eventQueueBacklog=17927
2015-04-12 23:25:25,466 WARN [AMShutdownThread] ats.ATSHistoryLoggingService:
ATSService being stopped, eventQueueBacklog=17927, maxTimeLeftToFlush=-1,
waitForever=true
Lot of ATS put errors
2015-04-12 23:32:53,197 INFO [AMShutdownThread] ats.ATSHistoryLoggingService:
Event queue empty, stopping ATS Service
2015-04-12 23:32:53,200 INFO [DelayedContainerManager]
rm.YarnTaskSchedulerService: AllocatedContainerManager Thread interrupted
2015-04-12 23:32:53,203 INFO [AMShutdownThread] rm.YarnTaskSchedulerService:
Unregistering application from RM, exitStatus=SUCCEEDED, exitMessage=Session
stats:submittedDAGs=1, successfulDAGs=0, failedDAGs=0, killedDAGs=1
,
trackingURL=bassniumtan-jt1.tan.ygrid.yahoo.com:4080/tez/#/?appid=application_1428329756093_325099
2015-04-12 23:32:53,210 INFO [AMShutdownThread] impl.AMRMClientImpl: Waiting
for application to be successfully unregistered.
2015-04-12 23:32:53,314 INFO [AMShutdownThread] rm.YarnTaskSchedulerService:
Successfully unregistered application from RM
2015-04-12 23:32:53,315 INFO [AMRM Callback Handler Thread]
impl.AMRMClientAsyncImpl: Interrupted while waiting for queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274)
2015-04-12 23:32:53,316 INFO [AMShutdownThread] ipc.Server: Stopping server on
51921
2015-04-12 23:32:53,319 INFO [IPC Server listener on 51921] ipc.Server:
Stopping IPC Server listener on 51921
2015-04-12 23:32:53,319 INFO [AMShutdownThread] ipc.Server: Stopping server on
50500
2015-04-12 23:32:53,320 INFO [IPC Server listener on 50500] ipc.Server:
Stopping IPC Server listener on 50500
2015-04-12 23:32:53,320 INFO [IPC Server Responder] ipc.Server: Stopping IPC
Server Responder
2015-04-12 23:32:53,320 INFO [IPC Server Responder] ipc.Server: Stopping IPC
Server Responder
2015-04-12 23:32:53,324 INFO [AMShutdownThread] app.DAGAppMaster: Completed
deletion of tez scratch data dir,
path=hdfs://bassniumtan-nn1.tan.ygrid.yahoo.com:8020/tmp/temp-1464028011/.tez/application_1428329756093_325099
2015-04-12 23:32:53,324 INFO [AMShutdownThread] app.DAGAppMaster: Exiting
DAGAppMaster..GoodBye!
2015-04-12 23:32:53,325 INFO [Thread-1] app.DAGAppMaster:
DAGAppMasterShutdownHook invoked
{code}
Jstack still running thread on AM :
{code}
"AMShutdownThread" prio=10 tid=0x0f3ad800 nid=0x2b0e runnable [0x04e4]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0xde56b700> (a java.io.BufferedInputStream)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:633)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:579)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1322)
- locked <0xde42d320> (a sun.net.www.protocol.http.HttpURLConnection)
at
java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
at
com.sun.jersey.client.ur
[jira] [Updated] (TEZ-2300) TezClient.stop() takes a lot of time or does not work sometimes
[ https://issues.apache.org/jira/browse/TEZ-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohini Palaniswamy updated TEZ-2300: Attachment: syslog_dag_1428329756093_325099_1_post > TezClient.stop() takes a lot of time or does not work sometimes > --- > > Key: TEZ-2300 > URL: https://issues.apache.org/jira/browse/TEZ-2300 > Project: Apache Tez > Issue Type: Bug >Reporter: Rohini Palaniswamy > Attachments: syslog_dag_1428329756093_325099_1_post > > > Noticed this with a couple of pig scripts which were not behaving well (AM > close to OOM, etc) and even with some that were running fine. Pig calls > Tezclient.stop() in shutdown hook. Ctrl+C to the pig script either exits > immediately or is hung. In both cases it either takes a long time for the > yarn application to go to KILLED state. Many times I just end up calling yarn > application -kill separately after waiting for 5 mins or more for it to get > killed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-2300) TezClient.stop() takes a lot of time or does not work sometimes
[
https://issues.apache.org/jira/browse/TEZ-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491781#comment-14491781
]
Rohini Palaniswamy edited comment on TEZ-2300 at 4/12/15 11:45 PM:
---
This one was a case of AM lingering long due to posting to ATS. [~jlowe], had
configured the tez.yarn.ats.max.events.per.batch to 500 in tez-site.xml as we
had earlier noticed that even on normal shutdown AM was lingering for a lot of
time when there are lot of events. But in this case it is still worse taking
more than 5 mins. I believe it is due to ATS put errors and retries.
2015-04-12 23:24:42,040 [Timer-0] INFO
org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status:
status=RUNNING, progress=TotalTasks: 51890 Succeeded: 0 Running: 50 Failed: 0
Killed: 0, diagnostics=, counters=null
Did Ctlr+C here on the pig client.
AM DAG log:
2015-04-12 23:24:52,015 INFO [AsyncDispatcher event handler] app.DAGAppMaster:
DAG completed, dagId=dag_1428329756093_325099_1, dagState=KILLED
2015-04-12 23:24:52,015 INFO [AsyncDispatcher event handler]
common.TezUtilsInternal: Redirecting log file based on addend:
dag_1428329756093_325099_1_post
In the dag_1428329756093_325099_1_post log (attached to the jira as well)
{code}
2015-04-12 23:24:57,029 INFO [AMShutdownThread] ats.ATSHistoryLoggingService:
Stopping ATSService, eventQueueBacklog=17927
2015-04-12 23:25:25,466 WARN [AMShutdownThread] ats.ATSHistoryLoggingService:
ATSService being stopped, eventQueueBacklog=17927, maxTimeLeftToFlush=-1,
waitForever=true
Lot of ATS put errors
2015-04-12 23:32:53,197 INFO [AMShutdownThread] ats.ATSHistoryLoggingService:
Event queue empty, stopping ATS Service
2015-04-12 23:32:53,200 INFO [DelayedContainerManager]
rm.YarnTaskSchedulerService: AllocatedContainerManager Thread interrupted
2015-04-12 23:32:53,203 INFO [AMShutdownThread] rm.YarnTaskSchedulerService:
Unregistering application from RM, exitStatus=SUCCEEDED, exitMessage=Session
stats:submittedDAGs=1, successfulDAGs=0, failedDAGs=0, killedDAGs=1
,
trackingURL=bassniumtan-jt1.tan.ygrid.yahoo.com:4080/tez/#/?appid=application_1428329756093_325099
2015-04-12 23:32:53,210 INFO [AMShutdownThread] impl.AMRMClientImpl: Waiting
for application to be successfully unregistered.
2015-04-12 23:32:53,314 INFO [AMShutdownThread] rm.YarnTaskSchedulerService:
Successfully unregistered application from RM
2015-04-12 23:32:53,315 INFO [AMRM Callback Handler Thread]
impl.AMRMClientAsyncImpl: Interrupted while waiting for queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274)
2015-04-12 23:32:53,316 INFO [AMShutdownThread] ipc.Server: Stopping server on
51921
2015-04-12 23:32:53,319 INFO [IPC Server listener on 51921] ipc.Server:
Stopping IPC Server listener on 51921
2015-04-12 23:32:53,319 INFO [AMShutdownThread] ipc.Server: Stopping server on
50500
2015-04-12 23:32:53,320 INFO [IPC Server listener on 50500] ipc.Server:
Stopping IPC Server listener on 50500
2015-04-12 23:32:53,320 INFO [IPC Server Responder] ipc.Server: Stopping IPC
Server Responder
2015-04-12 23:32:53,320 INFO [IPC Server Responder] ipc.Server: Stopping IPC
Server Responder
2015-04-12 23:32:53,324 INFO [AMShutdownThread] app.DAGAppMaster: Completed
deletion of tez scratch data dir,
path=hdfs://bassniumtan-nn1.tan.ygrid.yahoo.com:8020/tmp/temp-1464028011/.tez/application_1428329756093_325099
2015-04-12 23:32:53,324 INFO [AMShutdownThread] app.DAGAppMaster: Exiting
DAGAppMaster..GoodBye!
2015-04-12 23:32:53,325 INFO [Thread-1] app.DAGAppMaster:
DAGAppMasterShutdownHook invoked
{code}
Jstack still running thread on AM :
{code}
"AMShutdownThread" prio=10 tid=0x0f3ad800 nid=0x2b0e runnable [0x04e4]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0xde56b700> (a java.io.BufferedInputStream)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:633)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:579)
at
sun.net.www.protocol.http.Ht
[jira] [Comment Edited] (TEZ-2300) TezClient.stop() takes a lot of time or does not work sometimes
[
https://issues.apache.org/jira/browse/TEZ-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491781#comment-14491781
]
Rohini Palaniswamy edited comment on TEZ-2300 at 4/12/15 11:46 PM:
---
This one was a case of AM lingering long due to posting to ATS. [~jlowe], had
configured the tez.yarn.ats.max.events.per.batch to 500 in tez-site.xml as we
had earlier noticed that even on normal shutdown AM was lingering for a lot of
time when there are lot of events. But in this case it is still worse taking
more than 5 mins. I believe it is due to ATS put errors and retries.
2015-04-12 23:24:42,040 [Timer-0] INFO
org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status:
status=RUNNING, progress=TotalTasks: 51890 Succeeded: 0 Running: 50 Failed: 0
Killed: 0, diagnostics=, counters=null
Did Ctlr+C here on the pig client.
AM DAG log:
2015-04-12 23:24:52,015 INFO [AsyncDispatcher event handler] app.DAGAppMaster:
DAG completed, dagId=dag_1428329756093_325099_1, dagState=KILLED
2015-04-12 23:24:52,015 INFO [AsyncDispatcher event handler]
common.TezUtilsInternal: Redirecting log file based on addend:
dag_1428329756093_325099_1_post
In the dag_1428329756093_325099_1_post log (attached to the jira as well)
{code}
2015-04-12 23:24:57,029 INFO [AMShutdownThread] ats.ATSHistoryLoggingService:
Stopping ATSService, eventQueueBacklog=17927
2015-04-12 23:25:25,466 WARN [AMShutdownThread] ats.ATSHistoryLoggingService:
ATSService being stopped, eventQueueBacklog=17927, maxTimeLeftToFlush=-1,
waitForever=true
Lot of ATS put errors
2015-04-12 23:32:53,197 INFO [AMShutdownThread] ats.ATSHistoryLoggingService:
Event queue empty, stopping ATS Service
2015-04-12 23:32:53,200 INFO [DelayedContainerManager]
rm.YarnTaskSchedulerService: AllocatedContainerManager Thread interrupted
2015-04-12 23:32:53,203 INFO [AMShutdownThread] rm.YarnTaskSchedulerService:
Unregistering application from RM, exitStatus=SUCCEEDED, exitMessage=Session
stats:submittedDAGs=1, successfulDAGs=0, failedDAGs=0, killedDAGs=1
,
trackingURL=bassniumtan-jt1.tan.ygrid.yahoo.com:4080/tez/#/?appid=application_1428329756093_325099
2015-04-12 23:32:53,210 INFO [AMShutdownThread] impl.AMRMClientImpl: Waiting
for application to be successfully unregistered.
2015-04-12 23:32:53,314 INFO [AMShutdownThread] rm.YarnTaskSchedulerService:
Successfully unregistered application from RM
2015-04-12 23:32:53,315 INFO [AMRM Callback Handler Thread]
impl.AMRMClientAsyncImpl: Interrupted while waiting for queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274)
2015-04-12 23:32:53,316 INFO [AMShutdownThread] ipc.Server: Stopping server on
51921
2015-04-12 23:32:53,319 INFO [IPC Server listener on 51921] ipc.Server:
Stopping IPC Server listener on 51921
2015-04-12 23:32:53,319 INFO [AMShutdownThread] ipc.Server: Stopping server on
50500
2015-04-12 23:32:53,320 INFO [IPC Server listener on 50500] ipc.Server:
Stopping IPC Server listener on 50500
2015-04-12 23:32:53,320 INFO [IPC Server Responder] ipc.Server: Stopping IPC
Server Responder
2015-04-12 23:32:53,320 INFO [IPC Server Responder] ipc.Server: Stopping IPC
Server Responder
2015-04-12 23:32:53,324 INFO [AMShutdownThread] app.DAGAppMaster: Completed
deletion of tez scratch data dir,
path=hdfs://bassniumtan-nn1.tan.ygrid.yahoo.com:8020/tmp/temp-1464028011/.tez/application_1428329756093_325099
2015-04-12 23:32:53,324 INFO [AMShutdownThread] app.DAGAppMaster: Exiting
DAGAppMaster..GoodBye!
2015-04-12 23:32:53,325 INFO [Thread-1] app.DAGAppMaster:
DAGAppMasterShutdownHook invoked
{code}
Jstack still running thread on AM :
{code}
"AMShutdownThread" prio=10 tid=0x0f3ad800 nid=0x2b0e runnable [0x04e4]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0xde56b700> (a java.io.BufferedInputStream)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:633)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:579)
at
sun.net.www.protocol.http.Ht
[jira] [Updated] (TEZ-714) OutputCommitters should not run in the main AM dispatcher thread
[ https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-714: --- Attachment: TEZ-714-13.patch Upload new patch, minor update on e2e test. > OutputCommitters should not run in the main AM dispatcher thread > > > Key: TEZ-714 > URL: https://issues.apache.org/jira/browse/TEZ-714 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Jeff Zhang >Priority: Critical > Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-10.patch, > TEZ-714-11.patch, TEZ-714-12.patch, TEZ-714-13.patch, TEZ-714-2.patch, > TEZ-714-3.patch, TEZ-714-4.patch, TEZ-714-5.patch, TEZ-714-6.patch, > TEZ-714-7.patch, TEZ-714-8.patch, TEZ-714-9.patch, Vertex_2.pdf > > > Follow up jira from TEZ-41. > 1) If there's multiple OutputCommitters on a Vertex, they can be run in > parallel. > 2) Running an OutputCommitter in the main thread blocks all other event > handling, w.r.t the DAG, and causes the event queue to back up. > 3) This should also cover shared commits that happen in the DAG. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-714 PreCommit Build #446
Jira: https://issues.apache.org/jira/browse/TEZ-714
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/446/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 19 lines...]
No emails were triggered.
[PreCommit-TEZ-Build] $ /bin/bash /tmp/hudson8532898926665538831.sh
Running in Jenkins mode
==
==
Testing patch for TEZ-714.
==
==
HEAD is now at bd9b8d9 TEZ-2234. Add API for statistics information - allow
vertex managers to get output size per source vertex (bikas)
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
Current branch master is up to date.
TEZ-714 patch is being downloaded at Mon Apr 13 01:30:18 UTC 2015 from
http://issues.apache.org/jira/secure/attachment/12724852/TEZ-714-13.patch
The patch does not appear to apply with p0 to p2
PATCH APPLICATION FAILED
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12724852/TEZ-714-13.patch
against master revision bd9b8d9.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/446//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
46fc6f7f9ecf45cdfab641ffb84f01ae5961fef4 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
No tests ran.
[jira] [Commented] (TEZ-714) OutputCommitters should not run in the main AM dispatcher thread
[
https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491826#comment-14491826
]
TezQA commented on TEZ-714:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12724852/TEZ-714-13.patch
against master revision bd9b8d9.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/446//console
This message is automatically generated.
> OutputCommitters should not run in the main AM dispatcher thread
>
>
> Key: TEZ-714
> URL: https://issues.apache.org/jira/browse/TEZ-714
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Jeff Zhang
>Priority: Critical
> Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-10.patch,
> TEZ-714-11.patch, TEZ-714-12.patch, TEZ-714-13.patch, TEZ-714-2.patch,
> TEZ-714-3.patch, TEZ-714-4.patch, TEZ-714-5.patch, TEZ-714-6.patch,
> TEZ-714-7.patch, TEZ-714-8.patch, TEZ-714-9.patch, Vertex_2.pdf
>
>
> Follow up jira from TEZ-41.
> 1) If there's multiple OutputCommitters on a Vertex, they can be run in
> parallel.
> 2) Running an OutputCommitter in the main thread blocks all other event
> handling, w.r.t the DAG, and causes the event queue to back up.
> 3) This should also cover shared commits that happen in the DAG.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2305) MR compatibility sleep job fails with IOException: Undefined job output-path
[
https://issues.apache.org/jira/browse/TEZ-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491832#comment-14491832
]
Jeff Zhang commented on TEZ-2305:
-
[~hitesh] Thanks for your clarification. +1 on your last patch, and also
verify that the patch resolve TEZ-2271 .
> MR compatibility sleep job fails with IOException: Undefined job output-path
>
>
> Key: TEZ-2305
> URL: https://issues.apache.org/jira/browse/TEZ-2305
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Tassapol Athiapinya
>Priority: Critical
> Attachments: TEZ-2305-3.patch, TEZ-2305-4.patch, TEZ-2305.1.patch,
> TEZ-2305.2.patch
>
>
> Running MR sleep job has an IOException.
> {code}
> 15/04/09 20:52:25 INFO mapreduce.Job: Job job_1428612196442_0002 failed with
> state FAILED due to: Vertex failed, vertexName=initialmap,
> vertexId=vertex_1428612196442_0002_1_00, diagnostics=[Task failed,
> taskId=task_1428612196442_0002_1_00_01, diagnostics=[TaskAttempt 0
> failed, info=[Error: Failure while running task:java.io.IOException:
> Undefined job output-path
> at
> org.apache.hadoop.mapred.FileOutputFormat.getTaskOutputPath(FileOutputFormat.java:248)
> at
> org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:121)
> at
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:401)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:436)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:415)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ], TaskAttempt 1 failed, info=[Error: Failure while running
> task:java.io.IOException: Undefined job output-path
> at
> org.apache.hadoop.mapred.FileOutputFormat.getTaskOutputPath(FileOutputFormat.java:248)
> at
> org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:121)
> at
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:401)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:436)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:415)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ], TaskAttempt 2 failed, info=[Error: Failure while running
> task:java.io.IOException: Undefined job output-path
> at
> org.apache.hadoop.mapred.FileOutputFormat.getTaskOutputPath(FileOutputFormat.java:248)
> at
> org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:121)
> at
> org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:401)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:436)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:415)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ], TaskAttempt 3 failed, info=[Error: Failure while running
> task:java.io.IOException: Undefined
[jira] [Updated] (TEZ-714) OutputCommitters should not run in the main AM dispatcher thread
[ https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-714: --- Attachment: TEZ-714-14.patch Rebase the patch > OutputCommitters should not run in the main AM dispatcher thread > > > Key: TEZ-714 > URL: https://issues.apache.org/jira/browse/TEZ-714 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Jeff Zhang >Priority: Critical > Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-10.patch, > TEZ-714-11.patch, TEZ-714-12.patch, TEZ-714-13.patch, TEZ-714-14.patch, > TEZ-714-2.patch, TEZ-714-3.patch, TEZ-714-4.patch, TEZ-714-5.patch, > TEZ-714-6.patch, TEZ-714-7.patch, TEZ-714-8.patch, TEZ-714-9.patch, > Vertex_2.pdf > > > Follow up jira from TEZ-41. > 1) If there's multiple OutputCommitters on a Vertex, they can be run in > parallel. > 2) Running an OutputCommitter in the main thread blocks all other event > handling, w.r.t the DAG, and causes the event queue to back up. > 3) This should also cover shared commits that happen in the DAG. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-714 PreCommit Build #447
Jira: https://issues.apache.org/jira/browse/TEZ-714
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/447/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 19 lines...]
No emails were triggered.
[PreCommit-TEZ-Build] $ /bin/bash /tmp/hudson8093622342086229950.sh
Running in Jenkins mode
==
==
Testing patch for TEZ-714.
==
==
HEAD is now at bd9b8d9 TEZ-2234. Add API for statistics information - allow
vertex managers to get output size per source vertex (bikas)
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
Current branch master is up to date.
TEZ-714 patch is being downloaded at Mon Apr 13 02:10:19 UTC 2015 from
http://issues.apache.org/jira/secure/attachment/12724853/TEZ-714-14.patch
The patch does not appear to apply with p0 to p2
PATCH APPLICATION FAILED
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12724853/TEZ-714-14.patch
against master revision bd9b8d9.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/447//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
90627676be18301787524281b4c76922bbc7d8f4 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
No tests ran.
[jira] [Commented] (TEZ-714) OutputCommitters should not run in the main AM dispatcher thread
[
https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491848#comment-14491848
]
TezQA commented on TEZ-714:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12724853/TEZ-714-14.patch
against master revision bd9b8d9.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/447//console
This message is automatically generated.
> OutputCommitters should not run in the main AM dispatcher thread
>
>
> Key: TEZ-714
> URL: https://issues.apache.org/jira/browse/TEZ-714
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Jeff Zhang
>Priority: Critical
> Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-10.patch,
> TEZ-714-11.patch, TEZ-714-12.patch, TEZ-714-13.patch, TEZ-714-14.patch,
> TEZ-714-2.patch, TEZ-714-3.patch, TEZ-714-4.patch, TEZ-714-5.patch,
> TEZ-714-6.patch, TEZ-714-7.patch, TEZ-714-8.patch, TEZ-714-9.patch,
> Vertex_2.pdf
>
>
> Follow up jira from TEZ-41.
> 1) If there's multiple OutputCommitters on a Vertex, they can be run in
> parallel.
> 2) Running an OutputCommitter in the main thread blocks all other event
> handling, w.r.t the DAG, and causes the event queue to back up.
> 3) This should also cover shared commits that happen in the DAG.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-714) OutputCommitters should not run in the main AM dispatcher thread
[ https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-714: --- Attachment: TEZ-714-15.patch Rebase it again. > OutputCommitters should not run in the main AM dispatcher thread > > > Key: TEZ-714 > URL: https://issues.apache.org/jira/browse/TEZ-714 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Jeff Zhang >Priority: Critical > Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-10.patch, > TEZ-714-11.patch, TEZ-714-12.patch, TEZ-714-13.patch, TEZ-714-14.patch, > TEZ-714-15.patch, TEZ-714-2.patch, TEZ-714-3.patch, TEZ-714-4.patch, > TEZ-714-5.patch, TEZ-714-6.patch, TEZ-714-7.patch, TEZ-714-8.patch, > TEZ-714-9.patch, Vertex_2.pdf > > > Follow up jira from TEZ-41. > 1) If there's multiple OutputCommitters on a Vertex, they can be run in > parallel. > 2) Running an OutputCommitter in the main thread blocks all other event > handling, w.r.t the DAG, and causes the event queue to back up. > 3) This should also cover shared commits that happen in the DAG. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2225) Remove instances of LOG.isDebugEnabled
[ https://issues.apache.org/jira/browse/TEZ-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491860#comment-14491860 ] Bikas Saha commented on TEZ-2225: - If done correctly, sl4j takes care of the perf aspects - please see http://www.slf4j.org/faq.html#logging_performance However not all cases are covered by the sl4j technique - e.g. when some method is being called to generate the argument to the log statement. [~vasanthkumar] Could you please take a look at http://stackoverflow.com/questions/8444266/even-with-slf4j-should-you-guard-your-logging and verify that we are making the correct changes when the logger argument is invoking a method instead of accessing a local variable? Committing the patch would be easy once you have confirmed that. Thanks. Trying to trigger another build to verify findbugs. > Remove instances of LOG.isDebugEnabled > -- > > Key: TEZ-2225 > URL: https://issues.apache.org/jira/browse/TEZ-2225 > Project: Apache Tez > Issue Type: Improvement >Reporter: Vasanth kumar RJ >Assignee: Vasanth kumar RJ >Priority: Minor > Labels: performance > Attachments: TEZ-2225.1.patch, TEZ-2225.2.patch, TEZ-2225.3.patch > > > Remove LOG.isDebugEnabled() and use parameterized debug logging -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-714) OutputCommitters should not run in the main AM dispatcher thread
[
https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491862#comment-14491862
]
Bikas Saha commented on TEZ-714:
Using wrong parameter?
{code}+// add data sinks for v2
+for (int i=0;i OutputCommitters should not run in the main AM dispatcher thread
>
>
> Key: TEZ-714
> URL: https://issues.apache.org/jira/browse/TEZ-714
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Jeff Zhang
>Priority: Critical
> Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-10.patch,
> TEZ-714-11.patch, TEZ-714-12.patch, TEZ-714-13.patch, TEZ-714-14.patch,
> TEZ-714-15.patch, TEZ-714-2.patch, TEZ-714-3.patch, TEZ-714-4.patch,
> TEZ-714-5.patch, TEZ-714-6.patch, TEZ-714-7.patch, TEZ-714-8.patch,
> TEZ-714-9.patch, Vertex_2.pdf
>
>
> Follow up jira from TEZ-41.
> 1) If there's multiple OutputCommitters on a Vertex, they can be run in
> parallel.
> 2) Running an OutputCommitter in the main thread blocks all other event
> handling, w.r.t the DAG, and causes the event queue to back up.
> 3) This should also cover shared commits that happen in the DAG.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2304) InvalidStateTransitonException TA_SCHEDULE at START_WAIT during recovery
[
https://issues.apache.org/jira/browse/TEZ-2304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491867#comment-14491867
]
Jeff Zhang commented on TEZ-2304:
-
[~jlowe] Could you attach the full app logs ?
Initial doubt is that the attemptId may be set incorrectly.
{code}
2015-04-09 20:05:42,055 INFO [AsyncDispatcher event handler]
impl.TaskAttemptImpl: attempt_1428329756093_168563_1_00_006728_1 TaskAttempt
Transitioned from NEW to KILLED due to event TA_RECOVER
2015-04-09 20:05:45,748 INFO [AsyncDispatcher event handler]
impl.TaskAttemptImpl: attempt_1428329756093_168563_1_00_006728_1 TaskAttempt
Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
{code}
> InvalidStateTransitonException TA_SCHEDULE at START_WAIT during recovery
>
>
> Key: TEZ-2304
> URL: https://issues.apache.org/jira/browse/TEZ-2304
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Jason Lowe
>
> I saw a Tez AM throw a few InvalidStateTransitonException (sic) instances
> during recovery complaining about TA_SCHEDULE arriving at the START_WAIT
> state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (TEZ-2304) InvalidStateTransitonException TA_SCHEDULE at START_WAIT during recovery
[
https://issues.apache.org/jira/browse/TEZ-2304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491867#comment-14491867
]
Jeff Zhang edited comment on TEZ-2304 at 4/13/15 3:11 AM:
--
[~jlowe] Could you attach the full app logs ?
was (Author: zjffdu):
[~jlowe] Could you attach the full app logs ?
Initial doubt is that the attemptId may be set incorrectly.
{code}
2015-04-09 20:05:42,055 INFO [AsyncDispatcher event handler]
impl.TaskAttemptImpl: attempt_1428329756093_168563_1_00_006728_1 TaskAttempt
Transitioned from NEW to KILLED due to event TA_RECOVER
2015-04-09 20:05:45,748 INFO [AsyncDispatcher event handler]
impl.TaskAttemptImpl: attempt_1428329756093_168563_1_00_006728_1 TaskAttempt
Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
{code}
> InvalidStateTransitonException TA_SCHEDULE at START_WAIT during recovery
>
>
> Key: TEZ-2304
> URL: https://issues.apache.org/jira/browse/TEZ-2304
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Jason Lowe
>
> I saw a Tez AM throw a few InvalidStateTransitonException (sic) instances
> during recovery complaining about TA_SCHEDULE arriving at the START_WAIT
> state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (TEZ-714) OutputCommitters should not run in the main AM dispatcher thread
[ https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-714: --- Attachment: TEZ-714-16.patch [~bikassaha] Thanks for pointing out this, correct it in the new patch. > OutputCommitters should not run in the main AM dispatcher thread > > > Key: TEZ-714 > URL: https://issues.apache.org/jira/browse/TEZ-714 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Jeff Zhang >Priority: Critical > Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-10.patch, > TEZ-714-11.patch, TEZ-714-12.patch, TEZ-714-13.patch, TEZ-714-14.patch, > TEZ-714-15.patch, TEZ-714-16.patch, TEZ-714-2.patch, TEZ-714-3.patch, > TEZ-714-4.patch, TEZ-714-5.patch, TEZ-714-6.patch, TEZ-714-7.patch, > TEZ-714-8.patch, TEZ-714-9.patch, Vertex_2.pdf > > > Follow up jira from TEZ-41. > 1) If there's multiple OutputCommitters on a Vertex, they can be run in > parallel. > 2) Running an OutputCommitter in the main thread blocks all other event > handling, w.r.t the DAG, and causes the event queue to back up. > 3) This should also cover shared commits that happen in the DAG. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2252) Tez UI Graphical view looks wrong in some cases
[
https://issues.apache.org/jira/browse/TEZ-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491874#comment-14491874
]
TezQA commented on TEZ-2252:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12724844/TEZ-2252.1.patch
against master revision bd9b8d9.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.client.TestTezClientUtils
org.apache.tez.common.TestReflectionUtils
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/449//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/449//console
This message is automatically generated.
> Tez UI Graphical view looks wrong in some cases
> ---
>
> Key: TEZ-2252
> URL: https://issues.apache.org/jira/browse/TEZ-2252
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Reporter: Rohini Palaniswamy
>Assignee: Sreenath Somarajapuram
> Attachments: Graph1.dot, Graph1.png, Graph2.dot, Graph2.png,
> TEZ-2252.1.patch
>
>
> The information in the .dot file is correct and script runs fine. But the
> Tez UI Graphical view shows that output is being written from multiple
> vertices into one sink. Actually it is writing to multiple sinks (checking
> the html elements in Firebug), but the sink circles all overlap one another
> with exact coordinates and the tool tip only shows for the top one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Failed: TEZ-714 PreCommit Build #448
Jira: https://issues.apache.org/jira/browse/TEZ-714
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/448/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2629 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12724854/TEZ-714-15.patch
against master revision bd9b8d9.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestTezJobs
org.apache.tez.mapreduce.TestMRRJobs
org.apache.tez.mapreduce.TestMRRJobsDAGApi
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/448//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/448//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
2b1c9b4e28df14faa71410acb68f266ab8927656 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #441
Archived 44 artifacts
Archive block size is 32768
Received 4 blocks and 2612619 bytes
Compression is 4.8%
Took 0.94 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
8 tests failed.
REGRESSION: org.apache.tez.mapreduce.TestMRRJobs.testFailingAttempt
Error Message:
test timed out after 6 milliseconds
Stack Trace:
java.lang.Exception: test timed out after 6 milliseconds
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1359)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1317)
at
org.apache.tez.mapreduce.TestMRRJobs.testFailingAttempt(TestMRRJobs.java:251)
REGRESSION:
org.apache.tez.mapreduce.TestMRRJobsDAGApi.testMultipleMRRSleepJobViaSession
Error Message:
com.google.protobuf.ServiceException:
org.apache.hadoop.ipc.RemoteException(org.apache.tez.dag.api.TezException):
Duplicate dag name 'testMRRSleepJobDagSubmit-116'
at org.apache.tez.dag.app.DAGAppMaster.startDAG(DAGAppMaster.java:2025)
at
org.apache.tez.dag.app.DAGAppMaster.submitDAGToAppMaster(DAGAppMaster.java:1147)
at
org.apache.tez.dag.api.client.DAGClientHandler.submitDAG(DAGClientHandler.java:118)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.submitDAG(DAGClientAMProtocolBlockingPBServerImpl.java:163)
at
org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7471)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
Stack Trace:
org.apache.tez.dag.api.TezException: com.google.protobuf.ServiceException:
org.apache.hadoop.ipc.RemoteException(org.apache.tez.dag.api.TezException):
Duplicate dag name 'testMRRSleepJobDagSubmit-116'
at org.apache.tez.dag.app.DAGAppMaster.startDAG(DAGAppMaster
[jira] [Commented] (TEZ-714) OutputCommitters should not run in the main AM dispatcher thread
[
https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491876#comment-14491876
]
TezQA commented on TEZ-714:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12724854/TEZ-714-15.patch
against master revision bd9b8d9.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestTezJobs
org.apache.tez.mapreduce.TestMRRJobs
org.apache.tez.mapreduce.TestMRRJobsDAGApi
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/448//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/448//console
This message is automatically generated.
> OutputCommitters should not run in the main AM dispatcher thread
>
>
> Key: TEZ-714
> URL: https://issues.apache.org/jira/browse/TEZ-714
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Jeff Zhang
>Priority: Critical
> Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-10.patch,
> TEZ-714-11.patch, TEZ-714-12.patch, TEZ-714-13.patch, TEZ-714-14.patch,
> TEZ-714-15.patch, TEZ-714-16.patch, TEZ-714-2.patch, TEZ-714-3.patch,
> TEZ-714-4.patch, TEZ-714-5.patch, TEZ-714-6.patch, TEZ-714-7.patch,
> TEZ-714-8.patch, TEZ-714-9.patch, Vertex_2.pdf
>
>
> Follow up jira from TEZ-41.
> 1) If there's multiple OutputCommitters on a Vertex, they can be run in
> parallel.
> 2) Running an OutputCommitter in the main thread blocks all other event
> handling, w.r.t the DAG, and causes the event queue to back up.
> 3) This should also cover shared commits that happen in the DAG.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Failed: TEZ-2252 PreCommit Build #449
Jira: https://issues.apache.org/jira/browse/TEZ-2252
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/449/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 1992 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12724844/TEZ-2252.1.patch
against master revision bd9b8d9.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.client.TestTezClientUtils
org.apache.tez.common.TestReflectionUtils
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/449//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/449//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
8a3976367c6f304b356887c8adb7a07a24cc865e logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #441
Archived 44 artifacts
Archive block size is 32768
Received 8 blocks and 2432019 bytes
Compression is 9.7%
Took 1.3 sec
[description-setter] Could not determine description.
Recording test results
Publish JUnit test result report is waiting for a checkpoint on
PreCommit-TEZ-Build #448
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
9 tests failed.
REGRESSION:
org.apache.tez.client.TestTezClientUtils.validateSetTezJarLocalResourcesDefinedExistingDirectory
Error Message:
test timed out after 5000 milliseconds
Stack Trace:
java.lang.Exception: test timed out after 5000 milliseconds
at java.net.PlainDatagramSocketImpl.receive0(Native Method)
at
java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:145)
at java.net.DatagramSocket.receive(DatagramSocket.java:786)
at com.sun.jndi.dns.DnsClient.doUdpQuery(DnsClient.java:416)
at com.sun.jndi.dns.DnsClient.query(DnsClient.java:210)
at com.sun.jndi.dns.Resolver.query(Resolver.java:81)
at com.sun.jndi.dns.DnsContext.c_getAttributes(DnsContext.java:430)
at
com.sun.jndi.toolkit.ctx.ComponentDirContext.p_getAttributes(ComponentDirContext.java:231)
at
com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.getAttributes(PartialCompositeDirContext.java:139)
at
com.sun.jndi.toolkit.url.GenericURLDirContext.getAttributes(GenericURLDirContext.java:103)
at
sun.security.krb5.KrbServiceLocator.getKerberosService(KrbServiceLocator.java:87)
at sun.security.krb5.Config.checkRealm(Config.java:1295)
at sun.security.krb5.Config.getRealmFromDNS(Config.java:1268)
at sun.security.krb5.Config.getDefaultRealm(Config.java:1162)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:84)
at
org.apache.hadoop.security.authentication.util.KerberosName.(KerberosName.java:86)
at
org.apach
Failed: TEZ-714 PreCommit Build #450
Jira: https://issues.apache.org/jira/browse/TEZ-714 Build: https://builds.apache.org/job/PreCommit-TEZ-Build/450/ ### ## LAST 60 LINES OF THE CONSOLE ### Started by remote host 127.0.0.1 Building remotely on H0 (Hadoop Tez) in workspace /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url https://git-wip-us.apache.org/repos/asf/tez.git > # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/tez.git > git --version # timeout=10 FATAL: Failed to fetch from https://git-wip-us.apache.org/repos/asf/tez.git hudson.plugins.git.GitException: Failed to fetch from https://git-wip-us.apache.org/repos/asf/tez.git at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:647) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:889) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:914) at hudson.model.AbstractProject.checkout(AbstractProject.java:1252) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:615) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:524) at hudson.model.Run.execute(Run.java:1706) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:232) Caused by: hudson.plugins.git.GitException: Failed to connect to https://git-wip-us.apache.org/repos/asf/tez.git (exception: java.net.UnknownHostException: git-wip-us.apache.org: Name or service not known) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.checkCredentials(CliGitAPIImpl.java:2253) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1169) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:85) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:280) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:328) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ### ## FAILED TESTS (if any) ## No tests ran.
[jira] [Updated] (TEZ-714) OutputCommitters should not run in the main AM dispatcher thread
[ https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-714: --- Attachment: (was: TEZ-714-16.patch) > OutputCommitters should not run in the main AM dispatcher thread > > > Key: TEZ-714 > URL: https://issues.apache.org/jira/browse/TEZ-714 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Jeff Zhang >Priority: Critical > Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-10.patch, > TEZ-714-11.patch, TEZ-714-12.patch, TEZ-714-13.patch, TEZ-714-14.patch, > TEZ-714-15.patch, TEZ-714-16.patch, TEZ-714-2.patch, TEZ-714-3.patch, > TEZ-714-4.patch, TEZ-714-5.patch, TEZ-714-6.patch, TEZ-714-7.patch, > TEZ-714-8.patch, TEZ-714-9.patch, Vertex_2.pdf > > > Follow up jira from TEZ-41. > 1) If there's multiple OutputCommitters on a Vertex, they can be run in > parallel. > 2) Running an OutputCommitter in the main thread blocks all other event > handling, w.r.t the DAG, and causes the event queue to back up. > 3) This should also cover shared commits that happen in the DAG. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-714) OutputCommitters should not run in the main AM dispatcher thread
[ https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-714: --- Attachment: TEZ-714-16.patch > OutputCommitters should not run in the main AM dispatcher thread > > > Key: TEZ-714 > URL: https://issues.apache.org/jira/browse/TEZ-714 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Jeff Zhang >Priority: Critical > Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-10.patch, > TEZ-714-11.patch, TEZ-714-12.patch, TEZ-714-13.patch, TEZ-714-14.patch, > TEZ-714-15.patch, TEZ-714-16.patch, TEZ-714-2.patch, TEZ-714-3.patch, > TEZ-714-4.patch, TEZ-714-5.patch, TEZ-714-6.patch, TEZ-714-7.patch, > TEZ-714-8.patch, TEZ-714-9.patch, Vertex_2.pdf > > > Follow up jira from TEZ-41. > 1) If there's multiple OutputCommitters on a Vertex, they can be run in > parallel. > 2) Running an OutputCommitter in the main thread blocks all other event > handling, w.r.t the DAG, and causes the event queue to back up. > 3) This should also cover shared commits that happen in the DAG. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-714) OutputCommitters should not run in the main AM dispatcher thread
[
https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491897#comment-14491897
]
TezQA commented on TEZ-714:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12724863/TEZ-714-16.patch
against master revision bd9b8d9.
{color:red}-1 patch{color}. master compilation may be broken.
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/451//console
This message is automatically generated.
> OutputCommitters should not run in the main AM dispatcher thread
>
>
> Key: TEZ-714
> URL: https://issues.apache.org/jira/browse/TEZ-714
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Jeff Zhang
>Priority: Critical
> Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-10.patch,
> TEZ-714-11.patch, TEZ-714-12.patch, TEZ-714-13.patch, TEZ-714-14.patch,
> TEZ-714-15.patch, TEZ-714-16.patch, TEZ-714-2.patch, TEZ-714-3.patch,
> TEZ-714-4.patch, TEZ-714-5.patch, TEZ-714-6.patch, TEZ-714-7.patch,
> TEZ-714-8.patch, TEZ-714-9.patch, Vertex_2.pdf
>
>
> Follow up jira from TEZ-41.
> 1) If there's multiple OutputCommitters on a Vertex, they can be run in
> parallel.
> 2) Running an OutputCommitter in the main thread blocks all other event
> handling, w.r.t the DAG, and causes the event queue to back up.
> 3) This should also cover shared commits that happen in the DAG.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Failed: TEZ-714 PreCommit Build #451
Jira: https://issues.apache.org/jira/browse/TEZ-714
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/451/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 35 lines...]
TEZ-714 patch is being downloaded at Mon Apr 13 04:10:37 UTC 2015 from
http://issues.apache.org/jira/secure/attachment/12724863/TEZ-714-16.patch
==
==
Pre-build master to verify master stability and javac warnings
==
==
Compiling /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build
/home/jenkins/tools/maven/latest/bin/mvn clean test -DskipTests -Ptest-patch >
/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/../patchprocess/masterJavacWarnings.txt
2>&1
master compilation is broken?
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12724863/TEZ-714-16.patch
against master revision bd9b8d9.
{color:red}-1 patch{color}. master compilation may be broken.
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/451//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
7caea134b03657b7b9109f9c8031818fb5dfecee logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #441
Archived 1 artifacts
Archive block size is 32768
Received 0 blocks and 134660 bytes
Compression is 0.0%
Took 65 ms
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
No tests ran.
[jira] [Updated] (TEZ-2119) Counter for launched containers
[ https://issues.apache.org/jira/browse/TEZ-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2119: Assignee: (was: Jeff Zhang) > Counter for launched containers > --- > > Key: TEZ-2119 > URL: https://issues.apache.org/jira/browse/TEZ-2119 > Project: Apache Tez > Issue Type: Improvement >Reporter: Rohini Palaniswamy > > org.apache.tez.common.counters.DAGCounter > NUM_SUCCEEDED_TASKS=32976 > TOTAL_LAUNCHED_TASKS=32976 > OTHER_LOCAL_TASKS=2 > DATA_LOCAL_TASKS=9147 > RACK_LOCAL_TASKS=23761 > It would be very nice to have TOTAL_LAUNCHED_CONTAINERS counter added to > this. The difference between TOTAL_LAUNCHED_CONTAINERS and > TOTAL_LAUNCHED_TASKS should make it easy to see how much container reuse is > happening. It is very hard to find out now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (TEZ-2016) Tez UI Dag View Fit and Finish
[ https://issues.apache.org/jira/browse/TEZ-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram reassigned TEZ-2016: --- Assignee: Sreenath Somarajapuram > Tez UI Dag View Fit and Finish > -- > > Key: TEZ-2016 > URL: https://issues.apache.org/jira/browse/TEZ-2016 > Project: Apache Tez > Issue Type: Improvement > Components: UI >Reporter: Jonathan Eagles >Assignee: Sreenath Somarajapuram > Attachments: Check in checkbox is not centered.png, Customize vetex > should not have first item selected by default.png, Edge Id name.png, > Gaussian Blur is not Radial.png, Green success icon not flush with edge.png, > Wording and functionality.png > > > TEZ-1973 has added a major contribution to the Tez UI. The core of the > functionality is there and is committed to community to reduce effort on > keeping patch up-to-date and to allow community exposure to the contributions. > A few fit and finish items have been identified and need to be addressed. > - Gaussian blur around circles is not a circle (see attachment) > - Check in checkbox is not centered (see attachment) > - Success icon is not flush (see attachment) > - First item in customize vertex should not be selected by default (see > attachment) > - Edge Id name is not correct (see attachment) > - Double click: was never able to see this working. If functionality is > there please update the wording (no attachment) > -Click and drag on graph should not route tez ui to another page (no > attachment)- > - Prefer to see dag view view height to extend to bottom of screen (no > attachment) > # Make source and sink bubble apt for longer text > # Bring dag back into view if it go out of view port > # Add ellipsis for clipped text in tooltip > # Use grab & grabbing cursors to indicate that user can pan the diagram > # Put Edge Source Class before Edge Destination Class > # BUG-31616 - #3 > # Make source/sink link tooltip better -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2016) Tez UI Dag View Fit and Finish
[ https://issues.apache.org/jira/browse/TEZ-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-2016: Description: TEZ-1973 has added a major contribution to the Tez UI. The core of the functionality is there and is committed to community to reduce effort on keeping patch up-to-date and to allow community exposure to the contributions. A few fit and finish items have been identified and need to be addressed. - Gaussian blur around circles is not a circle (see attachment) - Check in checkbox is not centered (see attachment) - Success icon is not flush (see attachment) - First item in customize vertex should not be selected by default (see attachment) - Edge Id name is not correct (see attachment) - Double click: was never able to see this working. If functionality is there please update the wording (no attachment) -Click and drag on graph should not route tez ui to another page (no attachment)- - Prefer to see dag view view height to extend to bottom of screen (no attachment) # Make source and sink bubble apt for longer text # Bring dag back into view if it go out of view port # Add ellipsis for clipped text in tooltip # Use grab & grabbing cursors to indicate that user can pan the diagram # Put Edge Source Class before Edge Destination Class # BUG-31616 - #3 # Make source/sink link tooltip better was: TEZ-1973 has added a major contribution to the Tez UI. The core of the functionality is there and is committed to community to reduce effort on keeping patch up-to-date and to allow community exposure to the contributions. A few fit and finish items have been identified and need to be addressed. - Gaussian blur around circles is not a circle (see attachment) - Check in checkbox is not centered (see attachment) - Success icon is not flush (see attachment) - First item in customize vertex should not be selected by default (see attachment) - Edge Id name is not correct (see attachment) - Double click: was never able to see this working. If functionality is there please update the wording (no attachment) -Click and drag on graph should not route tez ui to another page (no attachment)- - Prefer to see dag view view height to extend to bottom of screen (no attachment) > Tez UI Dag View Fit and Finish > -- > > Key: TEZ-2016 > URL: https://issues.apache.org/jira/browse/TEZ-2016 > Project: Apache Tez > Issue Type: Improvement > Components: UI >Reporter: Jonathan Eagles > Attachments: Check in checkbox is not centered.png, Customize vetex > should not have first item selected by default.png, Edge Id name.png, > Gaussian Blur is not Radial.png, Green success icon not flush with edge.png, > Wording and functionality.png > > > TEZ-1973 has added a major contribution to the Tez UI. The core of the > functionality is there and is committed to community to reduce effort on > keeping patch up-to-date and to allow community exposure to the contributions. > A few fit and finish items have been identified and need to be addressed. > - Gaussian blur around circles is not a circle (see attachment) > - Check in checkbox is not centered (see attachment) > - Success icon is not flush (see attachment) > - First item in customize vertex should not be selected by default (see > attachment) > - Edge Id name is not correct (see attachment) > - Double click: was never able to see this working. If functionality is > there please update the wording (no attachment) > -Click and drag on graph should not route tez ui to another page (no > attachment)- > - Prefer to see dag view view height to extend to bottom of screen (no > attachment) > # Make source and sink bubble apt for longer text > # Bring dag back into view if it go out of view port > # Add ellipsis for clipped text in tooltip > # Use grab & grabbing cursors to indicate that user can pan the diagram > # Put Edge Source Class before Edge Destination Class > # BUG-31616 - #3 > # Make source/sink link tooltip better -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2016) Tez UI Dag View Fit and Finish
[ https://issues.apache.org/jira/browse/TEZ-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-2016: Attachment: TEZ-2016.1.patch [~jeagles] [~pramachandran] Please review and help to get the patch in. > Tez UI Dag View Fit and Finish > -- > > Key: TEZ-2016 > URL: https://issues.apache.org/jira/browse/TEZ-2016 > Project: Apache Tez > Issue Type: Improvement > Components: UI >Reporter: Jonathan Eagles >Assignee: Sreenath Somarajapuram > Attachments: Check in checkbox is not centered.png, Customize vetex > should not have first item selected by default.png, Edge Id name.png, > Gaussian Blur is not Radial.png, Green success icon not flush with edge.png, > TEZ-2016.1.patch, Wording and functionality.png > > > TEZ-1973 has added a major contribution to the Tez UI. The core of the > functionality is there and is committed to community to reduce effort on > keeping patch up-to-date and to allow community exposure to the contributions. > A few fit and finish items have been identified and need to be addressed. > - Gaussian blur around circles is not a circle (see attachment) > - Check in checkbox is not centered (see attachment) > - Success icon is not flush (see attachment) > - First item in customize vertex should not be selected by default (see > attachment) > - Edge Id name is not correct (see attachment) > - Double click: was never able to see this working. If functionality is > there please update the wording (no attachment) > -Click and drag on graph should not route tez ui to another page (no > attachment)- > - Prefer to see dag view view height to extend to bottom of screen (no > attachment) > # Make source and sink bubble apt for longer text > # Bring dag back into view if it go out of view port > # Add ellipsis for clipped text in tooltip > # Use grab & grabbing cursors to indicate that user can pan the diagram > # Put Edge Source Class before Edge Destination Class > # BUG-31616 - #3 > # Make source/sink link tooltip better -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2016) Tez UI Dag View Fit and Finish
[ https://issues.apache.org/jira/browse/TEZ-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491947#comment-14491947 ] Sreenath Somarajapuram commented on TEZ-2016: - Almost all the individual changes are one liners, hence thought to club them together. Extra items in the patch was added to the description. - Gaussian blur: have changed the source and sink into round rects. - Check in checkbox is not centered: Will be resolved on changing column selector to a bootstrap model. - First item in customize vertex should not be selected by default: Please elaborate, selected items are the default columns displayed in vertex table, and represent the basic data some would be looking for. Should we remove them? - Double click: Is the new message good? > Tez UI Dag View Fit and Finish > -- > > Key: TEZ-2016 > URL: https://issues.apache.org/jira/browse/TEZ-2016 > Project: Apache Tez > Issue Type: Improvement > Components: UI >Reporter: Jonathan Eagles >Assignee: Sreenath Somarajapuram > Attachments: Check in checkbox is not centered.png, Customize vetex > should not have first item selected by default.png, Edge Id name.png, > Gaussian Blur is not Radial.png, Green success icon not flush with edge.png, > TEZ-2016.1.patch, Wording and functionality.png > > > TEZ-1973 has added a major contribution to the Tez UI. The core of the > functionality is there and is committed to community to reduce effort on > keeping patch up-to-date and to allow community exposure to the contributions. > A few fit and finish items have been identified and need to be addressed. > - Gaussian blur around circles is not a circle (see attachment) > - Check in checkbox is not centered (see attachment) > - Success icon is not flush (see attachment) > - First item in customize vertex should not be selected by default (see > attachment) > - Edge Id name is not correct (see attachment) > - Double click: was never able to see this working. If functionality is > there please update the wording (no attachment) > -Click and drag on graph should not route tez ui to another page (no > attachment)- > - Prefer to see dag view view height to extend to bottom of screen (no > attachment) > # Make source and sink bubble apt for longer text > # Bring dag back into view if it go out of view port > # Add ellipsis for clipped text in tooltip > # Use grab & grabbing cursors to indicate that user can pan the diagram > # Put Edge Source Class before Edge Destination Class > # BUG-31616 - #3 > # Make source/sink link tooltip better -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-2016 PreCommit Build #452
Jira: https://issues.apache.org/jira/browse/TEZ-2016
Build: https://builds.apache.org/job/PreCommit-TEZ-Build/452/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 2770 lines...]
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12724879/TEZ-2016.1.patch
against master revision bd9b8d9.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/452//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/452//console
This message is automatically generated.
==
==
Adding comment to Jira.
==
==
Comment added.
5312a74f552694b03b2395ede8db9123e997d304 logged out
==
==
Finished build.
==
==
Build step 'Execute shell' marked build as failure
Archiving artifacts
Sending artifact delta relative to PreCommit-TEZ-Build #441
Archived 44 artifacts
Archive block size is 32768
Received 8 blocks and 2486960 bytes
Compression is 9.5%
Took 1.9 sec
[description-setter] Could not determine description.
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
###
## FAILED TESTS (if any)
##
All tests passed
[jira] [Commented] (TEZ-2016) Tez UI Dag View Fit and Finish
[
https://issues.apache.org/jira/browse/TEZ-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491987#comment-14491987
]
TezQA commented on TEZ-2016:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12724879/TEZ-2016.1.patch
against master revision bd9b8d9.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/452//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/452//console
This message is automatically generated.
> Tez UI Dag View Fit and Finish
> --
>
> Key: TEZ-2016
> URL: https://issues.apache.org/jira/browse/TEZ-2016
> Project: Apache Tez
> Issue Type: Improvement
> Components: UI
>Reporter: Jonathan Eagles
>Assignee: Sreenath Somarajapuram
> Attachments: Check in checkbox is not centered.png, Customize vetex
> should not have first item selected by default.png, Edge Id name.png,
> Gaussian Blur is not Radial.png, Green success icon not flush with edge.png,
> TEZ-2016.1.patch, Wording and functionality.png
>
>
> TEZ-1973 has added a major contribution to the Tez UI. The core of the
> functionality is there and is committed to community to reduce effort on
> keeping patch up-to-date and to allow community exposure to the contributions.
> A few fit and finish items have been identified and need to be addressed.
> - Gaussian blur around circles is not a circle (see attachment)
> - Check in checkbox is not centered (see attachment)
> - Success icon is not flush (see attachment)
> - First item in customize vertex should not be selected by default (see
> attachment)
> - Edge Id name is not correct (see attachment)
> - Double click: was never able to see this working. If functionality is
> there please update the wording (no attachment)
> -Click and drag on graph should not route tez ui to another page (no
> attachment)-
> - Prefer to see dag view view height to extend to bottom of screen (no
> attachment)
> # Make source and sink bubble apt for longer text
> # Bring dag back into view if it go out of view port
> # Add ellipsis for clipped text in tooltip
> # Use grab & grabbing cursors to indicate that user can pan the diagram
> # Put Edge Source Class before Edge Destination Class
> # BUG-31616 - #3
> # Make source/sink link tooltip better
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
