[jira] [Commented] (TEZ-145) Support a combiner processor that can run non-local to map/reduce nodes
[ https://issues.apache.org/jira/browse/TEZ-145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491620#comment-14491620 ] Bikas Saha commented on TEZ-145: Right, like I said in a previous comment, the transducer needs to maintain partition boundaries while doing its work for this to be useful. This would need a single vertex with its vertex manager (to do the rack aware grouping) and a single EdgeManager that does the custom routing from grouped maps to their transducer. This would be a fairly asymmetric edge because of arbitrary groupings. Not sure why pipelining is required for this? Essentially we are introducing another vertex that is doing some partial grouping. In fact, it could be done today in user land without Tez changes and we should be able to accomplish that in this jira. The completed map outputs are being aggregated transparently for the next stage. Where Tez support could be needed for efficiency is to be able to short circuit this stage. Lets say, the vertex manager figures out that the transducer stage is going to be useless (given data distribution, size and latency). Then Tez could allow removing this stage from the DAG so that the real consumer stage can be started with no overhead. > Support a combiner processor that can run non-local to map/reduce nodes > --- > > Key: TEZ-145 > URL: https://issues.apache.org/jira/browse/TEZ-145 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-145.2.patch, WIP-TEZ-145-001.patch > > > For aggregate operators that can benefit by running in multi-level trees, > support of being able to run a combiner in a non-local mode would allow > performance efficiencies to be gained by running a combiner at a rack-level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-145) Support a combiner processor that can run non-local to map/reduce nodes
[ https://issues.apache.org/jira/browse/TEZ-145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491654#comment-14491654 ] Gopal V commented on TEZ-145: - bq. Not sure why pipelining is required for this? Essentially we are introducing another vertex that is doing some partial grouping. Pipelining is perhaps the wrong word, but generating multiple partial events from a single task is the performance improvement. That allows a locality missed task to go from start to finish without moving any data across. Reducers tend to be spread across racks (which might be another thing to tune for when using a small queue on a big cluster). Transducers don't have that problem, they're optional - to avoid producing a cross-rack fetches for an intermediate stage, the AM would have to excise any task that runs with sub-optimal locality after the runtime-expansion & scheduling of the downstream with TaskSpecs. The addition of the state machine to handle partial/final events in order for Inputs, effectively allows forwarding of the entire "split" as-is with a final=false flag, making that an entirely local decision in the Task rather than re-adjusting vertex fan-out during runtime. > Support a combiner processor that can run non-local to map/reduce nodes > --- > > Key: TEZ-145 > URL: https://issues.apache.org/jira/browse/TEZ-145 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Tsuyoshi Ozawa > Attachments: TEZ-145.2.patch, WIP-TEZ-145-001.patch > > > For aggregate operators that can benefit by running in multi-level trees, > support of being able to run a combiner in a non-local mode would allow > performance efficiencies to be gained by running a combiner at a rack-level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2275) TEZ UI: Make data loading faster and caching better
[ https://issues.apache.org/jira/browse/TEZ-2275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-2275: Attachment: TEZ-2275.2.patch Thanks [~pramachandran]. Please find a fresh patch without duplicate code. > TEZ UI: Make data loading faster and caching better > --- > > Key: TEZ-2275 > URL: https://issues.apache.org/jira/browse/TEZ-2275 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-2275.1.patch, TEZ-2275.2.patch > > > Loading: > # Remove counter serialization for all entities to make loading faster. > # Ensure that counter tables works with unserialized data. > # Ensure display of counter data in entity tables. > Caching: > # Separate cache for each table: Refreshing vertex->tasks shouldn't partially > update values displayed in All Tasks table. > # Refreshing an entity, shouldn't change the respective record in any tables. > # Ensure all data displayed in a table are from a single time frame. (No part > of the table should be partially updated by actions elsewhere) > # Data cached would be limited to one DAG at a time. (To limit the values > stores in the browser side) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2252) Tez UI Graphical view looks wrong in some cases
[ https://issues.apache.org/jira/browse/TEZ-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-2252: Attachment: TEZ-2252.1.patch Overlapping of outputs Graph 1: graph-view.js : Issue was with dummy replace logic, fixed. More symmetric display Graph 2: data-processor.js : Instead of the linear map, we use centric/radial map to plot a more symmetric and readable graph. [~jeagles] [~pramachandran] Please review and help to get the patch in. > Tez UI Graphical view looks wrong in some cases > --- > > Key: TEZ-2252 > URL: https://issues.apache.org/jira/browse/TEZ-2252 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Rohini Palaniswamy >Assignee: Sreenath Somarajapuram > Attachments: Graph1.dot, Graph1.png, Graph2.dot, Graph2.png, > TEZ-2252.1.patch > > > The information in the .dot file is correct and script runs fine. But the > Tez UI Graphical view shows that output is being written from multiple > vertices into one sink. Actually it is writing to multiple sinks (checking > the html elements in Firebug), but the sink circles all overlap one another > with exact coordinates and the tool tip only shows for the top one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2275) TEZ UI: Make data loading faster and caching better
[ https://issues.apache.org/jira/browse/TEZ-2275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491734#comment-14491734 ] TezQA commented on TEZ-2275: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12724842/TEZ-2275.2.patch against master revision bd9b8d9. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/445//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/445//console This message is automatically generated. > TEZ UI: Make data loading faster and caching better > --- > > Key: TEZ-2275 > URL: https://issues.apache.org/jira/browse/TEZ-2275 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-2275.1.patch, TEZ-2275.2.patch > > > Loading: > # Remove counter serialization for all entities to make loading faster. > # Ensure that counter tables works with unserialized data. > # Ensure display of counter data in entity tables. > Caching: > # Separate cache for each table: Refreshing vertex->tasks shouldn't partially > update values displayed in All Tasks table. > # Refreshing an entity, shouldn't change the respective record in any tables. > # Ensure all data displayed in a table are from a single time frame. (No part > of the table should be partially updated by actions elsewhere) > # Data cached would be limited to one DAG at a time. (To limit the values > stores in the browser side) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-2275 PreCommit Build #445
Jira: https://issues.apache.org/jira/browse/TEZ-2275 Build: https://builds.apache.org/job/PreCommit-TEZ-Build/445/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 2781 lines...] {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12724842/TEZ-2275.2.patch against master revision bd9b8d9. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/445//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/445//console This message is automatically generated. == == Adding comment to Jira. == == Comment added. 1a3756fb1670b1059e6ccd3668b180e03035ee95 logged out == == Finished build. == == Build step 'Execute shell' marked build as failure Archiving artifacts Sending artifact delta relative to PreCommit-TEZ-Build #441 Archived 44 artifacts Archive block size is 32768 Received 8 blocks and 2483472 bytes Compression is 9.5% Took 0.97 sec [description-setter] Could not determine description. Recording test results Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## All tests passed
[jira] [Commented] (TEZ-2234) Add API for statistics information - allow vertex managers to get output size per source vertex
[ https://issues.apache.org/jira/browse/TEZ-2234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491748#comment-14491748 ] Rohini Palaniswamy commented on TEZ-2234: - Thanks [~bikassaha] > Add API for statistics information - allow vertex managers to get output size > per source vertex > --- > > Key: TEZ-2234 > URL: https://issues.apache.org/jira/browse/TEZ-2234 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha >Assignee: Bikas Saha > Fix For: 0.7.0 > > Attachments: TEZ-2234.1.patch, TEZ-2234.2.patch, TEZ-2234.3.patch, > TEZ-2234.4.patch > > > Vertex managers may need per source vertex output stats to make > reconfiguration decisions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2300) TezClient.stop() takes a lot of time or does not work sometimes
[ https://issues.apache.org/jira/browse/TEZ-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491781#comment-14491781 ] Rohini Palaniswamy commented on TEZ-2300: - This one was a case of AM lingering long due to posting to ATS. [~jlowe], did configure the tez.yarn.ats.max.events.per.batch to 500 in tez-site.xml to make the problem better. 2015-04-12 23:24:42,040 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 51890 Succeeded: 0 Running: 50 Failed: 0 Killed: 0, diagnostics=, counters=null Did Ctlr+C here on the pig client. AM DAG log: 2015-04-12 23:24:52,015 INFO [AsyncDispatcher event handler] app.DAGAppMaster: DAG completed, dagId=dag_1428329756093_325099_1, dagState=KILLED 2015-04-12 23:24:52,015 INFO [AsyncDispatcher event handler] common.TezUtilsInternal: Redirecting log file based on addend: dag_1428329756093_325099_1_post In the dag_1428329756093_325099_1_post log (attached to the jira as well) {code} 2015-04-12 23:24:57,029 INFO [AMShutdownThread] ats.ATSHistoryLoggingService: Stopping ATSService, eventQueueBacklog=17927 2015-04-12 23:25:25,466 WARN [AMShutdownThread] ats.ATSHistoryLoggingService: ATSService being stopped, eventQueueBacklog=17927, maxTimeLeftToFlush=-1, waitForever=true Lot of ATS put errors 2015-04-12 23:32:53,197 INFO [AMShutdownThread] ats.ATSHistoryLoggingService: Event queue empty, stopping ATS Service 2015-04-12 23:32:53,200 INFO [DelayedContainerManager] rm.YarnTaskSchedulerService: AllocatedContainerManager Thread interrupted 2015-04-12 23:32:53,203 INFO [AMShutdownThread] rm.YarnTaskSchedulerService: Unregistering application from RM, exitStatus=SUCCEEDED, exitMessage=Session stats:submittedDAGs=1, successfulDAGs=0, failedDAGs=0, killedDAGs=1 , trackingURL=bassniumtan-jt1.tan.ygrid.yahoo.com:4080/tez/#/?appid=application_1428329756093_325099 2015-04-12 23:32:53,210 INFO [AMShutdownThread] impl.AMRMClientImpl: Waiting for application to be successfully unregistered. 2015-04-12 23:32:53,314 INFO [AMShutdownThread] rm.YarnTaskSchedulerService: Successfully unregistered application from RM 2015-04-12 23:32:53,315 INFO [AMRM Callback Handler Thread] impl.AMRMClientAsyncImpl: Interrupted while waiting for queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274) 2015-04-12 23:32:53,316 INFO [AMShutdownThread] ipc.Server: Stopping server on 51921 2015-04-12 23:32:53,319 INFO [IPC Server listener on 51921] ipc.Server: Stopping IPC Server listener on 51921 2015-04-12 23:32:53,319 INFO [AMShutdownThread] ipc.Server: Stopping server on 50500 2015-04-12 23:32:53,320 INFO [IPC Server listener on 50500] ipc.Server: Stopping IPC Server listener on 50500 2015-04-12 23:32:53,320 INFO [IPC Server Responder] ipc.Server: Stopping IPC Server Responder 2015-04-12 23:32:53,320 INFO [IPC Server Responder] ipc.Server: Stopping IPC Server Responder 2015-04-12 23:32:53,324 INFO [AMShutdownThread] app.DAGAppMaster: Completed deletion of tez scratch data dir, path=hdfs://bassniumtan-nn1.tan.ygrid.yahoo.com:8020/tmp/temp-1464028011/.tez/application_1428329756093_325099 2015-04-12 23:32:53,324 INFO [AMShutdownThread] app.DAGAppMaster: Exiting DAGAppMaster..GoodBye! 2015-04-12 23:32:53,325 INFO [Thread-1] app.DAGAppMaster: DAGAppMasterShutdownHook invoked {code} Jstack still running thread on AM : {code} "AMShutdownThread" prio=10 tid=0x0f3ad800 nid=0x2b0e runnable [0x04e4] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) - locked <0xde56b700> (a java.io.BufferedInputStream) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:633) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:579) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1322) - locked <0xde42d320> (a sun.net.www.protocol.http.HttpURLConnection) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) at com.sun.jersey.client.ur
[jira] [Updated] (TEZ-2300) TezClient.stop() takes a lot of time or does not work sometimes
[ https://issues.apache.org/jira/browse/TEZ-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohini Palaniswamy updated TEZ-2300: Attachment: syslog_dag_1428329756093_325099_1_post > TezClient.stop() takes a lot of time or does not work sometimes > --- > > Key: TEZ-2300 > URL: https://issues.apache.org/jira/browse/TEZ-2300 > Project: Apache Tez > Issue Type: Bug >Reporter: Rohini Palaniswamy > Attachments: syslog_dag_1428329756093_325099_1_post > > > Noticed this with a couple of pig scripts which were not behaving well (AM > close to OOM, etc) and even with some that were running fine. Pig calls > Tezclient.stop() in shutdown hook. Ctrl+C to the pig script either exits > immediately or is hung. In both cases it either takes a long time for the > yarn application to go to KILLED state. Many times I just end up calling yarn > application -kill separately after waiting for 5 mins or more for it to get > killed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-2300) TezClient.stop() takes a lot of time or does not work sometimes
[ https://issues.apache.org/jira/browse/TEZ-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491781#comment-14491781 ] Rohini Palaniswamy edited comment on TEZ-2300 at 4/12/15 11:45 PM: --- This one was a case of AM lingering long due to posting to ATS. [~jlowe], had configured the tez.yarn.ats.max.events.per.batch to 500 in tez-site.xml as we had earlier noticed that even on normal shutdown AM was lingering for a lot of time when there are lot of events. But in this case it is still worse taking more than 5 mins. I believe it is due to ATS put errors and retries. 2015-04-12 23:24:42,040 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 51890 Succeeded: 0 Running: 50 Failed: 0 Killed: 0, diagnostics=, counters=null Did Ctlr+C here on the pig client. AM DAG log: 2015-04-12 23:24:52,015 INFO [AsyncDispatcher event handler] app.DAGAppMaster: DAG completed, dagId=dag_1428329756093_325099_1, dagState=KILLED 2015-04-12 23:24:52,015 INFO [AsyncDispatcher event handler] common.TezUtilsInternal: Redirecting log file based on addend: dag_1428329756093_325099_1_post In the dag_1428329756093_325099_1_post log (attached to the jira as well) {code} 2015-04-12 23:24:57,029 INFO [AMShutdownThread] ats.ATSHistoryLoggingService: Stopping ATSService, eventQueueBacklog=17927 2015-04-12 23:25:25,466 WARN [AMShutdownThread] ats.ATSHistoryLoggingService: ATSService being stopped, eventQueueBacklog=17927, maxTimeLeftToFlush=-1, waitForever=true Lot of ATS put errors 2015-04-12 23:32:53,197 INFO [AMShutdownThread] ats.ATSHistoryLoggingService: Event queue empty, stopping ATS Service 2015-04-12 23:32:53,200 INFO [DelayedContainerManager] rm.YarnTaskSchedulerService: AllocatedContainerManager Thread interrupted 2015-04-12 23:32:53,203 INFO [AMShutdownThread] rm.YarnTaskSchedulerService: Unregistering application from RM, exitStatus=SUCCEEDED, exitMessage=Session stats:submittedDAGs=1, successfulDAGs=0, failedDAGs=0, killedDAGs=1 , trackingURL=bassniumtan-jt1.tan.ygrid.yahoo.com:4080/tez/#/?appid=application_1428329756093_325099 2015-04-12 23:32:53,210 INFO [AMShutdownThread] impl.AMRMClientImpl: Waiting for application to be successfully unregistered. 2015-04-12 23:32:53,314 INFO [AMShutdownThread] rm.YarnTaskSchedulerService: Successfully unregistered application from RM 2015-04-12 23:32:53,315 INFO [AMRM Callback Handler Thread] impl.AMRMClientAsyncImpl: Interrupted while waiting for queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274) 2015-04-12 23:32:53,316 INFO [AMShutdownThread] ipc.Server: Stopping server on 51921 2015-04-12 23:32:53,319 INFO [IPC Server listener on 51921] ipc.Server: Stopping IPC Server listener on 51921 2015-04-12 23:32:53,319 INFO [AMShutdownThread] ipc.Server: Stopping server on 50500 2015-04-12 23:32:53,320 INFO [IPC Server listener on 50500] ipc.Server: Stopping IPC Server listener on 50500 2015-04-12 23:32:53,320 INFO [IPC Server Responder] ipc.Server: Stopping IPC Server Responder 2015-04-12 23:32:53,320 INFO [IPC Server Responder] ipc.Server: Stopping IPC Server Responder 2015-04-12 23:32:53,324 INFO [AMShutdownThread] app.DAGAppMaster: Completed deletion of tez scratch data dir, path=hdfs://bassniumtan-nn1.tan.ygrid.yahoo.com:8020/tmp/temp-1464028011/.tez/application_1428329756093_325099 2015-04-12 23:32:53,324 INFO [AMShutdownThread] app.DAGAppMaster: Exiting DAGAppMaster..GoodBye! 2015-04-12 23:32:53,325 INFO [Thread-1] app.DAGAppMaster: DAGAppMasterShutdownHook invoked {code} Jstack still running thread on AM : {code} "AMShutdownThread" prio=10 tid=0x0f3ad800 nid=0x2b0e runnable [0x04e4] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) - locked <0xde56b700> (a java.io.BufferedInputStream) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:633) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:579) at sun.net.www.protocol.http.Ht
[jira] [Comment Edited] (TEZ-2300) TezClient.stop() takes a lot of time or does not work sometimes
[ https://issues.apache.org/jira/browse/TEZ-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491781#comment-14491781 ] Rohini Palaniswamy edited comment on TEZ-2300 at 4/12/15 11:46 PM: --- This one was a case of AM lingering long due to posting to ATS. [~jlowe], had configured the tez.yarn.ats.max.events.per.batch to 500 in tez-site.xml as we had earlier noticed that even on normal shutdown AM was lingering for a lot of time when there are lot of events. But in this case it is still worse taking more than 5 mins. I believe it is due to ATS put errors and retries. 2015-04-12 23:24:42,040 [Timer-0] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezJob - DAG Status: status=RUNNING, progress=TotalTasks: 51890 Succeeded: 0 Running: 50 Failed: 0 Killed: 0, diagnostics=, counters=null Did Ctlr+C here on the pig client. AM DAG log: 2015-04-12 23:24:52,015 INFO [AsyncDispatcher event handler] app.DAGAppMaster: DAG completed, dagId=dag_1428329756093_325099_1, dagState=KILLED 2015-04-12 23:24:52,015 INFO [AsyncDispatcher event handler] common.TezUtilsInternal: Redirecting log file based on addend: dag_1428329756093_325099_1_post In the dag_1428329756093_325099_1_post log (attached to the jira as well) {code} 2015-04-12 23:24:57,029 INFO [AMShutdownThread] ats.ATSHistoryLoggingService: Stopping ATSService, eventQueueBacklog=17927 2015-04-12 23:25:25,466 WARN [AMShutdownThread] ats.ATSHistoryLoggingService: ATSService being stopped, eventQueueBacklog=17927, maxTimeLeftToFlush=-1, waitForever=true Lot of ATS put errors 2015-04-12 23:32:53,197 INFO [AMShutdownThread] ats.ATSHistoryLoggingService: Event queue empty, stopping ATS Service 2015-04-12 23:32:53,200 INFO [DelayedContainerManager] rm.YarnTaskSchedulerService: AllocatedContainerManager Thread interrupted 2015-04-12 23:32:53,203 INFO [AMShutdownThread] rm.YarnTaskSchedulerService: Unregistering application from RM, exitStatus=SUCCEEDED, exitMessage=Session stats:submittedDAGs=1, successfulDAGs=0, failedDAGs=0, killedDAGs=1 , trackingURL=bassniumtan-jt1.tan.ygrid.yahoo.com:4080/tez/#/?appid=application_1428329756093_325099 2015-04-12 23:32:53,210 INFO [AMShutdownThread] impl.AMRMClientImpl: Waiting for application to be successfully unregistered. 2015-04-12 23:32:53,314 INFO [AMShutdownThread] rm.YarnTaskSchedulerService: Successfully unregistered application from RM 2015-04-12 23:32:53,315 INFO [AMRM Callback Handler Thread] impl.AMRMClientAsyncImpl: Interrupted while waiting for queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274) 2015-04-12 23:32:53,316 INFO [AMShutdownThread] ipc.Server: Stopping server on 51921 2015-04-12 23:32:53,319 INFO [IPC Server listener on 51921] ipc.Server: Stopping IPC Server listener on 51921 2015-04-12 23:32:53,319 INFO [AMShutdownThread] ipc.Server: Stopping server on 50500 2015-04-12 23:32:53,320 INFO [IPC Server listener on 50500] ipc.Server: Stopping IPC Server listener on 50500 2015-04-12 23:32:53,320 INFO [IPC Server Responder] ipc.Server: Stopping IPC Server Responder 2015-04-12 23:32:53,320 INFO [IPC Server Responder] ipc.Server: Stopping IPC Server Responder 2015-04-12 23:32:53,324 INFO [AMShutdownThread] app.DAGAppMaster: Completed deletion of tez scratch data dir, path=hdfs://bassniumtan-nn1.tan.ygrid.yahoo.com:8020/tmp/temp-1464028011/.tez/application_1428329756093_325099 2015-04-12 23:32:53,324 INFO [AMShutdownThread] app.DAGAppMaster: Exiting DAGAppMaster..GoodBye! 2015-04-12 23:32:53,325 INFO [Thread-1] app.DAGAppMaster: DAGAppMasterShutdownHook invoked {code} Jstack still running thread on AM : {code} "AMShutdownThread" prio=10 tid=0x0f3ad800 nid=0x2b0e runnable [0x04e4] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) - locked <0xde56b700> (a java.io.BufferedInputStream) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:633) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:579) at sun.net.www.protocol.http.Ht
[jira] [Updated] (TEZ-714) OutputCommitters should not run in the main AM dispatcher thread
[ https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-714: --- Attachment: TEZ-714-13.patch Upload new patch, minor update on e2e test. > OutputCommitters should not run in the main AM dispatcher thread > > > Key: TEZ-714 > URL: https://issues.apache.org/jira/browse/TEZ-714 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Jeff Zhang >Priority: Critical > Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-10.patch, > TEZ-714-11.patch, TEZ-714-12.patch, TEZ-714-13.patch, TEZ-714-2.patch, > TEZ-714-3.patch, TEZ-714-4.patch, TEZ-714-5.patch, TEZ-714-6.patch, > TEZ-714-7.patch, TEZ-714-8.patch, TEZ-714-9.patch, Vertex_2.pdf > > > Follow up jira from TEZ-41. > 1) If there's multiple OutputCommitters on a Vertex, they can be run in > parallel. > 2) Running an OutputCommitter in the main thread blocks all other event > handling, w.r.t the DAG, and causes the event queue to back up. > 3) This should also cover shared commits that happen in the DAG. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-714 PreCommit Build #446
Jira: https://issues.apache.org/jira/browse/TEZ-714 Build: https://builds.apache.org/job/PreCommit-TEZ-Build/446/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 19 lines...] No emails were triggered. [PreCommit-TEZ-Build] $ /bin/bash /tmp/hudson8532898926665538831.sh Running in Jenkins mode == == Testing patch for TEZ-714. == == HEAD is now at bd9b8d9 TEZ-2234. Add API for statistics information - allow vertex managers to get output size per source vertex (bikas) Switched to branch 'master' Your branch is up-to-date with 'origin/master'. Current branch master is up to date. TEZ-714 patch is being downloaded at Mon Apr 13 01:30:18 UTC 2015 from http://issues.apache.org/jira/secure/attachment/12724852/TEZ-714-13.patch The patch does not appear to apply with p0 to p2 PATCH APPLICATION FAILED {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12724852/TEZ-714-13.patch against master revision bd9b8d9. {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/446//console This message is automatically generated. == == Adding comment to Jira. == == Comment added. 46fc6f7f9ecf45cdfab641ffb84f01ae5961fef4 logged out == == Finished build. == == Build step 'Execute shell' marked build as failure Archiving artifacts [description-setter] Could not determine description. Recording test results Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
[jira] [Commented] (TEZ-714) OutputCommitters should not run in the main AM dispatcher thread
[ https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491826#comment-14491826 ] TezQA commented on TEZ-714: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12724852/TEZ-714-13.patch against master revision bd9b8d9. {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/446//console This message is automatically generated. > OutputCommitters should not run in the main AM dispatcher thread > > > Key: TEZ-714 > URL: https://issues.apache.org/jira/browse/TEZ-714 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Jeff Zhang >Priority: Critical > Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-10.patch, > TEZ-714-11.patch, TEZ-714-12.patch, TEZ-714-13.patch, TEZ-714-2.patch, > TEZ-714-3.patch, TEZ-714-4.patch, TEZ-714-5.patch, TEZ-714-6.patch, > TEZ-714-7.patch, TEZ-714-8.patch, TEZ-714-9.patch, Vertex_2.pdf > > > Follow up jira from TEZ-41. > 1) If there's multiple OutputCommitters on a Vertex, they can be run in > parallel. > 2) Running an OutputCommitter in the main thread blocks all other event > handling, w.r.t the DAG, and causes the event queue to back up. > 3) This should also cover shared commits that happen in the DAG. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2305) MR compatibility sleep job fails with IOException: Undefined job output-path
[ https://issues.apache.org/jira/browse/TEZ-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491832#comment-14491832 ] Jeff Zhang commented on TEZ-2305: - [~hitesh] Thanks for your clarification. +1 on your last patch, and also verify that the patch resolve TEZ-2271 . > MR compatibility sleep job fails with IOException: Undefined job output-path > > > Key: TEZ-2305 > URL: https://issues.apache.org/jira/browse/TEZ-2305 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Tassapol Athiapinya >Priority: Critical > Attachments: TEZ-2305-3.patch, TEZ-2305-4.patch, TEZ-2305.1.patch, > TEZ-2305.2.patch > > > Running MR sleep job has an IOException. > {code} > 15/04/09 20:52:25 INFO mapreduce.Job: Job job_1428612196442_0002 failed with > state FAILED due to: Vertex failed, vertexName=initialmap, > vertexId=vertex_1428612196442_0002_1_00, diagnostics=[Task failed, > taskId=task_1428612196442_0002_1_00_01, diagnostics=[TaskAttempt 0 > failed, info=[Error: Failure while running task:java.io.IOException: > Undefined job output-path > at > org.apache.hadoop.mapred.FileOutputFormat.getTaskOutputPath(FileOutputFormat.java:248) > at > org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:121) > at > org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:401) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:436) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:415) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > ], TaskAttempt 1 failed, info=[Error: Failure while running > task:java.io.IOException: Undefined job output-path > at > org.apache.hadoop.mapred.FileOutputFormat.getTaskOutputPath(FileOutputFormat.java:248) > at > org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:121) > at > org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:401) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:436) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:415) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > ], TaskAttempt 2 failed, info=[Error: Failure while running > task:java.io.IOException: Undefined job output-path > at > org.apache.hadoop.mapred.FileOutputFormat.getTaskOutputPath(FileOutputFormat.java:248) > at > org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:121) > at > org.apache.tez.mapreduce.output.MROutput.initialize(MROutput.java:401) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:436) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:415) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > ], TaskAttempt 3 failed, info=[Error: Failure while running > task:java.io.IOException: Undefined
[jira] [Updated] (TEZ-714) OutputCommitters should not run in the main AM dispatcher thread
[ https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-714: --- Attachment: TEZ-714-14.patch Rebase the patch > OutputCommitters should not run in the main AM dispatcher thread > > > Key: TEZ-714 > URL: https://issues.apache.org/jira/browse/TEZ-714 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Jeff Zhang >Priority: Critical > Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-10.patch, > TEZ-714-11.patch, TEZ-714-12.patch, TEZ-714-13.patch, TEZ-714-14.patch, > TEZ-714-2.patch, TEZ-714-3.patch, TEZ-714-4.patch, TEZ-714-5.patch, > TEZ-714-6.patch, TEZ-714-7.patch, TEZ-714-8.patch, TEZ-714-9.patch, > Vertex_2.pdf > > > Follow up jira from TEZ-41. > 1) If there's multiple OutputCommitters on a Vertex, they can be run in > parallel. > 2) Running an OutputCommitter in the main thread blocks all other event > handling, w.r.t the DAG, and causes the event queue to back up. > 3) This should also cover shared commits that happen in the DAG. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-714 PreCommit Build #447
Jira: https://issues.apache.org/jira/browse/TEZ-714 Build: https://builds.apache.org/job/PreCommit-TEZ-Build/447/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 19 lines...] No emails were triggered. [PreCommit-TEZ-Build] $ /bin/bash /tmp/hudson8093622342086229950.sh Running in Jenkins mode == == Testing patch for TEZ-714. == == HEAD is now at bd9b8d9 TEZ-2234. Add API for statistics information - allow vertex managers to get output size per source vertex (bikas) Switched to branch 'master' Your branch is up-to-date with 'origin/master'. Current branch master is up to date. TEZ-714 patch is being downloaded at Mon Apr 13 02:10:19 UTC 2015 from http://issues.apache.org/jira/secure/attachment/12724853/TEZ-714-14.patch The patch does not appear to apply with p0 to p2 PATCH APPLICATION FAILED {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12724853/TEZ-714-14.patch against master revision bd9b8d9. {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/447//console This message is automatically generated. == == Adding comment to Jira. == == Comment added. 90627676be18301787524281b4c76922bbc7d8f4 logged out == == Finished build. == == Build step 'Execute shell' marked build as failure Archiving artifacts [description-setter] Could not determine description. Recording test results Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
[jira] [Commented] (TEZ-714) OutputCommitters should not run in the main AM dispatcher thread
[ https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491848#comment-14491848 ] TezQA commented on TEZ-714: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12724853/TEZ-714-14.patch against master revision bd9b8d9. {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/447//console This message is automatically generated. > OutputCommitters should not run in the main AM dispatcher thread > > > Key: TEZ-714 > URL: https://issues.apache.org/jira/browse/TEZ-714 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Jeff Zhang >Priority: Critical > Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-10.patch, > TEZ-714-11.patch, TEZ-714-12.patch, TEZ-714-13.patch, TEZ-714-14.patch, > TEZ-714-2.patch, TEZ-714-3.patch, TEZ-714-4.patch, TEZ-714-5.patch, > TEZ-714-6.patch, TEZ-714-7.patch, TEZ-714-8.patch, TEZ-714-9.patch, > Vertex_2.pdf > > > Follow up jira from TEZ-41. > 1) If there's multiple OutputCommitters on a Vertex, they can be run in > parallel. > 2) Running an OutputCommitter in the main thread blocks all other event > handling, w.r.t the DAG, and causes the event queue to back up. > 3) This should also cover shared commits that happen in the DAG. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-714) OutputCommitters should not run in the main AM dispatcher thread
[ https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-714: --- Attachment: TEZ-714-15.patch Rebase it again. > OutputCommitters should not run in the main AM dispatcher thread > > > Key: TEZ-714 > URL: https://issues.apache.org/jira/browse/TEZ-714 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Jeff Zhang >Priority: Critical > Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-10.patch, > TEZ-714-11.patch, TEZ-714-12.patch, TEZ-714-13.patch, TEZ-714-14.patch, > TEZ-714-15.patch, TEZ-714-2.patch, TEZ-714-3.patch, TEZ-714-4.patch, > TEZ-714-5.patch, TEZ-714-6.patch, TEZ-714-7.patch, TEZ-714-8.patch, > TEZ-714-9.patch, Vertex_2.pdf > > > Follow up jira from TEZ-41. > 1) If there's multiple OutputCommitters on a Vertex, they can be run in > parallel. > 2) Running an OutputCommitter in the main thread blocks all other event > handling, w.r.t the DAG, and causes the event queue to back up. > 3) This should also cover shared commits that happen in the DAG. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2225) Remove instances of LOG.isDebugEnabled
[ https://issues.apache.org/jira/browse/TEZ-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491860#comment-14491860 ] Bikas Saha commented on TEZ-2225: - If done correctly, sl4j takes care of the perf aspects - please see http://www.slf4j.org/faq.html#logging_performance However not all cases are covered by the sl4j technique - e.g. when some method is being called to generate the argument to the log statement. [~vasanthkumar] Could you please take a look at http://stackoverflow.com/questions/8444266/even-with-slf4j-should-you-guard-your-logging and verify that we are making the correct changes when the logger argument is invoking a method instead of accessing a local variable? Committing the patch would be easy once you have confirmed that. Thanks. Trying to trigger another build to verify findbugs. > Remove instances of LOG.isDebugEnabled > -- > > Key: TEZ-2225 > URL: https://issues.apache.org/jira/browse/TEZ-2225 > Project: Apache Tez > Issue Type: Improvement >Reporter: Vasanth kumar RJ >Assignee: Vasanth kumar RJ >Priority: Minor > Labels: performance > Attachments: TEZ-2225.1.patch, TEZ-2225.2.patch, TEZ-2225.3.patch > > > Remove LOG.isDebugEnabled() and use parameterized debug logging -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-714) OutputCommitters should not run in the main AM dispatcher thread
[ https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491862#comment-14491862 ] Bikas Saha commented on TEZ-714: Using wrong parameter? {code}+// add data sinks for v2 +for (int i=0;i OutputCommitters should not run in the main AM dispatcher thread > > > Key: TEZ-714 > URL: https://issues.apache.org/jira/browse/TEZ-714 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Jeff Zhang >Priority: Critical > Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-10.patch, > TEZ-714-11.patch, TEZ-714-12.patch, TEZ-714-13.patch, TEZ-714-14.patch, > TEZ-714-15.patch, TEZ-714-2.patch, TEZ-714-3.patch, TEZ-714-4.patch, > TEZ-714-5.patch, TEZ-714-6.patch, TEZ-714-7.patch, TEZ-714-8.patch, > TEZ-714-9.patch, Vertex_2.pdf > > > Follow up jira from TEZ-41. > 1) If there's multiple OutputCommitters on a Vertex, they can be run in > parallel. > 2) Running an OutputCommitter in the main thread blocks all other event > handling, w.r.t the DAG, and causes the event queue to back up. > 3) This should also cover shared commits that happen in the DAG. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2304) InvalidStateTransitonException TA_SCHEDULE at START_WAIT during recovery
[ https://issues.apache.org/jira/browse/TEZ-2304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491867#comment-14491867 ] Jeff Zhang commented on TEZ-2304: - [~jlowe] Could you attach the full app logs ? Initial doubt is that the attemptId may be set incorrectly. {code} 2015-04-09 20:05:42,055 INFO [AsyncDispatcher event handler] impl.TaskAttemptImpl: attempt_1428329756093_168563_1_00_006728_1 TaskAttempt Transitioned from NEW to KILLED due to event TA_RECOVER 2015-04-09 20:05:45,748 INFO [AsyncDispatcher event handler] impl.TaskAttemptImpl: attempt_1428329756093_168563_1_00_006728_1 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE {code} > InvalidStateTransitonException TA_SCHEDULE at START_WAIT during recovery > > > Key: TEZ-2304 > URL: https://issues.apache.org/jira/browse/TEZ-2304 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.0 >Reporter: Jason Lowe > > I saw a Tez AM throw a few InvalidStateTransitonException (sic) instances > during recovery complaining about TA_SCHEDULE arriving at the START_WAIT > state. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (TEZ-2304) InvalidStateTransitonException TA_SCHEDULE at START_WAIT during recovery
[ https://issues.apache.org/jira/browse/TEZ-2304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491867#comment-14491867 ] Jeff Zhang edited comment on TEZ-2304 at 4/13/15 3:11 AM: -- [~jlowe] Could you attach the full app logs ? was (Author: zjffdu): [~jlowe] Could you attach the full app logs ? Initial doubt is that the attemptId may be set incorrectly. {code} 2015-04-09 20:05:42,055 INFO [AsyncDispatcher event handler] impl.TaskAttemptImpl: attempt_1428329756093_168563_1_00_006728_1 TaskAttempt Transitioned from NEW to KILLED due to event TA_RECOVER 2015-04-09 20:05:45,748 INFO [AsyncDispatcher event handler] impl.TaskAttemptImpl: attempt_1428329756093_168563_1_00_006728_1 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE {code} > InvalidStateTransitonException TA_SCHEDULE at START_WAIT during recovery > > > Key: TEZ-2304 > URL: https://issues.apache.org/jira/browse/TEZ-2304 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.6.0 >Reporter: Jason Lowe > > I saw a Tez AM throw a few InvalidStateTransitonException (sic) instances > during recovery complaining about TA_SCHEDULE arriving at the START_WAIT > state. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-714) OutputCommitters should not run in the main AM dispatcher thread
[ https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-714: --- Attachment: TEZ-714-16.patch [~bikassaha] Thanks for pointing out this, correct it in the new patch. > OutputCommitters should not run in the main AM dispatcher thread > > > Key: TEZ-714 > URL: https://issues.apache.org/jira/browse/TEZ-714 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Jeff Zhang >Priority: Critical > Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-10.patch, > TEZ-714-11.patch, TEZ-714-12.patch, TEZ-714-13.patch, TEZ-714-14.patch, > TEZ-714-15.patch, TEZ-714-16.patch, TEZ-714-2.patch, TEZ-714-3.patch, > TEZ-714-4.patch, TEZ-714-5.patch, TEZ-714-6.patch, TEZ-714-7.patch, > TEZ-714-8.patch, TEZ-714-9.patch, Vertex_2.pdf > > > Follow up jira from TEZ-41. > 1) If there's multiple OutputCommitters on a Vertex, they can be run in > parallel. > 2) Running an OutputCommitter in the main thread blocks all other event > handling, w.r.t the DAG, and causes the event queue to back up. > 3) This should also cover shared commits that happen in the DAG. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2252) Tez UI Graphical view looks wrong in some cases
[ https://issues.apache.org/jira/browse/TEZ-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491874#comment-14491874 ] TezQA commented on TEZ-2252: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12724844/TEZ-2252.1.patch against master revision bd9b8d9. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in : org.apache.tez.client.TestTezClientUtils org.apache.tez.common.TestReflectionUtils Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/449//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/449//console This message is automatically generated. > Tez UI Graphical view looks wrong in some cases > --- > > Key: TEZ-2252 > URL: https://issues.apache.org/jira/browse/TEZ-2252 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Rohini Palaniswamy >Assignee: Sreenath Somarajapuram > Attachments: Graph1.dot, Graph1.png, Graph2.dot, Graph2.png, > TEZ-2252.1.patch > > > The information in the .dot file is correct and script runs fine. But the > Tez UI Graphical view shows that output is being written from multiple > vertices into one sink. Actually it is writing to multiple sinks (checking > the html elements in Firebug), but the sink circles all overlap one another > with exact coordinates and the tool tip only shows for the top one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-714 PreCommit Build #448
Jira: https://issues.apache.org/jira/browse/TEZ-714 Build: https://builds.apache.org/job/PreCommit-TEZ-Build/448/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 2629 lines...] {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12724854/TEZ-714-15.patch against master revision bd9b8d9. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 5 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in : org.apache.tez.test.TestTezJobs org.apache.tez.mapreduce.TestMRRJobs org.apache.tez.mapreduce.TestMRRJobsDAGApi Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/448//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/448//console This message is automatically generated. == == Adding comment to Jira. == == Comment added. 2b1c9b4e28df14faa71410acb68f266ab8927656 logged out == == Finished build. == == Build step 'Execute shell' marked build as failure Archiving artifacts Sending artifact delta relative to PreCommit-TEZ-Build #441 Archived 44 artifacts Archive block size is 32768 Received 4 blocks and 2612619 bytes Compression is 4.8% Took 0.94 sec [description-setter] Could not determine description. Recording test results Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## 8 tests failed. REGRESSION: org.apache.tez.mapreduce.TestMRRJobs.testFailingAttempt Error Message: test timed out after 6 milliseconds Stack Trace: java.lang.Exception: test timed out after 6 milliseconds at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1359) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1317) at org.apache.tez.mapreduce.TestMRRJobs.testFailingAttempt(TestMRRJobs.java:251) REGRESSION: org.apache.tez.mapreduce.TestMRRJobsDAGApi.testMultipleMRRSleepJobViaSession Error Message: com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.tez.dag.api.TezException): Duplicate dag name 'testMRRSleepJobDagSubmit-116' at org.apache.tez.dag.app.DAGAppMaster.startDAG(DAGAppMaster.java:2025) at org.apache.tez.dag.app.DAGAppMaster.submitDAGToAppMaster(DAGAppMaster.java:1147) at org.apache.tez.dag.api.client.DAGClientHandler.submitDAG(DAGClientHandler.java:118) at org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.submitDAG(DAGClientAMProtocolBlockingPBServerImpl.java:163) at org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7471) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033) Stack Trace: org.apache.tez.dag.api.TezException: com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.tez.dag.api.TezException): Duplicate dag name 'testMRRSleepJobDagSubmit-116' at org.apache.tez.dag.app.DAGAppMaster.startDAG(DAGAppMaster
[jira] [Commented] (TEZ-714) OutputCommitters should not run in the main AM dispatcher thread
[ https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491876#comment-14491876 ] TezQA commented on TEZ-714: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12724854/TEZ-714-15.patch against master revision bd9b8d9. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 5 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in : org.apache.tez.test.TestTezJobs org.apache.tez.mapreduce.TestMRRJobs org.apache.tez.mapreduce.TestMRRJobsDAGApi Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/448//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/448//console This message is automatically generated. > OutputCommitters should not run in the main AM dispatcher thread > > > Key: TEZ-714 > URL: https://issues.apache.org/jira/browse/TEZ-714 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Jeff Zhang >Priority: Critical > Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-10.patch, > TEZ-714-11.patch, TEZ-714-12.patch, TEZ-714-13.patch, TEZ-714-14.patch, > TEZ-714-15.patch, TEZ-714-16.patch, TEZ-714-2.patch, TEZ-714-3.patch, > TEZ-714-4.patch, TEZ-714-5.patch, TEZ-714-6.patch, TEZ-714-7.patch, > TEZ-714-8.patch, TEZ-714-9.patch, Vertex_2.pdf > > > Follow up jira from TEZ-41. > 1) If there's multiple OutputCommitters on a Vertex, they can be run in > parallel. > 2) Running an OutputCommitter in the main thread blocks all other event > handling, w.r.t the DAG, and causes the event queue to back up. > 3) This should also cover shared commits that happen in the DAG. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-2252 PreCommit Build #449
Jira: https://issues.apache.org/jira/browse/TEZ-2252 Build: https://builds.apache.org/job/PreCommit-TEZ-Build/449/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 1992 lines...] {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12724844/TEZ-2252.1.patch against master revision bd9b8d9. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in : org.apache.tez.client.TestTezClientUtils org.apache.tez.common.TestReflectionUtils Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/449//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/449//console This message is automatically generated. == == Adding comment to Jira. == == Comment added. 8a3976367c6f304b356887c8adb7a07a24cc865e logged out == == Finished build. == == Build step 'Execute shell' marked build as failure Archiving artifacts Sending artifact delta relative to PreCommit-TEZ-Build #441 Archived 44 artifacts Archive block size is 32768 Received 8 blocks and 2432019 bytes Compression is 9.7% Took 1.3 sec [description-setter] Could not determine description. Recording test results Publish JUnit test result report is waiting for a checkpoint on PreCommit-TEZ-Build #448 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## 9 tests failed. REGRESSION: org.apache.tez.client.TestTezClientUtils.validateSetTezJarLocalResourcesDefinedExistingDirectory Error Message: test timed out after 5000 milliseconds Stack Trace: java.lang.Exception: test timed out after 5000 milliseconds at java.net.PlainDatagramSocketImpl.receive0(Native Method) at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:145) at java.net.DatagramSocket.receive(DatagramSocket.java:786) at com.sun.jndi.dns.DnsClient.doUdpQuery(DnsClient.java:416) at com.sun.jndi.dns.DnsClient.query(DnsClient.java:210) at com.sun.jndi.dns.Resolver.query(Resolver.java:81) at com.sun.jndi.dns.DnsContext.c_getAttributes(DnsContext.java:430) at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_getAttributes(ComponentDirContext.java:231) at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.getAttributes(PartialCompositeDirContext.java:139) at com.sun.jndi.toolkit.url.GenericURLDirContext.getAttributes(GenericURLDirContext.java:103) at sun.security.krb5.KrbServiceLocator.getKerberosService(KrbServiceLocator.java:87) at sun.security.krb5.Config.checkRealm(Config.java:1295) at sun.security.krb5.Config.getRealmFromDNS(Config.java:1268) at sun.security.krb5.Config.getDefaultRealm(Config.java:1162) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:84) at org.apache.hadoop.security.authentication.util.KerberosName.(KerberosName.java:86) at org.apach
Failed: TEZ-714 PreCommit Build #450
Jira: https://issues.apache.org/jira/browse/TEZ-714 Build: https://builds.apache.org/job/PreCommit-TEZ-Build/450/ ### ## LAST 60 LINES OF THE CONSOLE ### Started by remote host 127.0.0.1 Building remotely on H0 (Hadoop Tez) in workspace /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url https://git-wip-us.apache.org/repos/asf/tez.git > # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/tez.git > git --version # timeout=10 FATAL: Failed to fetch from https://git-wip-us.apache.org/repos/asf/tez.git hudson.plugins.git.GitException: Failed to fetch from https://git-wip-us.apache.org/repos/asf/tez.git at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:647) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:889) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:914) at hudson.model.AbstractProject.checkout(AbstractProject.java:1252) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:615) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:524) at hudson.model.Run.execute(Run.java:1706) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:232) Caused by: hudson.plugins.git.GitException: Failed to connect to https://git-wip-us.apache.org/repos/asf/tez.git (exception: java.net.UnknownHostException: git-wip-us.apache.org: Name or service not known) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.checkCredentials(CliGitAPIImpl.java:2253) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1169) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:85) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:280) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:328) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ### ## FAILED TESTS (if any) ## No tests ran.
[jira] [Updated] (TEZ-714) OutputCommitters should not run in the main AM dispatcher thread
[ https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-714: --- Attachment: (was: TEZ-714-16.patch) > OutputCommitters should not run in the main AM dispatcher thread > > > Key: TEZ-714 > URL: https://issues.apache.org/jira/browse/TEZ-714 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Jeff Zhang >Priority: Critical > Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-10.patch, > TEZ-714-11.patch, TEZ-714-12.patch, TEZ-714-13.patch, TEZ-714-14.patch, > TEZ-714-15.patch, TEZ-714-16.patch, TEZ-714-2.patch, TEZ-714-3.patch, > TEZ-714-4.patch, TEZ-714-5.patch, TEZ-714-6.patch, TEZ-714-7.patch, > TEZ-714-8.patch, TEZ-714-9.patch, Vertex_2.pdf > > > Follow up jira from TEZ-41. > 1) If there's multiple OutputCommitters on a Vertex, they can be run in > parallel. > 2) Running an OutputCommitter in the main thread blocks all other event > handling, w.r.t the DAG, and causes the event queue to back up. > 3) This should also cover shared commits that happen in the DAG. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-714) OutputCommitters should not run in the main AM dispatcher thread
[ https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-714: --- Attachment: TEZ-714-16.patch > OutputCommitters should not run in the main AM dispatcher thread > > > Key: TEZ-714 > URL: https://issues.apache.org/jira/browse/TEZ-714 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Jeff Zhang >Priority: Critical > Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-10.patch, > TEZ-714-11.patch, TEZ-714-12.patch, TEZ-714-13.patch, TEZ-714-14.patch, > TEZ-714-15.patch, TEZ-714-16.patch, TEZ-714-2.patch, TEZ-714-3.patch, > TEZ-714-4.patch, TEZ-714-5.patch, TEZ-714-6.patch, TEZ-714-7.patch, > TEZ-714-8.patch, TEZ-714-9.patch, Vertex_2.pdf > > > Follow up jira from TEZ-41. > 1) If there's multiple OutputCommitters on a Vertex, they can be run in > parallel. > 2) Running an OutputCommitter in the main thread blocks all other event > handling, w.r.t the DAG, and causes the event queue to back up. > 3) This should also cover shared commits that happen in the DAG. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-714) OutputCommitters should not run in the main AM dispatcher thread
[ https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491897#comment-14491897 ] TezQA commented on TEZ-714: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12724863/TEZ-714-16.patch against master revision bd9b8d9. {color:red}-1 patch{color}. master compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/451//console This message is automatically generated. > OutputCommitters should not run in the main AM dispatcher thread > > > Key: TEZ-714 > URL: https://issues.apache.org/jira/browse/TEZ-714 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Jeff Zhang >Priority: Critical > Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-10.patch, > TEZ-714-11.patch, TEZ-714-12.patch, TEZ-714-13.patch, TEZ-714-14.patch, > TEZ-714-15.patch, TEZ-714-16.patch, TEZ-714-2.patch, TEZ-714-3.patch, > TEZ-714-4.patch, TEZ-714-5.patch, TEZ-714-6.patch, TEZ-714-7.patch, > TEZ-714-8.patch, TEZ-714-9.patch, Vertex_2.pdf > > > Follow up jira from TEZ-41. > 1) If there's multiple OutputCommitters on a Vertex, they can be run in > parallel. > 2) Running an OutputCommitter in the main thread blocks all other event > handling, w.r.t the DAG, and causes the event queue to back up. > 3) This should also cover shared commits that happen in the DAG. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-714 PreCommit Build #451
Jira: https://issues.apache.org/jira/browse/TEZ-714 Build: https://builds.apache.org/job/PreCommit-TEZ-Build/451/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 35 lines...] TEZ-714 patch is being downloaded at Mon Apr 13 04:10:37 UTC 2015 from http://issues.apache.org/jira/secure/attachment/12724863/TEZ-714-16.patch == == Pre-build master to verify master stability and javac warnings == == Compiling /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build /home/jenkins/tools/maven/latest/bin/mvn clean test -DskipTests -Ptest-patch > /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/../patchprocess/masterJavacWarnings.txt 2>&1 master compilation is broken? {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12724863/TEZ-714-16.patch against master revision bd9b8d9. {color:red}-1 patch{color}. master compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/451//console This message is automatically generated. == == Adding comment to Jira. == == Comment added. 7caea134b03657b7b9109f9c8031818fb5dfecee logged out == == Finished build. == == Build step 'Execute shell' marked build as failure Archiving artifacts Sending artifact delta relative to PreCommit-TEZ-Build #441 Archived 1 artifacts Archive block size is 32768 Received 0 blocks and 134660 bytes Compression is 0.0% Took 65 ms [description-setter] Could not determine description. Recording test results Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
[jira] [Updated] (TEZ-2119) Counter for launched containers
[ https://issues.apache.org/jira/browse/TEZ-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-2119: Assignee: (was: Jeff Zhang) > Counter for launched containers > --- > > Key: TEZ-2119 > URL: https://issues.apache.org/jira/browse/TEZ-2119 > Project: Apache Tez > Issue Type: Improvement >Reporter: Rohini Palaniswamy > > org.apache.tez.common.counters.DAGCounter > NUM_SUCCEEDED_TASKS=32976 > TOTAL_LAUNCHED_TASKS=32976 > OTHER_LOCAL_TASKS=2 > DATA_LOCAL_TASKS=9147 > RACK_LOCAL_TASKS=23761 > It would be very nice to have TOTAL_LAUNCHED_CONTAINERS counter added to > this. The difference between TOTAL_LAUNCHED_CONTAINERS and > TOTAL_LAUNCHED_TASKS should make it easy to see how much container reuse is > happening. It is very hard to find out now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (TEZ-2016) Tez UI Dag View Fit and Finish
[ https://issues.apache.org/jira/browse/TEZ-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram reassigned TEZ-2016: --- Assignee: Sreenath Somarajapuram > Tez UI Dag View Fit and Finish > -- > > Key: TEZ-2016 > URL: https://issues.apache.org/jira/browse/TEZ-2016 > Project: Apache Tez > Issue Type: Improvement > Components: UI >Reporter: Jonathan Eagles >Assignee: Sreenath Somarajapuram > Attachments: Check in checkbox is not centered.png, Customize vetex > should not have first item selected by default.png, Edge Id name.png, > Gaussian Blur is not Radial.png, Green success icon not flush with edge.png, > Wording and functionality.png > > > TEZ-1973 has added a major contribution to the Tez UI. The core of the > functionality is there and is committed to community to reduce effort on > keeping patch up-to-date and to allow community exposure to the contributions. > A few fit and finish items have been identified and need to be addressed. > - Gaussian blur around circles is not a circle (see attachment) > - Check in checkbox is not centered (see attachment) > - Success icon is not flush (see attachment) > - First item in customize vertex should not be selected by default (see > attachment) > - Edge Id name is not correct (see attachment) > - Double click: was never able to see this working. If functionality is > there please update the wording (no attachment) > -Click and drag on graph should not route tez ui to another page (no > attachment)- > - Prefer to see dag view view height to extend to bottom of screen (no > attachment) > # Make source and sink bubble apt for longer text > # Bring dag back into view if it go out of view port > # Add ellipsis for clipped text in tooltip > # Use grab & grabbing cursors to indicate that user can pan the diagram > # Put Edge Source Class before Edge Destination Class > # BUG-31616 - #3 > # Make source/sink link tooltip better -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2016) Tez UI Dag View Fit and Finish
[ https://issues.apache.org/jira/browse/TEZ-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-2016: Description: TEZ-1973 has added a major contribution to the Tez UI. The core of the functionality is there and is committed to community to reduce effort on keeping patch up-to-date and to allow community exposure to the contributions. A few fit and finish items have been identified and need to be addressed. - Gaussian blur around circles is not a circle (see attachment) - Check in checkbox is not centered (see attachment) - Success icon is not flush (see attachment) - First item in customize vertex should not be selected by default (see attachment) - Edge Id name is not correct (see attachment) - Double click: was never able to see this working. If functionality is there please update the wording (no attachment) -Click and drag on graph should not route tez ui to another page (no attachment)- - Prefer to see dag view view height to extend to bottom of screen (no attachment) # Make source and sink bubble apt for longer text # Bring dag back into view if it go out of view port # Add ellipsis for clipped text in tooltip # Use grab & grabbing cursors to indicate that user can pan the diagram # Put Edge Source Class before Edge Destination Class # BUG-31616 - #3 # Make source/sink link tooltip better was: TEZ-1973 has added a major contribution to the Tez UI. The core of the functionality is there and is committed to community to reduce effort on keeping patch up-to-date and to allow community exposure to the contributions. A few fit and finish items have been identified and need to be addressed. - Gaussian blur around circles is not a circle (see attachment) - Check in checkbox is not centered (see attachment) - Success icon is not flush (see attachment) - First item in customize vertex should not be selected by default (see attachment) - Edge Id name is not correct (see attachment) - Double click: was never able to see this working. If functionality is there please update the wording (no attachment) -Click and drag on graph should not route tez ui to another page (no attachment)- - Prefer to see dag view view height to extend to bottom of screen (no attachment) > Tez UI Dag View Fit and Finish > -- > > Key: TEZ-2016 > URL: https://issues.apache.org/jira/browse/TEZ-2016 > Project: Apache Tez > Issue Type: Improvement > Components: UI >Reporter: Jonathan Eagles > Attachments: Check in checkbox is not centered.png, Customize vetex > should not have first item selected by default.png, Edge Id name.png, > Gaussian Blur is not Radial.png, Green success icon not flush with edge.png, > Wording and functionality.png > > > TEZ-1973 has added a major contribution to the Tez UI. The core of the > functionality is there and is committed to community to reduce effort on > keeping patch up-to-date and to allow community exposure to the contributions. > A few fit and finish items have been identified and need to be addressed. > - Gaussian blur around circles is not a circle (see attachment) > - Check in checkbox is not centered (see attachment) > - Success icon is not flush (see attachment) > - First item in customize vertex should not be selected by default (see > attachment) > - Edge Id name is not correct (see attachment) > - Double click: was never able to see this working. If functionality is > there please update the wording (no attachment) > -Click and drag on graph should not route tez ui to another page (no > attachment)- > - Prefer to see dag view view height to extend to bottom of screen (no > attachment) > # Make source and sink bubble apt for longer text > # Bring dag back into view if it go out of view port > # Add ellipsis for clipped text in tooltip > # Use grab & grabbing cursors to indicate that user can pan the diagram > # Put Edge Source Class before Edge Destination Class > # BUG-31616 - #3 > # Make source/sink link tooltip better -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (TEZ-2016) Tez UI Dag View Fit and Finish
[ https://issues.apache.org/jira/browse/TEZ-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Somarajapuram updated TEZ-2016: Attachment: TEZ-2016.1.patch [~jeagles] [~pramachandran] Please review and help to get the patch in. > Tez UI Dag View Fit and Finish > -- > > Key: TEZ-2016 > URL: https://issues.apache.org/jira/browse/TEZ-2016 > Project: Apache Tez > Issue Type: Improvement > Components: UI >Reporter: Jonathan Eagles >Assignee: Sreenath Somarajapuram > Attachments: Check in checkbox is not centered.png, Customize vetex > should not have first item selected by default.png, Edge Id name.png, > Gaussian Blur is not Radial.png, Green success icon not flush with edge.png, > TEZ-2016.1.patch, Wording and functionality.png > > > TEZ-1973 has added a major contribution to the Tez UI. The core of the > functionality is there and is committed to community to reduce effort on > keeping patch up-to-date and to allow community exposure to the contributions. > A few fit and finish items have been identified and need to be addressed. > - Gaussian blur around circles is not a circle (see attachment) > - Check in checkbox is not centered (see attachment) > - Success icon is not flush (see attachment) > - First item in customize vertex should not be selected by default (see > attachment) > - Edge Id name is not correct (see attachment) > - Double click: was never able to see this working. If functionality is > there please update the wording (no attachment) > -Click and drag on graph should not route tez ui to another page (no > attachment)- > - Prefer to see dag view view height to extend to bottom of screen (no > attachment) > # Make source and sink bubble apt for longer text > # Bring dag back into view if it go out of view port > # Add ellipsis for clipped text in tooltip > # Use grab & grabbing cursors to indicate that user can pan the diagram > # Put Edge Source Class before Edge Destination Class > # BUG-31616 - #3 > # Make source/sink link tooltip better -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2016) Tez UI Dag View Fit and Finish
[ https://issues.apache.org/jira/browse/TEZ-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491947#comment-14491947 ] Sreenath Somarajapuram commented on TEZ-2016: - Almost all the individual changes are one liners, hence thought to club them together. Extra items in the patch was added to the description. - Gaussian blur: have changed the source and sink into round rects. - Check in checkbox is not centered: Will be resolved on changing column selector to a bootstrap model. - First item in customize vertex should not be selected by default: Please elaborate, selected items are the default columns displayed in vertex table, and represent the basic data some would be looking for. Should we remove them? - Double click: Is the new message good? > Tez UI Dag View Fit and Finish > -- > > Key: TEZ-2016 > URL: https://issues.apache.org/jira/browse/TEZ-2016 > Project: Apache Tez > Issue Type: Improvement > Components: UI >Reporter: Jonathan Eagles >Assignee: Sreenath Somarajapuram > Attachments: Check in checkbox is not centered.png, Customize vetex > should not have first item selected by default.png, Edge Id name.png, > Gaussian Blur is not Radial.png, Green success icon not flush with edge.png, > TEZ-2016.1.patch, Wording and functionality.png > > > TEZ-1973 has added a major contribution to the Tez UI. The core of the > functionality is there and is committed to community to reduce effort on > keeping patch up-to-date and to allow community exposure to the contributions. > A few fit and finish items have been identified and need to be addressed. > - Gaussian blur around circles is not a circle (see attachment) > - Check in checkbox is not centered (see attachment) > - Success icon is not flush (see attachment) > - First item in customize vertex should not be selected by default (see > attachment) > - Edge Id name is not correct (see attachment) > - Double click: was never able to see this working. If functionality is > there please update the wording (no attachment) > -Click and drag on graph should not route tez ui to another page (no > attachment)- > - Prefer to see dag view view height to extend to bottom of screen (no > attachment) > # Make source and sink bubble apt for longer text > # Bring dag back into view if it go out of view port > # Add ellipsis for clipped text in tooltip > # Use grab & grabbing cursors to indicate that user can pan the diagram > # Put Edge Source Class before Edge Destination Class > # BUG-31616 - #3 > # Make source/sink link tooltip better -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Failed: TEZ-2016 PreCommit Build #452
Jira: https://issues.apache.org/jira/browse/TEZ-2016 Build: https://builds.apache.org/job/PreCommit-TEZ-Build/452/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 2770 lines...] {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12724879/TEZ-2016.1.patch against master revision bd9b8d9. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/452//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/452//console This message is automatically generated. == == Adding comment to Jira. == == Comment added. 5312a74f552694b03b2395ede8db9123e997d304 logged out == == Finished build. == == Build step 'Execute shell' marked build as failure Archiving artifacts Sending artifact delta relative to PreCommit-TEZ-Build #441 Archived 44 artifacts Archive block size is 32768 Received 8 blocks and 2486960 bytes Compression is 9.5% Took 1.9 sec [description-setter] Could not determine description. Recording test results Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## All tests passed
[jira] [Commented] (TEZ-2016) Tez UI Dag View Fit and Finish
[ https://issues.apache.org/jira/browse/TEZ-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491987#comment-14491987 ] TezQA commented on TEZ-2016: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12724879/TEZ-2016.1.patch against master revision bd9b8d9. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/452//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/452//console This message is automatically generated. > Tez UI Dag View Fit and Finish > -- > > Key: TEZ-2016 > URL: https://issues.apache.org/jira/browse/TEZ-2016 > Project: Apache Tez > Issue Type: Improvement > Components: UI >Reporter: Jonathan Eagles >Assignee: Sreenath Somarajapuram > Attachments: Check in checkbox is not centered.png, Customize vetex > should not have first item selected by default.png, Edge Id name.png, > Gaussian Blur is not Radial.png, Green success icon not flush with edge.png, > TEZ-2016.1.patch, Wording and functionality.png > > > TEZ-1973 has added a major contribution to the Tez UI. The core of the > functionality is there and is committed to community to reduce effort on > keeping patch up-to-date and to allow community exposure to the contributions. > A few fit and finish items have been identified and need to be addressed. > - Gaussian blur around circles is not a circle (see attachment) > - Check in checkbox is not centered (see attachment) > - Success icon is not flush (see attachment) > - First item in customize vertex should not be selected by default (see > attachment) > - Edge Id name is not correct (see attachment) > - Double click: was never able to see this working. If functionality is > there please update the wording (no attachment) > -Click and drag on graph should not route tez ui to another page (no > attachment)- > - Prefer to see dag view view height to extend to bottom of screen (no > attachment) > # Make source and sink bubble apt for longer text > # Bring dag back into view if it go out of view port > # Add ellipsis for clipped text in tooltip > # Use grab & grabbing cursors to indicate that user can pan the diagram > # Put Edge Source Class before Edge Destination Class > # BUG-31616 - #3 > # Make source/sink link tooltip better -- This message was sent by Atlassian JIRA (v6.3.4#6332)