[jira] [Commented] (TEZ-3708) Improve parallelism and auto grouping of unpartitioned cartesian product

2017-05-10 Thread Zhiyuan Yang (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005947#comment-16005947 ] Zhiyuan Yang commented on TEZ-3708: --- Upload new patch to fix test failure and address [~ss

[jira] [Updated] (TEZ-3708) Improve parallelism and auto grouping of unpartitioned cartesian product

2017-05-10 Thread Zhiyuan Yang (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhiyuan Yang updated TEZ-3708: -- Attachment: TEZ-3708.4.patch > Improve parallelism and auto grouping of unpartitioned cartesian product >

[jira] [Commented] (TEZ-3719) DAGImpl.computeProgress slows down dispatcher and ipc threads

2017-05-10 Thread TezQA (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005930#comment-16005930 ] TezQA commented on TEZ-3719: {color:red}-1 overall{color}. Here are the results of testing the

Failed: TEZ-3719 PreCommit Build #2428

2017-05-10 Thread Apache Jenkins Server
Jira: https://issues.apache.org/jira/browse/TEZ-3719 Build: https://builds.apache.org/job/PreCommit-TEZ-Build/2428/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated

[jira] [Commented] (TEZ-3719) DAGImpl.computeProgress slows down dispatcher and ipc threads

2017-05-10 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005870#comment-16005870 ] Rajesh Balamohan commented on TEZ-3719: --- Patch lgtm. +1 pending jenkins tests. > DAGI

[jira] [Created] (TEZ-3721) Extend CallableWithNdc to also clone MDC

2017-05-10 Thread Prasanth Jayachandran (JIRA)
Prasanth Jayachandran created TEZ-3721: -- Summary: Extend CallableWithNdc to also clone MDC Key: TEZ-3721 URL: https://issues.apache.org/jira/browse/TEZ-3721 Project: Apache Tez Issue Typ

[jira] [Commented] (TEZ-3719) DAGImpl.computeProgress slows down dispatcher and ipc threads

2017-05-10 Thread Gopal V (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005809#comment-16005809 ] Gopal V commented on TEZ-3719: -- [~jeagles]: I think I have made all of those changes and increa

[jira] [Updated] (TEZ-3719) DAGImpl.computeProgress slows down dispatcher and ipc threads

2017-05-10 Thread Gopal V (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated TEZ-3719: - Description: The Vertex readLock is held while computing progress, which prevents scheduling of tasks from happenin

[jira] [Updated] (TEZ-3720) Optimize SimpleHistoryLoggingService: FileSystem$Stats updates are expensive than writes

2017-05-10 Thread Gopal V (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated TEZ-3720: - Priority: Minor (was: Major) > Optimize SimpleHistoryLoggingService: FileSystem$Stats updates are expensive > than

[jira] [Updated] (TEZ-3720) Optimize SimpleHistoryLoggingService: FileSystem$Stats updates are expensive than writes

2017-05-10 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-3720: -- Attachment: FS$Stats.incrementBytesWritten.png > Optimize SimpleHistoryLoggingService: FileSystem$

[jira] [Created] (TEZ-3720) Optimize SimpleHistoryLoggingService: FileSystem$Stats updates are expensive than writes

2017-05-10 Thread Rajesh Balamohan (JIRA)
Rajesh Balamohan created TEZ-3720: - Summary: Optimize SimpleHistoryLoggingService: FileSystem$Stats updates are expensive than writes Key: TEZ-3720 URL: https://issues.apache.org/jira/browse/TEZ-3720

[jira] [Commented] (TEZ-3708) Improve parallelism and auto grouping of unpartitioned cartesian product

2017-05-10 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005671#comment-16005671 ] Siddharth Seth commented on TEZ-3708: - - {code} +minNumRecordForEstimation = +

[jira] [Updated] (TEZ-3719) DAGImpl.computeProgress slows down dispatcher and ipc threads

2017-05-10 Thread Gopal V (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated TEZ-3719: - Attachment: TEZ-3719.1.patch > DAGImpl.computeProgress slows down dispatcher and ipc threads > -

[jira] [Commented] (TEZ-3719) DAGImpl.computeProgress slows down dispatcher and ipc threads

2017-05-10 Thread Gopal V (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005429#comment-16005429 ] Gopal V commented on TEZ-3719: -- One of these getProgress() calls is coming from the AM-RM heart

[jira] [Commented] (TEZ-3712) Use Local FileContext for deleting dag level directories

2017-05-10 Thread Kuhu Shukla (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005420#comment-16005420 ] Kuhu Shukla commented on TEZ-3712: -- Did some more analysis as to why we see discrepancy in

[jira] [Commented] (TEZ-3718) Better handling of 'bad' nodes

2017-05-10 Thread Jason Lowe (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005282#comment-16005282 ] Jason Lowe commented on TEZ-3718: - Last I checked, MapReduce doesn't have task duration fact

[jira] [Commented] (TEZ-3719) DAGImpl.computeProgress slows down dispatcher and ipc threads

2017-05-10 Thread Jonathan Eagles (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005251#comment-16005251 ] Jonathan Eagles commented on TEZ-3719: -- I noticed this exact slow down when I was profi

[jira] [Comment Edited] (TEZ-3718) Better handling of 'bad' nodes

2017-05-10 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005247#comment-16005247 ] Siddharth Seth edited comment on TEZ-3718 at 5/10/17 7:15 PM: --

[jira] [Commented] (TEZ-3718) Better handling of 'bad' nodes

2017-05-10 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005247#comment-16005247 ] Siddharth Seth commented on TEZ-3718: - bq. Not specifically related to bad node handling

[jira] [Assigned] (TEZ-3719) DAGImpl.computeProgress slows down dispatcher and ipc threads

2017-05-10 Thread Gopal V (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V reassigned TEZ-3719: Assignee: Gopal V > DAGImpl.computeProgress slows down dispatcher and ipc threads > -

[jira] [Updated] (TEZ-3719) DAGImpl.computeProgress slows down dispatcher and ipc threads

2017-05-10 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-3719: -- Attachment: TEZ-3719.wip.patch wip to reduce read lock contention in DAGImpl and VertexImpl getPro

[jira] [Updated] (TEZ-3719) DAGImpl.computeProgress slows down dispatcher and ipc threads

2017-05-10 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-3719: -- Attachment: tez_computeProgress_waiting_for_lock.png dispatcher_holding_lock.png E

[jira] [Created] (TEZ-3719) DAGImpl.computeProgress slows down dispatcher and ipc threads

2017-05-10 Thread Rajesh Balamohan (JIRA)
Rajesh Balamohan created TEZ-3719: - Summary: DAGImpl.computeProgress slows down dispatcher and ipc threads Key: TEZ-3719 URL: https://issues.apache.org/jira/browse/TEZ-3719 Project: Apache Tez