[jira] [Updated] (TEZ-1656) Grouping of splits should maintain the original ordering of splits within a group

2014-10-14 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-1656: Attachment: TEZ-1656.2.patch Updated patch with minor refactoring. Grouping of splits should maintain the

[jira] [Commented] (TEZ-1579) MR examples should be setting mapreduce.framework.name to yarn-tez

2014-10-14 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170565#comment-14170565 ] Siddharth Seth commented on TEZ-1579: - [~keyki], this mostly looks good. The setting is

[jira] [Commented] (TEZ-1634) BlockCompressorStream.finish() is called twice in IFile.close leading to Shuffle errors

2014-10-14 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170601#comment-14170601 ] Siddharth Seth commented on TEZ-1634: - [~rajesh.balamohan], [~gopalv] - does this have

[jira] [Updated] (TEZ-1649) ShuffleVertexManager auto reduce parallelism can cause jobs to hang indefinitely (with ScatterGather edges)

2014-10-14 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-1649: -- Attachment: TEZ-1649.4.patch Thanks [~bikassaha]. Attaching the patch with the suggested

[jira] [Commented] (TEZ-1659) One Pig on tez hang due to a tez setting

2014-10-14 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170622#comment-14170622 ] Rajesh Balamohan commented on TEZ-1659: --- [~sseth]: Yes, this is handled in TEZ-1649.

[jira] [Commented] (TEZ-1579) MR examples should be setting mapreduce.framework.name to yarn-tez

2014-10-14 Thread Krisztian Horvath (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170690#comment-14170690 ] Krisztian Horvath commented on TEZ-1579: You're right, I'll update it soon. MR

[jira] [Updated] (TEZ-1579) MR examples should be setting mapreduce.framework.name to yarn-tez

2014-10-14 Thread Krisztian Horvath (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Horvath updated TEZ-1579: --- Attachment: TEZ-1579-3.patch MR examples should be setting mapreduce.framework.name to

[jira] [Commented] (TEZ-1323) Insufficent diagnostics on console when dag fails due to an exception in a task

2014-10-14 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170896#comment-14170896 ] Jeff Zhang commented on TEZ-1323: - It is resolved in

[jira] [Assigned] (TEZ-1566) Reduce log verbosity

2014-10-14 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan reassigned TEZ-1566: - Assignee: Rajesh Balamohan Reduce log verbosity

[jira] [Updated] (TEZ-1566) Reduce log verbosity

2014-10-14 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-1566: -- Attachment: TEZ-1566.1.patch Post some analysis done on local cluster Package, Size of logs,

[jira] [Comment Edited] (TEZ-1566) Reduce log verbosity

2014-10-14 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170976#comment-14170976 ] Rajesh Balamohan edited comment on TEZ-1566 at 10/14/14 2:14 PM:

[jira] [Comment Edited] (TEZ-1566) Reduce log verbosity

2014-10-14 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170976#comment-14170976 ] Rajesh Balamohan edited comment on TEZ-1566 at 10/14/14 2:14 PM:

[jira] [Resolved] (TEZ-1323) Insufficent diagnostics on console when dag fails due to an exception in a task

2014-10-14 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah resolved TEZ-1323. -- Resolution: Duplicate Insufficent diagnostics on console when dag fails due to an exception in a task

[jira] [Commented] (TEZ-1579) MR examples should be setting mapreduce.framework.name to yarn-tez

2014-10-14 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171276#comment-14171276 ] Siddharth Seth commented on TEZ-1579: - +1. Looks good. Thanks [~keyki] MR examples

[jira] [Created] (TEZ-1664) Add checks to ensure that the client and AM are compatible

2014-10-14 Thread Hitesh Shah (JIRA)
Hitesh Shah created TEZ-1664: Summary: Add checks to ensure that the client and AM are compatible Key: TEZ-1664 URL: https://issues.apache.org/jira/browse/TEZ-1664 Project: Apache Tez Issue

[jira] [Commented] (TEZ-1664) Add checks to ensure that the client and AM are compatible

2014-10-14 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171303#comment-14171303 ] Bikas Saha commented on TEZ-1664: - Should this be optional so that latency is not affected

[jira] [Created] (TEZ-1665) DAGScheduler should provide a priority range instead of an exact priority

2014-10-14 Thread Bikas Saha (JIRA)
Bikas Saha created TEZ-1665: --- Summary: DAGScheduler should provide a priority range instead of an exact priority Key: TEZ-1665 URL: https://issues.apache.org/jira/browse/TEZ-1665 Project: Apache Tez

[jira] [Commented] (TEZ-1637) Improved shuffle error handling across NM restarts

2014-10-14 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171495#comment-14171495 ] Siddharth Seth commented on TEZ-1637: - Comments on the patch - In the ScatterGather

[jira] [Commented] (TEZ-1664) Add checks to ensure that the client and AM are compatible

2014-10-14 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171496#comment-14171496 ] Hitesh Shah commented on TEZ-1664: -- I was considering an option where the AM would fail to

[jira] [Commented] (TEZ-1566) Reduce log verbosity

2014-10-14 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171519#comment-14171519 ] Siddharth Seth commented on TEZ-1566: - - {code}- mapOutput.getType()); +

[jira] [Updated] (TEZ-1176) Set parallelism should end up sending an update to ATS if numTasks are updated at run-time

2014-10-14 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-1176: - Attachment: TEZ-1176.1.patch Review please. Set parallelism should end up sending an update to ATS if

[jira] [Updated] (TEZ-1665) DAGScheduler should provide a priority range instead of an exact priority

2014-10-14 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-1665: Attachment: TEZ-1665.1.patch 1) Patch replaces priority with priority range within which attempts can assign

[jira] [Created] (TEZ-1666) UserPayload should be null if the payload is not specified

2014-10-14 Thread Siddharth Seth (JIRA)
Siddharth Seth created TEZ-1666: --- Summary: UserPayload should be null if the payload is not specified Key: TEZ-1666 URL: https://issues.apache.org/jira/browse/TEZ-1666 Project: Apache Tez

[jira] [Commented] (TEZ-1666) UserPayload should be null if the payload is not specified

2014-10-14 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171740#comment-14171740 ] Jeff Zhang commented on TEZ-1666: - It looks like Context.getUserPayload won't be null, found

[jira] [Created] (TEZ-1667) Add a system test for InitializerEvents

2014-10-14 Thread Siddharth Seth (JIRA)
Siddharth Seth created TEZ-1667: --- Summary: Add a system test for InitializerEvents Key: TEZ-1667 URL: https://issues.apache.org/jira/browse/TEZ-1667 Project: Apache Tez Issue Type: Improvement

[jira] [Created] (TEZ-1668) InputInitializers should be able to register for Vertex state updates in the constructor itself

2014-10-14 Thread Siddharth Seth (JIRA)
Siddharth Seth created TEZ-1668: --- Summary: InputInitializers should be able to register for Vertex state updates in the constructor itself Key: TEZ-1668 URL: https://issues.apache.org/jira/browse/TEZ-1668

[jira] [Commented] (TEZ-1666) UserPayload should be null if the payload is not specified

2014-10-14 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171761#comment-14171761 ] Siddharth Seth commented on TEZ-1666: - Ideally, UserPayload should be null - because

[jira] [Commented] (TEZ-1665) DAGScheduler should provide a priority range instead of an exact priority

2014-10-14 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171783#comment-14171783 ] Siddharth Seth commented on TEZ-1665: - I'm not sure different priorities for speculative

[jira] [Updated] (TEZ-1566) Reduce log verbosity

2014-10-14 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-1566: -- Attachment: TEZ-1566.2.patch Isn't type useful, and doesn't add too much overhead. Type is

[jira] [Commented] (TEZ-1590) Fetchers should not report failures after the Processor on the task completes

2014-10-14 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171800#comment-14171800 ] Siddharth Seth commented on TEZ-1590: - Planning to get to this later in the week.

[jira] [Commented] (TEZ-1566) Reduce log verbosity

2014-10-14 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171804#comment-14171804 ] Siddharth Seth commented on TEZ-1566: - +1. Looks good. If possible, it'd be better to

[jira] [Resolved] (TEZ-1566) Reduce log verbosity

2014-10-14 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan resolved TEZ-1566. --- Resolution: Fixed Hadoop Flags: Reviewed Thanks [~sseth]. Committed to master and

[jira] [Commented] (TEZ-1579) MR examples should be setting mapreduce.framework.name to yarn-tez

2014-10-14 Thread Krisztian Horvath (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171926#comment-14171926 ] Krisztian Horvath commented on TEZ-1579: My pleasure, you could check TEZ-1608 as

[jira] [Updated] (TEZ-1637) Improved shuffle error handling across NM restarts

2014-10-14 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-1637: -- Attachment: TEZ-1637.2.patch In the ScatterGather Fetcher, putBackRemainingMapOutputs(host);

[jira] [Comment Edited] (TEZ-1637) Improved shuffle error handling across NM restarts

2014-10-14 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171956#comment-14171956 ] Rajesh Balamohan edited comment on TEZ-1637 at 10/15/14 4:10 AM:

[jira] [Commented] (TEZ-1083) Enable IFile RLE for DefaultSorter

2014-10-14 Thread Gopal V (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171963#comment-14171963 ] Gopal V commented on TEZ-1083: -- +1 - this enables RLE on the map-side spill. Reduce-side

[jira] [Commented] (TEZ-1277) Tez Spill handler should truncate files to reserve space on disk

2014-10-14 Thread Gopal V (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171964#comment-14171964 ] Gopal V commented on TEZ-1277: -- Need to add a NativeIO impl to do {{fallocate}}. Tez Spill