[jira] [Created] (TEZ-1383) Move recovery related code into inner class

2014-08-06 Thread Jeff Zhang (JIRA)
Jeff Zhang created TEZ-1383: --- Summary: Move recovery related code into inner class Key: TEZ-1383 URL: https://issues.apache.org/jira/browse/TEZ-1383 Project: Apache Tez Issue Type: Improvement

[jira] [Created] (TEZ-1384) Move recovery related code into inner class

2014-08-06 Thread Jeff Zhang (JIRA)
Jeff Zhang created TEZ-1384: --- Summary: Move recovery related code into inner class Key: TEZ-1384 URL: https://issues.apache.org/jira/browse/TEZ-1384 Project: Apache Tez Issue Type: Improvement

[jira] [Updated] (TEZ-1384) Move recovery related code into inner class

2014-08-06 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-1384: Description: Currently each entity (DAG, Vertex, Task, TaskAttempt) has some common recovery code like log h

[jira] [Updated] (TEZ-978) Enhance auto parallelism tuning for queries having empty outputs or data skewness

2014-08-06 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-978: - Attachment: TEZ-978.4.wip.patch For testing, I manually set "mapred.reduce.tasks=200" for the job an

[jira] [Comment Edited] (TEZ-978) Enhance auto parallelism tuning for queries having empty outputs or data skewness

2014-08-06 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087566#comment-14087566 ] Rajesh Balamohan edited comment on TEZ-978 at 8/6/14 11:31 AM: ---

[jira] [Commented] (TEZ-1360) Provide vertex parallelism to each vertex task

2014-08-06 Thread Johannes Zillmann (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087726#comment-14087726 ] Johannes Zillmann commented on TEZ-1360: Hey [~gopalv], so there is now a getVertexP

[jira] [Commented] (TEZ-1360) Provide vertex parallelism to each vertex task

2014-08-06 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087819#comment-14087819 ] Hitesh Shah commented on TEZ-1360: -- [~gopalv] The simpler fix might be to make the api avai

[jira] [Commented] (TEZ-1345) Add checks to guarantee all init events are written to recovery to consider vertex initialized

2014-08-06 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087823#comment-14087823 ] Hitesh Shah commented on TEZ-1345: -- Sort of. In some cases, the vertex initialized event ge

[jira] [Commented] (TEZ-1372) Fix preWarm to work after recent API changes

2014-08-06 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087970#comment-14087970 ] Bikas Saha commented on TEZ-1372: - Thanks for trying it out. Why are you using MyDummyProc

[jira] [Commented] (TEZ-1194) Make TezUserPayload user facing for payload specification and change to ByteBuffer

2014-08-06 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087972#comment-14087972 ] Bikas Saha commented on TEZ-1194: - Its fine. Lets go with TezUserPayload being null if actua

[jira] [Commented] (TEZ-1194) Make TezUserPayload user facing for payload specification and change to ByteBuffer

2014-08-06 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088037#comment-14088037 ] Bikas Saha commented on TEZ-1194: - About the patch itself. "new TezUserPayload(null)" can g

[jira] [Commented] (TEZ-1372) Fix preWarm to work after recent API changes

2014-08-06 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088178#comment-14088178 ] Bikas Saha commented on TEZ-1372: - [~gopalv] [~hitesh] Review please! > Fix preWarm to work

[jira] [Updated] (TEZ-1382) Change ObjectRegistry API to allow for future extensions

2014-08-06 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated TEZ-1382: Attachment: TEZ-1382.2.patch Thanks for the review. Updated docs and attached commit patch. > Change ObjectR

[jira] [Commented] (TEZ-1231) Clean up TezRuntimeConfiguration

2014-08-06 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088207#comment-14088207 ] Bikas Saha commented on TEZ-1231: - There are some items in TezConfiguration like DagAppMaste

[jira] [Commented] (TEZ-1194) Make TezUserPayload user facing for payload specification and change to ByteBuffer

2014-08-06 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088320#comment-14088320 ] Bikas Saha commented on TEZ-1194: - bq. Its fine. Lets go with TezUserPayload being null if a

[jira] [Updated] (TEZ-671) Support View/Modify ACLs for DAGs

2014-08-06 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-671: Attachment: TEZ-671.1.patch [~sseth] [~jeagles] Mind taking a look? > Support View/Modify ACLs for DAGs > --

[jira] [Commented] (TEZ-671) Support View/Modify ACLs for DAGs

2014-08-06 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088532#comment-14088532 ] Hitesh Shah commented on TEZ-671: - Nevermind - forgot to add more docs. Will upload a new pat

[jira] [Updated] (TEZ-671) Support View/Modify ACLs for DAGs

2014-08-06 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-671: Attachment: (was: TEZ-671.1.patch) > Support View/Modify ACLs for DAGs > -

[jira] [Comment Edited] (TEZ-1372) Fix preWarm to work after recent API changes

2014-08-06 Thread Thaddeus Diamond (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088545#comment-14088545 ] Thaddeus Diamond edited comment on TEZ-1372 at 8/7/14 12:54 AM: --

[jira] [Commented] (TEZ-1372) Fix preWarm to work after recent API changes

2014-08-06 Thread Thaddeus Diamond (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088545#comment-14088545 ] Thaddeus Diamond commented on TEZ-1372: --- Woops! I was trying to make it more "sample-l

[jira] [Updated] (TEZ-671) Support View/Modify ACLs for DAGs

2014-08-06 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-671: Attachment: TEZ-671.2.patch Updated doc attached. [~sseth] [~jeagles] please review. > Support View/Modify AC

[jira] [Comment Edited] (TEZ-1372) Fix preWarm to work after recent API changes

2014-08-06 Thread Thaddeus Diamond (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088545#comment-14088545 ] Thaddeus Diamond edited comment on TEZ-1372 at 8/7/14 1:12 AM: ---

[jira] [Updated] (TEZ-1384) Move recovery related code into inner class

2014-08-06 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-1384: Issue Type: Sub-task (was: Improvement) Parent: TEZ-15 > Move recovery related code into inner class

[jira] [Updated] (TEZ-1384) Move recovery related code into inner class

2014-08-06 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-1384: Attachment: Tez-1384.patch > Move recovery related code into inner class > --

[jira] [Created] (TEZ-1385) Disk Direct fails for MapOutput when trying to use OnDiskMerger

2014-08-06 Thread Jonathan Eagles (JIRA)
Jonathan Eagles created TEZ-1385: Summary: Disk Direct fails for MapOutput when trying to use OnDiskMerger Key: TEZ-1385 URL: https://issues.apache.org/jira/browse/TEZ-1385 Project: Apache Tez

[jira] [Commented] (TEZ-1384) Move recovery related code into inner class

2014-08-06 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088625#comment-14088625 ] Jeff Zhang commented on TEZ-1384: - Attach the patch. * This patch move the logHistoryEven

[jira] [Updated] (TEZ-1385) Disk Direct fails for MapOutput when trying to use OnDiskMerger

2014-08-06 Thread Jonathan Eagles (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-1385: - Description: Stack Trace 2014-08-06 20:08:52,869 [DiskToDiskMerger [scope_75]] INFO org.apache.t

[jira] [Assigned] (TEZ-1385) Disk Direct fails for MapOutput when trying to use OnDiskMerger

2014-08-06 Thread Prakash Ramachandran (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakash Ramachandran reassigned TEZ-1385: - Assignee: Prakash Ramachandran > Disk Direct fails for MapOutput when trying to use

[jira] [Updated] (TEZ-1385) Disk Direct fails for MapOutput when trying to use OnDiskMerger

2014-08-06 Thread Prakash Ramachandran (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakash Ramachandran updated TEZ-1385: -- Attachment: TEZ-1385.1.patch [~jeagles] - I have attached the patch. were you getting thi

[jira] [Commented] (TEZ-1385) Disk Direct fails for MapOutput when trying to use OnDiskMerger

2014-08-06 Thread Jonathan Eagles (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088740#comment-14088740 ] Jonathan Eagles commented on TEZ-1385: -- +1 (non-binding). [~pramachandran], this is in

[jira] [Commented] (TEZ-1372) Fix preWarm to work after recent API changes

2014-08-06 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088757#comment-14088757 ] Bikas Saha commented on TEZ-1372: - The intent is to remove the sleep. If its added as an API

[jira] [Commented] (TEZ-1372) Fix preWarm to work after recent API changes

2014-08-06 Thread Thaddeus Diamond (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14088761#comment-14088761 ] Thaddeus Diamond commented on TEZ-1372: --- Okay great. Set that config up and will igno

[jira] [Updated] (TEZ-1385) Disk Direct fails for MapOutput when trying to use OnDiskMerger

2014-08-06 Thread Prakash Ramachandran (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakash Ramachandran updated TEZ-1385: -- Attachment: TEZ-1385.2.patch minor addition to previous patch for equals. > Disk Direct