[jira] [Commented] (TEZ-1647) Issue with caching of events in VertexManager::onRootVertexInitialized

2014-10-09 Thread Vikram Dixit K (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164887#comment-14164887 ] Vikram Dixit K commented on TEZ-1647: - Yes. It is the custom vertex manager. > Issue wi

[jira] [Commented] (TEZ-1647) Issue with caching of events in VertexManager::onRootVertexInitialized

2014-10-09 Thread Vikram Dixit K (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164889#comment-14164889 ] Vikram Dixit K commented on TEZ-1647: - Also take a look at HIVE-8409 for some fixes that

[jira] [Created] (TEZ-1649) ShuffleVertexManager auto reduce parallelism can cause jobs to hang indefinitely (with ScatterGather edges)

2014-10-09 Thread Rajesh Balamohan (JIRA)
Rajesh Balamohan created TEZ-1649: - Summary: ShuffleVertexManager auto reduce parallelism can cause jobs to hang indefinitely (with ScatterGather edges) Key: TEZ-1649 URL: https://issues.apache.org/jira/browse/TEZ

[jira] [Updated] (TEZ-1649) ShuffleVertexManager auto reduce parallelism can cause jobs to hang indefinitely (with ScatterGather edges)

2014-10-09 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-1649: -- Attachment: TEZ-1649.png > ShuffleVertexManager auto reduce parallelism can cause jobs to hang >

[jira] [Updated] (TEZ-1649) ShuffleVertexManager auto reduce parallelism can cause jobs to hang indefinitely (with ScatterGather edges)

2014-10-09 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-1649: -- Attachment: TEZ-1649.1.patch [~bikassaha] - Can you please review? https://reviews.apache.org/r/26

[jira] [Created] (TEZ-1650) Please create a DOAP file for your TLP

2014-10-09 Thread Sebb (JIRA)
Sebb created TEZ-1650: - Summary: Please create a DOAP file for your TLP Key: TEZ-1650 URL: https://issues.apache.org/jira/browse/TEZ-1650 Project: Apache Tez Issue Type: Task Reporter: Sebb

[jira] [Updated] (TEZ-1647) Issue with caching of events in VertexManager::onRootVertexInitialized

2014-10-09 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-1647: Attachment: Tez-1647.patch > Issue with caching of events in VertexManager::onRootVertexInitialized > ---

[jira] [Commented] (TEZ-1647) Issue with caching of events in VertexManager::onRootVertexInitialized

2014-10-09 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14165152#comment-14165152 ] Jeff Zhang commented on TEZ-1647: - Check the CustomPartitionVertex, it indeed have the case

[jira] [Updated] (TEZ-1640) Unable to achieve Secured Impersonation

2014-10-09 Thread Subroto Sanyal (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subroto Sanyal updated TEZ-1640: Attachment: client_log.txt app_master_syslog.txt hi [~bikassaha], Attached the client

[jira] [Commented] (TEZ-1647) Issue with caching of events in VertexManager::onRootVertexInitialized

2014-10-09 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14165345#comment-14165345 ] Hitesh Shah commented on TEZ-1647: -- [~zjffdu] A linkedblockingqueue might help instead of u

[jira] [Assigned] (TEZ-1605) Landing page for Tez UI

2014-10-09 Thread Prakash Ramachandran (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakash Ramachandran reassigned TEZ-1605: - Assignee: Prakash Ramachandran > Landing page for Tez UI > ---

[jira] [Updated] (TEZ-1605) Landing page for Tez UI

2014-10-09 Thread Prakash Ramachandran (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakash Ramachandran updated TEZ-1605: -- Attachment: tez-ui.2.patch - added pages for dag, dag details, counters, tasks, task attem

[jira] [Comment Edited] (TEZ-1605) Landing page for Tez UI

2014-10-09 Thread Prakash Ramachandran (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14165450#comment-14165450 ] Prakash Ramachandran edited comment on TEZ-1605 at 10/9/14 6:10 PM: --

[jira] [Updated] (TEZ-1645) Add support for specifying additional local resources via config

2014-10-09 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-1645: - Attachment: TEZ-1645.1.patch [~bikassaha] [~jeagles] please review. > Add support for specifying additiona

[jira] [Assigned] (TEZ-1646) Add support for augmenting classpath via configs

2014-10-09 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah reassigned TEZ-1646: Assignee: Hitesh Shah > Add support for augmenting classpath via configs > --

[jira] [Commented] (TEZ-1590) Fetchers should not report failures after the Processor on the task completes

2014-10-09 Thread Rohini Palaniswamy (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14165699#comment-14165699 ] Rohini Palaniswamy commented on TEZ-1590: - [~sseth], Will you be fixing this soon

[jira] [Commented] (TEZ-1645) Add support for specifying additional local resources via config

2014-10-09 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14165769#comment-14165769 ] Bikas Saha commented on TEZ-1645: - We should probably allow file paths (non-dir) paths in au

[jira] [Commented] (TEZ-1643) DAGAppMaster kills DAG & shuts down, when RM is restarted

2014-10-09 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14165778#comment-14165778 ] Bikas Saha commented on TEZ-1643: - [~hitesh] suggests that we could do the simple thing of s

[jira] [Commented] (TEZ-1614) Use setFromConfiguration() in SortMergeJoinExample to demonstrate the usage

2014-10-09 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14165780#comment-14165780 ] Bikas Saha commented on TEZ-1614: - was this committed? Is this done? > Use setFromConfigura

[jira] [Resolved] (TEZ-1562) DAGImpl commitOrAbortOutputs takes long time (300+ seconds) for reducer vertex with 4000+ tasks

2014-10-09 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha resolved TEZ-1562. - Resolution: Invalid This is not a Tez issue. Its a MapReduce issue. > DAGImpl commitOrAbortOutputs takes lo

[jira] [Commented] (TEZ-1649) ShuffleVertexManager auto reduce parallelism can cause jobs to hang indefinitely (with ScatterGather edges)

2014-10-09 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14165791#comment-14165791 ] Bikas Saha commented on TEZ-1649: - Do we also need to augment this with a fix similar to TEZ

[jira] [Comment Edited] (TEZ-1649) ShuffleVertexManager auto reduce parallelism can cause jobs to hang indefinitely (with ScatterGather edges)

2014-10-09 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14165791#comment-14165791 ] Bikas Saha edited comment on TEZ-1649 at 10/9/14 9:43 PM: -- Do we al

[jira] [Commented] (TEZ-1649) ShuffleVertexManager auto reduce parallelism can cause jobs to hang indefinitely (with ScatterGather edges)

2014-10-09 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14165816#comment-14165816 ] Bikas Saha commented on TEZ-1649: - On the patch itself. Minor comment. sourceVertex? {code}+

[jira] [Commented] (TEZ-1614) Use setFromConfiguration() in SortMergeJoinExample to demonstrate the usage

2014-10-09 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14165862#comment-14165862 ] Rajesh Balamohan commented on TEZ-1614: --- Not yet committed [~bikassaha]. I can commit

[jira] [Comment Edited] (TEZ-1614) Use setFromConfiguration() in SortMergeJoinExample to demonstrate the usage

2014-10-09 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14165862#comment-14165862 ] Rajesh Balamohan edited comment on TEZ-1614 at 10/9/14 10:16 PM: -

[jira] [Updated] (TEZ-1645) Add support for specifying additional local resources via config

2014-10-09 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-1645: - Attachment: TEZ-1645.2.patch Addressed comments and added a test case to catch scanning for files/dirs. >

[jira] [Commented] (TEZ-1645) Add support for specifying additional local resources via config

2014-10-09 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14165897#comment-14165897 ] Hitesh Shah commented on TEZ-1645: -- [~bikassaha] please review. > Add support for specify

[jira] [Commented] (TEZ-1647) Issue with caching of events in VertexManager::onRootVertexInitialized

2014-10-09 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14165904#comment-14165904 ] Rajesh Balamohan commented on TEZ-1647: --- Tested it on my local-vm where the hive job w

[jira] [Commented] (TEZ-1614) Use setFromConfiguration() in SortMergeJoinExample to demonstrate the usage

2014-10-09 Thread Bikas Saha (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14165902#comment-14165902 ] Bikas Saha commented on TEZ-1614: - lgtm. > Use setFromConfiguration() in SortMergeJoinExamp

[jira] [Resolved] (TEZ-1614) Use setFromConfiguration() in SortMergeJoinExample to demonstrate the usage

2014-10-09 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan resolved TEZ-1614. --- Resolution: Fixed Fix Version/s: 0.5.2 Hadoop Flags: Reviewed Thanks [~bikassaha]. C

[jira] [Updated] (TEZ-1646) Add support for augmenting classpath via configs

2014-10-09 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-1646: - Attachment: TEZ-1646.1.patch > Add support for augmenting classpath via configs > --

[jira] [Commented] (TEZ-1646) Add support for augmenting classpath via configs

2014-10-09 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14165968#comment-14165968 ] Hitesh Shah commented on TEZ-1646: -- [~bikassaha] review please. > Add support for augment

[jira] [Resolved] (TEZ-1641) Add debug logs in VertexManager to help debugging custom VertexManagerPlugins

2014-10-09 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan resolved TEZ-1641. --- Resolution: Fixed Fix Version/s: 0.5.2 Hadoop Flags: Reviewed committed to master an

[jira] [Updated] (TEZ-1644) Issue due to Vertex/Task/TaskAttempt can finish multiple times

2014-10-09 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-1644: Description: Vertex/Task/TaskAttempt can move from SUCCEEDED to FAILED/KILLED, that means it would finish mul

[jira] [Updated] (TEZ-1644) Issue due to Vertex/Task/TaskAttempt can finish multiple times

2014-10-09 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhang updated TEZ-1644: Description: Vertex/Task/TaskAttempt can move from SUCCEEDED to FAILED/KILLED, that means it would finish mul

[jira] [Updated] (TEZ-1643) DAGAppMaster kills DAG & shuts down, when RM is restarted

2014-10-09 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-1643: - Attachment: TEZ-1643.wip.patch A work-in-progress patch if anyone wants to take a look. [~bikassaha] [~raj

[jira] [Updated] (TEZ-1643) DAGAppMaster kills DAG & shuts down, when RM is restarted

2014-10-09 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-1643: - Attachment: TEZ-1643.wip.patch > DAGAppMaster kills DAG & shuts down, when RM is restarted > ---

[jira] [Updated] (TEZ-1643) DAGAppMaster kills DAG & shuts down, when RM is restarted

2014-10-09 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-1643: - Attachment: (was: TEZ-1643.wip.patch) > DAGAppMaster kills DAG & shuts down, when RM is restarted >

[jira] [Updated] (TEZ-1643) DAGAppMaster kills DAG & shuts down, when RM is restarted

2014-10-09 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated TEZ-1643: - Attachment: TEZ-1643.wip.2.patch Added patch with missing file. > DAGAppMaster kills DAG & shuts down, whe

[jira] [Commented] (TEZ-1470) Recovery fail due to TaskAttemptFinishedEvent is recorded multiple times for the same task

2014-10-09 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14166373#comment-14166373 ] Hitesh Shah commented on TEZ-1470: -- Looking at the patch in general, had a question: - d

[jira] [Updated] (TEZ-1605) Landing page for Tez UI

2014-10-09 Thread Prakash Ramachandran (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prakash Ramachandran updated TEZ-1605: -- Attachment: tez-ui.3.patch minor fix to previous patch - timelineserver url was hardcoded

[jira] [Assigned] (TEZ-1649) ShuffleVertexManager auto reduce parallelism can cause jobs to hang indefinitely (with ScatterGather edges)

2014-10-09 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan reassigned TEZ-1649: - Assignee: Rajesh Balamohan > ShuffleVertexManager auto reduce parallelism can cause jobs to

[jira] [Updated] (TEZ-1649) ShuffleVertexManager auto reduce parallelism can cause jobs to hang indefinitely (with ScatterGather edges)

2014-10-09 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated TEZ-1649: -- Attachment: TEZ-1649.2.patch Attaching the revised patch which follows similar approach to TEZ-149

[jira] [Commented] (TEZ-1470) Recovery fail due to TaskAttemptFinishedEvent is recorded multiple times for the same task

2014-10-09 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/TEZ-1470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14166432#comment-14166432 ] Jeff Zhang commented on TEZ-1470: - [~hitesh] Use map looks clean to me, I saw some ugly code