[jira] [Commented] (TEZ-3362) Delete intermediate data at DAG level for Shuffle Handler

[ https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503138#comment-17503138 ] Thomas Graves commented on TEZ-3362: [~jeagles] [~kshukla]  I know its been a while, I was looking at

[jira] [Commented] (TEZ-3362) Delete intermediate data at DAG level for Shuffle Handler

[ https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15589194#comment-15589194 ] Jonathan Eagles commented on TEZ-3362: -- [~kshukla], latest patch is in line with proposed design and

[jira] [Commented] (TEZ-3362) Delete intermediate data at DAG level for Shuffle Handler

[ https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536590#comment-15536590 ] Kuhu Shukla commented on TEZ-3362: -- bq. 1. dagDeleteService - if I'm not mistaken this is setup each time a

[jira] [Commented] (TEZ-3362) Delete intermediate data at DAG level for Shuffle Handler

[ https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536556#comment-15536556 ] Siddharth Seth commented on TEZ-3362: - The dagComplete notification that goes out to the ServicePlugins

[jira] [Commented] (TEZ-3362) Delete intermediate data at DAG level for Shuffle Handler

[ https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536448#comment-15536448 ] Hitesh Shah commented on TEZ-3362: -- bq. The DagAppMaster will initialize this "Deletion Service" in a

[jira] [Commented] (TEZ-3362) Delete intermediate data at DAG level for Shuffle Handler

[ https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536443#comment-15536443 ] Hitesh Shah commented on TEZ-3362: -- Sounds good. I would separate path based deletion and service plugin

[jira] [Commented] (TEZ-3362) Delete intermediate data at DAG level for Shuffle Handler

[ https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536352#comment-15536352 ] Kuhu Shukla commented on TEZ-3362: -- [~hitesh], we can move the generic service plugin effort, which would

[jira] [Commented] (TEZ-3362) Delete intermediate data at DAG level for Shuffle Handler

[ https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534755#comment-15534755 ] Kuhu Shukla commented on TEZ-3362: -- bq. I am not sure how this works in the hive-llap mode. This

[jira] [Commented] (TEZ-3362) Delete intermediate data at DAG level for Shuffle Handler

[ https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533903#comment-15533903 ] Jonathan Eagles commented on TEZ-3362: -- +1. Looks good to me. We can address the vertex design in the

[jira] [Commented] (TEZ-3362) Delete intermediate data at DAG level for Shuffle Handler

[ https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533633#comment-15533633 ] Jonathan Eagles commented on TEZ-3362: -- [~kshukla], couple of minor things. In general, I think we can

[jira] [Commented] (TEZ-3362) Delete intermediate data at DAG level for Shuffle Handler

[ https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533391#comment-15533391 ] Kuhu Shukla commented on TEZ-3362: -- The patch missed out some checks in LocalContainerLauncher and

[jira] [Commented] (TEZ-3362) Delete intermediate data at DAG level for Shuffle Handler

[ https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530792#comment-15530792 ] Kuhu Shukla commented on TEZ-3362: -- Thanks [~hitesh] for the comments. bq. 1) Current impl seems to be a

[jira] [Commented] (TEZ-3362) Delete intermediate data at DAG level for Shuffle Handler

[ https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530780#comment-15530780 ] Hitesh Shah commented on TEZ-3362: -- 2 main concerns with the current impl: 1) Current impl seems to be a

[jira] [Commented] (TEZ-3362) Delete intermediate data at DAG level for Shuffle Handler

[ https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530293#comment-15530293 ] Kuhu Shukla commented on TEZ-3362: -- Apologies for the accidental Summary line change. > Delete

[jira] [Commented] (TEZ-3362) Delete intermediate data at DAG level for Shuffle Handler

[ https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15458773#comment-15458773 ] Kuhu Shukla commented on TEZ-3362: -- Thanks [~ebadger] for the comment. That is right. I am currently

[jira] [Commented] (TEZ-3362) Delete intermediate data at DAG level for Shuffle Handler

[ https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15458757#comment-15458757 ] Eric Badger commented on TEZ-3362: -- Hi [~kshukla], thanks for the patch! I have just one comment before