[jira] [Commented] (TEZ-3362) Delete intermediate data at DAG level for Shuffle Handler

[Jonathan Eagles (JIRA)]

    [ 
https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533633#comment-15533633
 ] 

Jonathan Eagles commented on TEZ-3362:
--------------------------------------

[~kshukla], couple of minor things. In general, I think we can live with the 
current design for dag level delete and can do the future vertex level design 
in a follow up jira. Once these are fixed, I think this patch can go in.

{code:title=AMNodeImpl}
  // Access should be package
  public int shufflePort = ShuffleUtils.UNDEFINED_PORT;
{code}

{code:title=AMNodeTracker#nodeSeen}
  // This get is using the getName instead of getID().getId()
  Set<NodeId> nodeIds = perDagNodeMap.get(appContext.getCurrentDAG().getName());
{code}

{code:title=AMNodeTracker#dagDelete}
  // we should protect ourselves from null pointer
  for (NodeId nodeId : getPerDagNodeMap().get(dag.getID().getId())) {
{code}


> Delete intermediate data at DAG level for Shuffle Handler
> ---------------------------------------------------------
>
>                 Key: TEZ-3362
>                 URL: https://issues.apache.org/jira/browse/TEZ-3362
>             Project: Apache Tez
>          Issue Type: Sub-task
>            Reporter: Jonathan Eagles
>            Assignee: Kuhu Shukla
>         Attachments: TEZ-3362.001.patch, TEZ-3362.002.patch, 
> TEZ-3362.003.patch, TEZ-3362.004.patch
>
>
> Applications like hive that use tez in session mode need the ability to 
> delete intermediate data after a DAG completes and while the application 
> continues to run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)