[
https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15458773#comment-15458773
]
Kuhu Shukla commented on TEZ-3362:
----------------------------------
Thanks [~ebadger] for the comment. That is right. I am currently working on how
to "discover" aux service port per NM from the AM, which is non-trivial.
DEFAULT_SHUFFLE_PORT is not readily available to the AM and can also be
misleading, for example, if the SHUFFLE_PORT_CONFIG_KEY is set to 0, we will
bind the aux service on any port and then using default port for Dag deletion
will no longer work.
Also, the serial execution will be replaced by a threadpool so that we can
improve Dag deletion when the number of nodes is large from my previous comment.
> Delete intermediate data at DAG level for Shuffle Handler
> ---------------------------------------------------------
>
> Key: TEZ-3362
> URL: https://issues.apache.org/jira/browse/TEZ-3362
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Jonathan Eagles
> Assignee: Kuhu Shukla
> Attachments: TEZ-3362.001.patch
>
>
> Applications like hive that use tez in session mode need the ability to
> delete intermediate data after a DAG completes and while the application
> continues to run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
