[
https://issues.apache.org/jira/browse/TEZ-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999084#comment-13999084
]
Hitesh Shah commented on TEZ-1119:
----------------------------------
Assumptions:
- User payloads can be in any format
- User payloads can only be understood by the entities/objects that consume
them. Therefore, converting them into meaningful imformation that a UI can
consume can only be done by these entities. There cannot be a generic
translator.
Options for handling user payloads:
1) For each user payload in the dag plan, the appropriate entity can be used to
convert this into some consumable form. The form could be a simple string or
json or some defined format. This would be done in the AM via a separate
thread/threadpool to ensure it does not block the main processing flow. This
data would be pushed into the history queue as a separate event type. Main
disadvantage of this approach is that it adds more weight on the AM that has to
do all this work.
2) This option offloads some of the burden of user payload conversion into a
given task. For entities that exist only within the AM ( vertex managers, edge
managers - the work would be done in the AM ). This requires the task to send
the necessary event to ATS for history ( or back to the AM if a non-ATS history
is configured ). The complexity in this is that the AM has to designate a task
to this work. It could be as simple as designating the first task to be
launched to do this work or first task index. In cases of vertices with no
tasks, this work would still need to be handled in the AM. This option however
breaks if the task fails to launch - not sure if YARN makes that visible to the
AM.
[~bikassaha] [~sseth] [~jeagles] [~rohini] Comments/Suggestions?
> Support display of user payloads in Tez UI
> ------------------------------------------
>
> Key: TEZ-1119
> URL: https://issues.apache.org/jira/browse/TEZ-1119
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Hitesh Shah
> Assignee: Hitesh Shah
>
> Applications like Pig rely heavily on configurations bundled in as part of
> the user payload for debugging. This requires each Input/Output/Processor's
> user payload to be converted into something that easily consumable by the UI
> layer to be displayed back to the user as meaningful data.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
