[
https://issues.apache.org/jira/browse/TEZ-776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000732#comment-14000732
]
Siddharth Seth commented on TEZ-776:
------------------------------------
The first option keeps everything within framework code
(EdgeManager/VertexManager), the second one pushes this into user code (actual
plugins), and the third one is a hybrid approach. The drawback with the second
approach, ofcourse, is that this is something users should typically not need
to worry about, but is likely to be the most optimal in terms of memory
utilization. Again, Option 1 with a possibility of a hybrid approach in the
future is what I'm leaning towards.
In terms of event fetching, that goes hand in hand with this change. Storing
events with the Edge/Vertex Manager rather than the Vertex has the advantage of
easily controlling obsoletion of events, etc. Essentially, a fetch event call
would end up going to all Vertex/EdgeManagers which maintain per-task
information.
> Reduce AM mem usage caused by storing TezEvents
> -----------------------------------------------
>
> Key: TEZ-776
> URL: https://issues.apache.org/jira/browse/TEZ-776
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
>
> This is open ended at the moment.
> A fair chunk of the AM heap is taken up by TezEvents (specifically
> DataMovementEvents - 64 bytes per event).
> Depending on the connection pattern - this puts limits on the number of tasks
> that can be processed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
