[ https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14377105#comment-14377105 ]
Bikas Saha commented on TEZ-714: -------------------------------- Not seen the patch yet because it may change if you agree with these comments bq. Regarding the issue of "not sure why group-commit and non-group commit need to be differentiated in different transitions. Can this be fixed by having the events for both be different? But still handled in the same transition. The transition can check if its a group commit event vs normal commit event (based on event type) - and then log for group commit. Maybe group commit event can derive from normal commit event. Is this recovery log written relevant only in the non-commit-at-end case where group commits can happen before the DAG finishes? bq. Unit test is still not perfect. Because currently in the DAGImpl/VertexImpl we run the shared thread pool in the AsynDispatcher For these tests we could choose to use the normal thread pool by overriding the setup. Since this is a new test, it can try to not depend on ordering like the existing tests do. If so, then it should be fine to use the real threadpool instead of the fake thread pool that delegates to the dispatcher. Maybe you can create a new TestCommit that starts from scratch without the hacks in TestVertexImpl. bq. For the some existing transition, like (RUNNING to ERROR due to INTERNAL ERROR) Is this for VertexImpl or DAGImpl? That sounds like a bug. Is that relevant to the commit operation though? > OutputCommitters should not run in the main AM dispatcher thread > ---------------------------------------------------------------- > > Key: TEZ-714 > URL: https://issues.apache.org/jira/browse/TEZ-714 > Project: Apache Tez > Issue Type: Improvement > Reporter: Siddharth Seth > Assignee: Jeff Zhang > Priority: Critical > Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-2.patch, Vertex_2.pdf > > > Follow up jira from TEZ-41. > 1) If there's multiple OutputCommitters on a Vertex, they can be run in > parallel. > 2) Running an OutputCommitter in the main thread blocks all other event > handling, w.r.t the DAG, and causes the event queue to back up. > 3) This should also cover shared commits that happen in the DAG. -- This message was sent by Atlassian JIRA (v6.3.4#6332)