[jira] [Commented] (TEZ-2410) VertexGroupCommitFinishedEvent & VertexCommitStartedEvent is not logged correctly
[ https://issues.apache.org/jira/browse/TEZ-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14533851#comment-14533851 ] Jeff Zhang commented on TEZ-2410: - Committed to branch-0.7 & master. > VertexGroupCommitFinishedEvent & VertexCommitStartedEvent is not logged > correctly > - > > Key: TEZ-2410 > URL: https://issues.apache.org/jira/browse/TEZ-2410 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jeff Zhang >Assignee: Jeff Zhang >Priority: Blocker > Fix For: 0.7.0 > > Attachments: TEZ-2410-1.patch, TEZ-2410-1.patch, TEZ-2410-2.patch, > TEZ-2410-3.patch > > > VertexGroupCommitFinishedEvent may be logged for non-vertex group commits. > VertexGroupCommitFinishedEvent may be logged for each member vertex of the > group instead of once per group. > VertexCommitStartedEvent may be logged for each output of vertex -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2410) VertexGroupCommitFinishedEvent & VertexCommitStartedEvent is not logged correctly
[
https://issues.apache.org/jira/browse/TEZ-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14533775#comment-14533775
]
TezQA commented on TEZ-2410:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12731315/TEZ-2410-3.patch
against master revision 05f77fe.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/652//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/652//console
This message is automatically generated.
> VertexGroupCommitFinishedEvent & VertexCommitStartedEvent is not logged
> correctly
> -
>
> Key: TEZ-2410
> URL: https://issues.apache.org/jira/browse/TEZ-2410
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
>Priority: Blocker
> Attachments: TEZ-2410-1.patch, TEZ-2410-1.patch, TEZ-2410-2.patch,
> TEZ-2410-3.patch
>
>
> VertexGroupCommitFinishedEvent may be logged for non-vertex group commits.
> VertexGroupCommitFinishedEvent may be logged for each member vertex of the
> group instead of once per group.
> VertexCommitStartedEvent may be logged for each output of vertex
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2410) VertexGroupCommitFinishedEvent & VertexCommitStartedEvent is not logged correctly
[
https://issues.apache.org/jira/browse/TEZ-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14533733#comment-14533733
]
Jeff Zhang commented on TEZ-2410:
-
bq. Did not find any testDAGCommitSucceeded_OnDAGSuccess. Somewhere there
should be this check for vertex v1 commit on dag success, right?
historyEventHandler.verifyVertexGroupCommitFinishedEvent("v1", 0);
What do you mean ?
historyEventHandler.verifyVertexGroupCommitFinishedEvent("v1", 0); is in
TestCommit#testDAGCommitSucceeded_OnDAGSuccess
> VertexGroupCommitFinishedEvent & VertexCommitStartedEvent is not logged
> correctly
> -
>
> Key: TEZ-2410
> URL: https://issues.apache.org/jira/browse/TEZ-2410
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
>Priority: Blocker
> Attachments: TEZ-2410-1.patch, TEZ-2410-1.patch, TEZ-2410-2.patch,
> TEZ-2410-3.patch
>
>
> VertexGroupCommitFinishedEvent may be logged for non-vertex group commits.
> VertexGroupCommitFinishedEvent may be logged for each member vertex of the
> group instead of once per group.
> VertexCommitStartedEvent may be logged for each output of vertex
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2410) VertexGroupCommitFinishedEvent & VertexCommitStartedEvent is not logged correctly
[
https://issues.apache.org/jira/browse/TEZ-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14533726#comment-14533726
]
Bikas Saha commented on TEZ-2410:
-
+1 lgtm.
Did not find any testDAGCommitSucceeded_OnDAGSuccess.
Somewhere there should be this check for vertex v1 commit on dag success, right?
historyEventHandler.verifyVertexGroupCommitFinishedEvent("v1", 0);
> VertexGroupCommitFinishedEvent & VertexCommitStartedEvent is not logged
> correctly
> -
>
> Key: TEZ-2410
> URL: https://issues.apache.org/jira/browse/TEZ-2410
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
>Priority: Blocker
> Attachments: TEZ-2410-1.patch, TEZ-2410-1.patch, TEZ-2410-2.patch,
> TEZ-2410-3.patch
>
>
> VertexGroupCommitFinishedEvent may be logged for non-vertex group commits.
> VertexGroupCommitFinishedEvent may be logged for each member vertex of the
> group instead of once per group.
> VertexCommitStartedEvent may be logged for each output of vertex
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2410) VertexGroupCommitFinishedEvent & VertexCommitStartedEvent is not logged correctly
[ https://issues.apache.org/jira/browse/TEZ-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14533677#comment-14533677 ] Jeff Zhang commented on TEZ-2410: - [~bikassaha] Sorry for my ugly mistake in my last patch. Upload new patch to address issue in comments. bq. Which test case is covered the VertexImpl change? testVertexCommit_OnVertexSuccess()? All the verification of VertexCommitStartedEvent cover this (Especially testVertexCommit_OnDAGSuccess & testVertexCommit_OnVertexSuccess ). With the change in VertexImpl, VertexCommitStartedEvent may be logged multiple times (one time for each output) bq. Which test/check is covering that vertexgroupcommit event is not written for a non-group vertex when all commits happen on dag success? testDAGCommitSucceeded_OnDAGSuccess > VertexGroupCommitFinishedEvent & VertexCommitStartedEvent is not logged > correctly > - > > Key: TEZ-2410 > URL: https://issues.apache.org/jira/browse/TEZ-2410 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jeff Zhang >Assignee: Jeff Zhang >Priority: Blocker > Attachments: TEZ-2410-1.patch, TEZ-2410-1.patch, TEZ-2410-2.patch, > TEZ-2410-3.patch > > > VertexGroupCommitFinishedEvent may be logged for non-vertex group commits. > VertexGroupCommitFinishedEvent may be logged for each member vertex of the > group instead of once per group. > VertexCommitStartedEvent may be logged for each output of vertex -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2410) VertexGroupCommitFinishedEvent & VertexCommitStartedEvent is not logged correctly
[
https://issues.apache.org/jira/browse/TEZ-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14533146#comment-14533146
]
Bikas Saha commented on TEZ-2410:
-
Shouldnt vertexGroup.isCommitted=true be replaced by
vertexGroup.commitStarted=true ? This is when the commit process starts, right?
Without this vertexGroup.isInCommitting() will return false. Not sure how tests
are passing with this.
{code}
for (final VertexGroupInfo groupInfo : vertexGroups.values()) {
if (!groupInfo.outputs.isEmpty()) {
-groupInfo.committed = true;
final Vertex v = getVertex(groupInfo.groupMembers.iterator().next());
for (final String outputName : groupInfo.outputs) {
final OutputKey outputKey = new OutputKey(outputName,
groupInfo.groupName, true);
@@ -1920,7 +1931,6 @@ public class DAGImpl implements
org.apache.tez.dag.app.dag.DAG,
+ " data, groupName=" + groupInfo.groupName);
continue;
}
- groupInfo.committed = true;{code}
This is probably not going to work with the above code
{code} // partial output may already have been in committing or committed.
fail if so
List groupList = vertexGroupInfo.get(vertex.getName());
if (groupList != null) {
for (VertexGroupInfo groupInfo : groupList) {
if (groupInfo.isInCommitting()) {
String msg = "Aborting job as committing vertex: "
+ vertex.getLogIdentifier() + " is re-running";
LOG.info(msg);
addDiagnostic(msg);
enactKill(DAGTerminationCause.VERTEX_RERUN_IN_COMMITTING,
VertexTerminationCause.VERTEX_RERUN_IN_COMMITTING);
return true;
} else if (groupInfo.isCommitted()) {{code}
1) succeededCommits looks unused - we could remove it
2) Why is vertexGroup.commitStarted=true here? this is where commit finishes,
right?
3) if condition can be replaced by vertexGroup.isCommitted();
4) unnecessary space before ++
5) missing { after if stmt
{code}
+ OutputKey outputKey = commitCompletedEvent.getOutputKey();
+ succeededCommits.add(outputKey); <<< unused
+ if (outputKey.isVertexGroupOutput){
+VertexGroupInfo vertexGroup =
vertexGroups.get(outputKey.getEntityName());
+vertexGroup.commitStarted = true; <<< why here at finish time?
+vertexGroup.successfulCommits ++; << space
+if (vertexGroup.successfulCommits == vertexGroup.outputs.size()) {
<< replace with isCommitted()
+ if (!commitAllOutputsOnSuccess) < missing {
+ try {
{code}
Which test case is covered the VertexImpl change?
testVertexCommit_OnVertexSuccess()?
Which test/check is covering that vertexgroupcommit event is not written for a
non-group vertex when all commits happen on dag success?
Rename testVertexSucceed_OnDAGSuccess() to testVertexCommit_OnDAGSuccess()?
> VertexGroupCommitFinishedEvent & VertexCommitStartedEvent is not logged
> correctly
> -
>
> Key: TEZ-2410
> URL: https://issues.apache.org/jira/browse/TEZ-2410
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
>Priority: Blocker
> Attachments: TEZ-2410-1.patch, TEZ-2410-1.patch, TEZ-2410-2.patch
>
>
> VertexGroupCommitFinishedEvent may be logged for non-vertex group commits.
> VertexGroupCommitFinishedEvent may be logged for each member vertex of the
> group instead of once per group.
> VertexCommitStartedEvent may be logged for each output of vertex
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-2410) VertexGroupCommitFinishedEvent & VertexCommitStartedEvent is not logged correctly
[ https://issues.apache.org/jira/browse/TEZ-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14531903#comment-14531903 ] Jeff Zhang commented on TEZ-2410: - Upload new patch to address the issues in comments. [~bikassaha] Please help review it. * Remove VertexGroupStatus, use method to get the status. * Add test for commit related events * Also found the VertexCommitStarted is not logged correctly, fix it in the patch. > VertexGroupCommitFinishedEvent & VertexCommitStartedEvent is not logged > correctly > - > > Key: TEZ-2410 > URL: https://issues.apache.org/jira/browse/TEZ-2410 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Jeff Zhang >Assignee: Jeff Zhang >Priority: Blocker > Attachments: TEZ-2410-1.patch, TEZ-2410-1.patch, TEZ-2410-2.patch > > > VertexGroupCommitFinishedEvent may be logged for non-vertex group commits. > VertexGroupCommitFinishedEvent may be logged for each member vertex of the > group instead of once per group. > VertexCommitStartedEvent may be logged for each output of vertex -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2410) VertexGroupCommitFinishedEvent & VertexCommitStartedEvent is not logged correctly
[
https://issues.apache.org/jira/browse/TEZ-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14531895#comment-14531895
]
TezQA commented on TEZ-2410:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12731045/TEZ-2410-2.patch
against master revision d5a0f39.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/646//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/646//console
This message is automatically generated.
> VertexGroupCommitFinishedEvent & VertexCommitStartedEvent is not logged
> correctly
> -
>
> Key: TEZ-2410
> URL: https://issues.apache.org/jira/browse/TEZ-2410
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
>Priority: Blocker
> Attachments: TEZ-2410-1.patch, TEZ-2410-1.patch, TEZ-2410-2.patch
>
>
> VertexGroupCommitFinishedEvent may be logged for non-vertex group commits.
> VertexGroupCommitFinishedEvent may be logged for each member vertex of the
> group instead of once per group.
> VertexCommitStartedEvent may be logged for each output of vertex
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
