[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[ https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14513252#comment-14513252 ] Jaydeep Vishwakarma commented on OOZIE-2129: [~shwethags], I have verified this patch by running job, It is working fine. You can commit this patch. Thank you for review. Duplicate child jobs per instance - Key: OOZIE-2129 URL: https://issues.apache.org/jira/browse/OOZIE-2129 Project: Oozie Issue Type: Bug Reporter: Shwetha G S Assignee: Jaydeep Vishwakarma Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch, OOZIE-2129-v10.patch, OOZIE-2129-v2.patch, OOZIE-2129-v3.patch, OOZIE-2129-v4.patch, OOZIE-2129-v5.patch, OOZIE-2129-v6.patch, OOZIE-2129-v7.patch, OOZIE-2129-v8.patch, OOZIE-2129-v9.patch OOZIE-1722 ensures that child job is killed at launcher restart. But this doesn't work for java actions as the tag is not passed to the child job. In case of coord action rerun, new workflow is created and hence new tag. So, it doesn't ensure old child job is killed at launcher start -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[
https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505037#comment-14505037
]
Hadoop QA commented on OOZIE-2129:
--
Testing JIRA OOZIE-2129
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1650
.Tests failed: 1
.Tests errors: 0
.The patch failed the following testcases:
.
testCoordRerunDateNeg(org.apache.oozie.command.coord.TestCoordRerunXCommand)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/2321/
Duplicate child jobs per instance
-
Key: OOZIE-2129
URL: https://issues.apache.org/jira/browse/OOZIE-2129
Project: Oozie
Issue Type: Bug
Reporter: Shwetha G S
Assignee: Jaydeep Vishwakarma
Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch,
OOZIE-2129-v10.patch, OOZIE-2129-v2.patch, OOZIE-2129-v3.patch,
OOZIE-2129-v4.patch, OOZIE-2129-v5.patch, OOZIE-2129-v6.patch,
OOZIE-2129-v7.patch, OOZIE-2129-v8.patch, OOZIE-2129-v9.patch
OOZIE-1722 ensures that child job is killed at launcher restart. But this
doesn't work for java actions as the tag is not passed to the child job.
In case of coord action rerun, new workflow is created and hence new tag. So,
it doesn't ensure old child job is killed at launcher start
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[ https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504850#comment-14504850 ] Shwetha G S commented on OOZIE-2129: Few things, I have fixed them. Can you test to see if its ok? 1. CHILD_MAPREDUCE_JOB_TAGS should be defined just once 2. Simplified LauncherMapper.propagateToHadoopConf() Duplicate child jobs per instance - Key: OOZIE-2129 URL: https://issues.apache.org/jira/browse/OOZIE-2129 Project: Oozie Issue Type: Bug Reporter: Shwetha G S Assignee: Jaydeep Vishwakarma Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch, OOZIE-2129-v2.patch, OOZIE-2129-v3.patch, OOZIE-2129-v4.patch, OOZIE-2129-v5.patch, OOZIE-2129-v6.patch, OOZIE-2129-v7.patch, OOZIE-2129-v8.patch, OOZIE-2129-v9.patch OOZIE-1722 ensures that child job is killed at launcher restart. But this doesn't work for java actions as the tag is not passed to the child job. In case of coord action rerun, new workflow is created and hence new tag. So, it doesn't ensure old child job is killed at launcher start -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[
https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504864#comment-14504864
]
Hadoop QA commented on OOZIE-2129:
--
Testing JIRA OOZIE-2129
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:red}-1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:red}-1{color} patch does not compile
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:red}-1 DISTRO{color}
.{color:red}-1{color} distro tarball fails with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/2320/
Duplicate child jobs per instance
-
Key: OOZIE-2129
URL: https://issues.apache.org/jira/browse/OOZIE-2129
Project: Oozie
Issue Type: Bug
Reporter: Shwetha G S
Assignee: Jaydeep Vishwakarma
Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch,
OOZIE-2129-v2.patch, OOZIE-2129-v3.patch, OOZIE-2129-v4.patch,
OOZIE-2129-v5.patch, OOZIE-2129-v6.patch, OOZIE-2129-v7.patch,
OOZIE-2129-v8.patch, OOZIE-2129-v9.patch
OOZIE-1722 ensures that child job is killed at launcher restart. But this
doesn't work for java actions as the tag is not passed to the child job.
In case of coord action rerun, new workflow is created and hence new tag. So,
it doesn't ensure old child job is killed at launcher start
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[ https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493908#comment-14493908 ] Jaydeep Vishwakarma commented on OOZIE-2129: Looks like no more issue with this patch. [~shwethags] , If you do not have any open issue with this patch, Can you please commit it. Duplicate child jobs per instance - Key: OOZIE-2129 URL: https://issues.apache.org/jira/browse/OOZIE-2129 Project: Oozie Issue Type: Bug Reporter: Shwetha G S Assignee: Jaydeep Vishwakarma Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch, OOZIE-2129-v2.patch, OOZIE-2129-v3.patch, OOZIE-2129-v4.patch, OOZIE-2129-v5.patch, OOZIE-2129-v6.patch, OOZIE-2129-v7.patch, OOZIE-2129-v8.patch OOZIE-1722 ensures that child job is killed at launcher restart. But this doesn't work for java actions as the tag is not passed to the child job. In case of coord action rerun, new workflow is created and hence new tag. So, it doesn't ensure old child job is killed at launcher start -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[ https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377443#comment-14377443 ] Jaydeep Vishwakarma commented on OOZIE-2129: These failures are not related to patch. I have checked in my local, All those test cases are working fine. Duplicate child jobs per instance - Key: OOZIE-2129 URL: https://issues.apache.org/jira/browse/OOZIE-2129 Project: Oozie Issue Type: Bug Reporter: Shwetha G S Assignee: Jaydeep Vishwakarma Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch, OOZIE-2129-v2.patch, OOZIE-2129-v3.patch, OOZIE-2129-v4.patch, OOZIE-2129-v5.patch, OOZIE-2129-v6.patch, OOZIE-2129-v7.patch, OOZIE-2129-v8.patch OOZIE-1722 ensures that child job is killed at launcher restart. But this doesn't work for java actions as the tag is not passed to the child job. In case of coord action rerun, new workflow is created and hence new tag. So, it doesn't ensure old child job is killed at launcher start -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[
https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375961#comment-14375961
]
Hadoop QA commented on OOZIE-2129:
--
Testing JIRA OOZIE-2129
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1623
.Tests failed: 20
.Tests errors: 0
.The patch failed the following testcases:
. testActionStart(org.apache.oozie.command.wf.TestActionStartXCommand)
.
testActionWithEscapedStringAndCDATA(org.apache.oozie.command.wf.TestActionStartXCommand)
.
testWorkflowActionRecoveryUserRetry(org.apache.oozie.service.TestRecoveryService)
. testRerun(org.apache.oozie.action.hadoop.TestRerun)
. testActionCheck(org.apache.oozie.command.wf.TestActionCheckXCommand)
.
testActionCheckTransientDuringLauncher(org.apache.oozie.command.wf.TestActionCheckXCommand)
.
testActionCheckTransientDuringMRAction(org.apache.oozie.command.wf.TestActionCheckXCommand)
. testMain(org.apache.oozie.action.hadoop.TestMapReduceMain)
.
testCoordRerunDateNeg(org.apache.oozie.command.coord.TestCoordRerunXCommand)
. testMain(org.apache.oozie.action.hadoop.TestStreamingMain)
.
testSetMapredJobName(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testSetExecutionStats_when_user_has_specified_stats_write_FALSE(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
. testMapReduce(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testMapReduceWithConfigClass(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testMapReduceWithCredentials(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testMapReduceWithUberJarEnabled(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
. testStreaming(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testStreamingConfOverride(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testSetExecutionStats_when_user_has_specified_stats_write_TRUE(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testEndWithoutConfiguration(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/2271/
Duplicate child jobs per instance
-
Key: OOZIE-2129
URL: https://issues.apache.org/jira/browse/OOZIE-2129
Project: Oozie
Issue Type: Bug
Reporter: Shwetha G S
Assignee: Jaydeep Vishwakarma
Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch,
OOZIE-2129-v2.patch, OOZIE-2129-v3.patch, OOZIE-2129-v4.patch,
OOZIE-2129-v5.patch, OOZIE-2129-v6.patch, OOZIE-2129-v7.patch
OOZIE-1722 ensures that child job is killed at launcher restart. But this
doesn't work for java actions as the tag is not passed to the child job.
In case of coord action rerun, new workflow is created and hence new tag. So,
it doesn't ensure old child job is killed at launcher start
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[
https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376122#comment-14376122
]
Hadoop QA commented on OOZIE-2129:
--
Testing JIRA OOZIE-2129
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1623
.Tests failed: 21
.Tests errors: 0
.The patch failed the following testcases:
.
testRequeueOnException(org.apache.oozie.command.coord.TestCoordPushDependencyCheckXCommand)
. testActionStart(org.apache.oozie.command.wf.TestActionStartXCommand)
.
testActionWithEscapedStringAndCDATA(org.apache.oozie.command.wf.TestActionStartXCommand)
.
testWorkflowActionRecoveryUserRetry(org.apache.oozie.service.TestRecoveryService)
. testRerun(org.apache.oozie.action.hadoop.TestRerun)
. testActionCheck(org.apache.oozie.command.wf.TestActionCheckXCommand)
.
testActionCheckTransientDuringLauncher(org.apache.oozie.command.wf.TestActionCheckXCommand)
.
testActionCheckTransientDuringMRAction(org.apache.oozie.command.wf.TestActionCheckXCommand)
. testMain(org.apache.oozie.action.hadoop.TestMapReduceMain)
.
testCoordRerunDateNeg(org.apache.oozie.command.coord.TestCoordRerunXCommand)
. testMain(org.apache.oozie.action.hadoop.TestStreamingMain)
.
testSetMapredJobName(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testSetExecutionStats_when_user_has_specified_stats_write_FALSE(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
. testMapReduce(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testMapReduceWithConfigClass(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testMapReduceWithCredentials(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testMapReduceWithUberJarEnabled(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
. testStreaming(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testStreamingConfOverride(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testSetExecutionStats_when_user_has_specified_stats_write_TRUE(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
.
testEndWithoutConfiguration(org.apache.oozie.action.hadoop.TestMapReduceActionExecutor)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/2272/
Duplicate child jobs per instance
-
Key: OOZIE-2129
URL: https://issues.apache.org/jira/browse/OOZIE-2129
Project: Oozie
Issue Type: Bug
Reporter: Shwetha G S
Assignee: Jaydeep Vishwakarma
Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch,
OOZIE-2129-v2.patch, OOZIE-2129-v3.patch, OOZIE-2129-v4.patch,
OOZIE-2129-v5.patch, OOZIE-2129-v6.patch, OOZIE-2129-v7.patch
OOZIE-1722 ensures that child job is killed at launcher restart. But this
doesn't work for java actions as the tag is not passed to the child job.
In case of coord action rerun, new workflow is created and hence new tag. So,
it doesn't ensure old child job is killed at launcher start
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[
https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376656#comment-14376656
]
Hadoop QA commented on OOZIE-2129:
--
Testing JIRA OOZIE-2129
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1623
.Tests failed: 3
.Tests errors: 0
.The patch failed the following testcases:
.
testMemoryUsageAndSpeed(org.apache.oozie.service.TestPartitionDependencyManagerService)
.
testCoordRerunDateNeg(org.apache.oozie.command.coord.TestCoordRerunXCommand)
.
testNone(org.apache.oozie.command.coord.TestCoordActionInputCheckXCommand)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/2274/
Duplicate child jobs per instance
-
Key: OOZIE-2129
URL: https://issues.apache.org/jira/browse/OOZIE-2129
Project: Oozie
Issue Type: Bug
Reporter: Shwetha G S
Assignee: Jaydeep Vishwakarma
Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch,
OOZIE-2129-v2.patch, OOZIE-2129-v3.patch, OOZIE-2129-v4.patch,
OOZIE-2129-v5.patch, OOZIE-2129-v6.patch, OOZIE-2129-v7.patch,
OOZIE-2129-v8.patch
OOZIE-1722 ensures that child job is killed at launcher restart. But this
doesn't work for java actions as the tag is not passed to the child job.
In case of coord action rerun, new workflow is created and hence new tag. So,
it doesn't ensure old child job is killed at launcher start
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[
https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372857#comment-14372857
]
Hadoop QA commented on OOZIE-2129:
--
Testing JIRA OOZIE-2129
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:red}-1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:red}-1{color} patch does not compile
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:red}-1 DISTRO{color}
.{color:red}-1{color} distro tarball fails with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/2268/
Duplicate child jobs per instance
-
Key: OOZIE-2129
URL: https://issues.apache.org/jira/browse/OOZIE-2129
Project: Oozie
Issue Type: Bug
Reporter: Shwetha G S
Assignee: Jaydeep Vishwakarma
Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch,
OOZIE-2129-v2.patch, OOZIE-2129-v3.patch, OOZIE-2129-v4.patch,
OOZIE-2129-v5.patch, OOZIE-2129-v6.patch
OOZIE-1722 ensures that child job is killed at launcher restart. But this
doesn't work for java actions as the tag is not passed to the child job.
In case of coord action rerun, new workflow is created and hence new tag. So,
it doesn't ensure old child job is killed at launcher start
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[
https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14373031#comment-14373031
]
Hadoop QA commented on OOZIE-2129:
--
Testing JIRA OOZIE-2129
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1623
.Tests failed: 10
.Tests errors: 56
.The patch failed the following testcases:
. testMain(org.apache.oozie.action.hadoop.TestStreamingMain)
.
testActionWithEscapedStringAndCDATA(org.apache.oozie.command.wf.TestActionStartXCommand)
. testActionStart(org.apache.oozie.command.wf.TestActionStartXCommand)
. testMain(org.apache.oozie.action.hadoop.TestMapReduceMain)
. testRerun(org.apache.oozie.action.hadoop.TestRerun)
.
testCoordRerunDateNeg(org.apache.oozie.command.coord.TestCoordRerunXCommand)
.
testActionCheckTransientDuringLauncher(org.apache.oozie.command.wf.TestActionCheckXCommand)
.
testActionCheckTransientDuringMRAction(org.apache.oozie.command.wf.TestActionCheckXCommand)
. testActionCheck(org.apache.oozie.command.wf.TestActionCheckXCommand)
.
testWorkflowActionRecoveryUserRetry(org.apache.oozie.service.TestRecoveryService)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/2269/
Duplicate child jobs per instance
-
Key: OOZIE-2129
URL: https://issues.apache.org/jira/browse/OOZIE-2129
Project: Oozie
Issue Type: Bug
Reporter: Shwetha G S
Assignee: Jaydeep Vishwakarma
Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch,
OOZIE-2129-v2.patch, OOZIE-2129-v3.patch, OOZIE-2129-v4.patch,
OOZIE-2129-v5.patch, OOZIE-2129-v6.patch
OOZIE-1722 ensures that child job is killed at launcher restart. But this
doesn't work for java actions as the tag is not passed to the child job.
In case of coord action rerun, new workflow is created and hence new tag. So,
it doesn't ensure old child job is killed at launcher start
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[ https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14368605#comment-14368605 ] Robert Kanter commented on OOZIE-2129: -- My main motivation for not wanting to add propagation.xml was not to add yet another small file in HDFS. But looking at it again, it's in the local filesystem, so that's not an issue. In that case, I'm fine with leaving the file as-is in the patch. Duplicate child jobs per instance - Key: OOZIE-2129 URL: https://issues.apache.org/jira/browse/OOZIE-2129 Project: Oozie Issue Type: Bug Reporter: Shwetha G S Assignee: Jaydeep Vishwakarma Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch, OOZIE-2129-v2.patch, OOZIE-2129-v3.patch, OOZIE-2129-v4.patch, OOZIE-2129-v5.patch OOZIE-1722 ensures that child job is killed at launcher restart. But this doesn't work for java actions as the tag is not passed to the child job. In case of coord action rerun, new workflow is created and hence new tag. So, it doesn't ensure old child job is killed at launcher start -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[ https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14370466#comment-14370466 ] Robert Kanter commented on OOZIE-2129: -- LGTM. +1 [~shwethags], did you have any last comments? Duplicate child jobs per instance - Key: OOZIE-2129 URL: https://issues.apache.org/jira/browse/OOZIE-2129 Project: Oozie Issue Type: Bug Reporter: Shwetha G S Assignee: Jaydeep Vishwakarma Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch, OOZIE-2129-v2.patch, OOZIE-2129-v3.patch, OOZIE-2129-v4.patch, OOZIE-2129-v5.patch OOZIE-1722 ensures that child job is killed at launcher restart. But this doesn't work for java actions as the tag is not passed to the child job. In case of coord action rerun, new workflow is created and hence new tag. So, it doesn't ensure old child job is killed at launcher start -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[
https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14370538#comment-14370538
]
Shwetha G S commented on OOZIE-2129:
Thanks [~rkanter] for reviewing.
Few minor comments:
{code}
+} else {
+launcherTag = action.getId();
+}
{code}
Not required as ActionStartXCommand handles all cases
{code}
+if (actionConf.get(child.mapreduce.job.tags) == null) {
{code}
use variable LauncherMain.CHILD_MAPREDUCE_JOB_TAGS. Use
LauncherMain.MAPREDUCE_JOB_TAGS for 'mapreduce.job.tags'
{code}
+public static final String CHILD_MAPREDUCE_JOB_TAGS =
child.mapreduce.job.tags;
{code}
rename to 'oozie.mapreduce.job.tags'?
In LauncherMapper.propagateConf(), re-use the function for loading action conf
{{writeJobIdFile(idFile, jobId)}} is not required anymore??
{code}
+arguments.add(mapreduce.job.tags= +
actionConf.get(mapreduce.job.tags));
{code}
Shouldn't this be actionConf.get(CHILD_MAPREDUCE_JOB_TAGS)?
Duplicate child jobs per instance
-
Key: OOZIE-2129
URL: https://issues.apache.org/jira/browse/OOZIE-2129
Project: Oozie
Issue Type: Bug
Reporter: Shwetha G S
Assignee: Jaydeep Vishwakarma
Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch,
OOZIE-2129-v2.patch, OOZIE-2129-v3.patch, OOZIE-2129-v4.patch,
OOZIE-2129-v5.patch
OOZIE-1722 ensures that child job is killed at launcher restart. But this
doesn't work for java actions as the tag is not passed to the child job.
In case of coord action rerun, new workflow is created and hence new tag. So,
it doesn't ensure old child job is killed at launcher start
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[ https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360045#comment-14360045 ] Shwetha G S commented on OOZIE-2129: Yes, it makes sense to add action conf as part of job conf, but it may create some configuration conflicts(depending on what user's configs are). We can make this default, but we should also give an option for the user to override(in case of issues). So, if this is overridden, we need another option to pass the yarn tags. Hence, it makes sense leave propage-conf as in the patch Duplicate child jobs per instance - Key: OOZIE-2129 URL: https://issues.apache.org/jira/browse/OOZIE-2129 Project: Oozie Issue Type: Bug Reporter: Shwetha G S Assignee: Jaydeep Vishwakarma Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch, OOZIE-2129-v2.patch, OOZIE-2129-v3.patch, OOZIE-2129-v4.patch OOZIE-1722 ensures that child job is killed at launcher restart. But this doesn't work for java actions as the tag is not passed to the child job. In case of coord action rerun, new workflow is created and hence new tag. So, it doesn't ensure old child job is killed at launcher start -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[
https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358545#comment-14358545
]
Hadoop QA commented on OOZIE-2129:
--
Testing JIRA OOZIE-2129
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:red}-1{color} the patch contains 1 line(s) with trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1596
.Tests failed: 2
.Tests errors: 0
.The patch failed the following testcases:
.
testCoordRerunDateNeg(org.apache.oozie.command.coord.TestCoordRerunXCommand)
.
testMessage_withMixedStatus(org.apache.oozie.command.coord.TestAbandonedCoordChecker)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/2251/
Duplicate child jobs per instance
-
Key: OOZIE-2129
URL: https://issues.apache.org/jira/browse/OOZIE-2129
Project: Oozie
Issue Type: Bug
Reporter: Shwetha G S
Assignee: Jaydeep Vishwakarma
Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch,
OOZIE-2129-v2.patch, OOZIE-2129-v3.patch, OOZIE-2129-v4.patch
OOZIE-1722 ensures that child job is killed at launcher restart. But this
doesn't work for java actions as the tag is not passed to the child job.
In case of coord action rerun, new workflow is created and hence new tag. So,
it doesn't ensure old child job is killed at launcher start
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[
https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343040#comment-14343040
]
Shwetha G S commented on OOZIE-2129:
[~rkanter]
Ideally, users shouldn't be using java action to launch mr job, but typically
thats how its used and its difficult to get users to migrate. Most of the users
don't know that conf is available in action conf file and they should load that
extra file (In fact, I was planning to add it by default as part of another
jira as its an un-necessary extra step for user). If they don't load action
conf, {{LauncherMainHadoopUtils.killChildYarnJobs(actionConf)}} will not work
as tags are not set. {{Configuration.addDefaultResource(PROPAGATION_CONF_XML)}}
will add the propagation conf as default conf and {{new Configuration()}} will
always have the tags(loading of an extra file is not required). So, unless
users use {{new Configuration(false)}}, tag is always passed to child job.
About using coordinator action id for tag, take the case of workflow thats
launched from the coord action. If the child job is long running and the
launcher job gets killed, the workflow and coord action will be killed. But the
child job will still be running. If the user re-runs this coord action, new
workflow will be created and hence new tag. The new launcher will not kill the
old child job in this case. Hence the tag should be on coord action id(if
workflow has parent).
Duplicate child jobs per instance
-
Key: OOZIE-2129
URL: https://issues.apache.org/jira/browse/OOZIE-2129
Project: Oozie
Issue Type: Bug
Reporter: Shwetha G S
Assignee: Jaydeep Vishwakarma
Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch,
OOZIE-2129-v2.patch, OOZIE-2129-v3.patch
OOZIE-1722 ensures that child job is killed at launcher restart. But this
doesn't work for java actions as the tag is not passed to the child job.
In case of coord action rerun, new workflow is created and hence new tag. So,
it doesn't ensure old child job is killed at launcher start
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[ https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14342919#comment-14342919 ] Jaydeep Vishwakarma commented on OOZIE-2129: 1. Added back the launch time in filter. 3. Yes this patch will fix the issue. I meant without this patch it will not work. For 2 and 5 : they have to implement some code propagation.xml is the way for the launcher to share details in case they need to take advantage of it, since there are no other simple ways or established contract to share info in a backward compatible way. 6. Yes, when any coord_action/workflow/launcher are running, oozie will not allow to rerun. The issues comes when the job launched by launcher is in running state and coord_action/workflow/launcher is in killed state, In this case, before rerunning the coord_action/workflow/launcher none of the previous launched job is running on cluster. Check my second comment in the same jira that explain in what cases this condition can occur. Duplicate child jobs per instance - Key: OOZIE-2129 URL: https://issues.apache.org/jira/browse/OOZIE-2129 Project: Oozie Issue Type: Bug Reporter: Shwetha G S Assignee: Jaydeep Vishwakarma Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch, OOZIE-2129-v2.patch OOZIE-1722 ensures that child job is killed at launcher restart. But this doesn't work for java actions as the tag is not passed to the child job. In case of coord action rerun, new workflow is created and hence new tag. So, it doesn't ensure old child job is killed at launcher start -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[
https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343049#comment-14343049
]
Hadoop QA commented on OOZIE-2129:
--
Testing JIRA OOZIE-2129
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:red}-1{color} the patch contains 1 line(s) with trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1587
.Tests failed: 1
.Tests errors: 0
.The patch failed the following testcases:
.
testCoordRerunDateNeg(org.apache.oozie.command.coord.TestCoordRerunXCommand)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/2235/
Duplicate child jobs per instance
-
Key: OOZIE-2129
URL: https://issues.apache.org/jira/browse/OOZIE-2129
Project: Oozie
Issue Type: Bug
Reporter: Shwetha G S
Assignee: Jaydeep Vishwakarma
Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch,
OOZIE-2129-v2.patch, OOZIE-2129-v3.patch
OOZIE-1722 ensures that child job is killed at launcher restart. But this
doesn't work for java actions as the tag is not passed to the child job.
In case of coord action rerun, new workflow is created and hence new tag. So,
it doesn't ensure old child job is killed at launcher start
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[ https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341054#comment-14341054 ] Robert Kanter commented on OOZIE-2129: -- I double checked with the dev who did a lot of the yarn tags work, and he said It is better to give the time range at the very least, that saves us transfering all data. And, the intent is to improve YARN's filtering in the future. So we should leave that part in. Duplicate child jobs per instance - Key: OOZIE-2129 URL: https://issues.apache.org/jira/browse/OOZIE-2129 Project: Oozie Issue Type: Bug Reporter: Shwetha G S Assignee: Jaydeep Vishwakarma Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch, OOZIE-2129-v2.patch OOZIE-1722 ensures that child job is killed at launcher restart. But this doesn't work for java actions as the tag is not passed to the child job. In case of coord action rerun, new workflow is created and hence new tag. So, it doesn't ensure old child job is killed at launcher start -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[
https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341030#comment-14341030
]
Robert Kanter commented on OOZIE-2129:
--
Sorry for the late response. By the way, there’s a “preview” button so you can
verify your formatting before posting.
# Even though they’re not indexed, it should still be faster because times are
in order. Consider if there’s 1 million jobs going back 3 years. Our job only
started yesterday, so there’s no reason to look through all 1 million jobs;
only the smaller number that occurred today and yesterday.
# I know that many users run MR jobs from the Java action, but in general,
we’ve avoided adding any MR specific extras to that action; they belong in the
MR action. The Java action is designed for arbitrary Java tasks. For example,
we don’t load the Configuration or handle the delegation tokens for MR jobs in
the Java action. Like those, if the user wants to take advantage of this
feature in their Java action, they have to implement some code. In fact, we
already have {{LauncherMainHadoopUtils.killChildYarnJobs(actionConf);}} in
JavaMain, so if they launched their MR job using the prepared action conf, that
should work automatically.
# {quote}Even now also LauncherMapper does not pass tag to child jobs{quote}
Isn’t that exactly what the original OOZIE-2129 did? It kills launched jobs
via tags when starting the launcher job.
# thanks
# I’m not sure I understand what you mean here. I don’t see why we’re adding a
new file to pass info to launcher jobs, when we already have the action conf
file.
Backing up for a moment, why does this matter to coord action rerun? You
shouldn’t be able to rerun a coord action if the original coord action, and
therefore workflow, and therefore any launcher jobs in that workflow, are still
running.
Duplicate child jobs per instance
-
Key: OOZIE-2129
URL: https://issues.apache.org/jira/browse/OOZIE-2129
Project: Oozie
Issue Type: Bug
Reporter: Shwetha G S
Assignee: Jaydeep Vishwakarma
Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch,
OOZIE-2129-v2.patch
OOZIE-1722 ensures that child job is killed at launcher restart. But this
doesn't work for java actions as the tag is not passed to the child job.
In case of coord action rerun, new workflow is created and hence new tag. So,
it doesn't ensure old child job is killed at launcher start
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[ https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318762#comment-14318762 ] Jaydeep Vishwakarma commented on OOZIE-2129: I am still unable to upload the code in review board, Am I the only person who facing this problem? Anyone knows who manages review board? Duplicate child jobs per instance - Key: OOZIE-2129 URL: https://issues.apache.org/jira/browse/OOZIE-2129 Project: Oozie Issue Type: Bug Reporter: Shwetha G S Assignee: Jaydeep Vishwakarma Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch OOZIE-1722 ensures that child job is killed at launcher restart. But this doesn't work for java actions as the tag is not passed to the child job. In case of coord action rerun, new workflow is created and hence new tag. So, it doesn't ensure old child job is killed at launcher start -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[
https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319410#comment-14319410
]
Robert Kanter commented on OOZIE-2129:
--
RB might require the {{--full-index}} option when doing {{git diff}}. Our
internal RB does.
I'm not surprised the hadoop-3 profile isn't working; I don't think anyone's
tried running it for a while. Copy-pasting the same code has hadoop-2 is fine
for now.
I took a quick look; a few comments:
# Why do we get rid of {{gar.setStartRange(startTime,
System.currentTimeMillis());}}? IIRC, I put this in to help lower the cost on
YARN's side of looking up apps by tags. We don't have to look any earlier than
{{startTime}}
# I'm not sure we should add support for this to the Java action. While you
can run MR jobs from it, and many users do, we've typically avoided adding
MR-specific features to it to encourage users to use the MR action. The Java
action's primary purpose is to run Java programs
# I'm not sure if this is a problem, but if the user upgrades to a version of
Oozie with this patch, any actions running during the upgrade won't be able to
properly use the tags, right? Even if that's a problem, it's not likely to
occur.
# In {{LauncherMapper}}, don't replace the {{java.io}} import with {{import
java.io.*}}; we don't typically use {{*}} for imports
# I'm not sure I understand how the {{propagation-conf.xml}} is used; aren't
all of the necessary configs appropriately set in the launcher and action confs
already?
Duplicate child jobs per instance
-
Key: OOZIE-2129
URL: https://issues.apache.org/jira/browse/OOZIE-2129
Project: Oozie
Issue Type: Bug
Reporter: Shwetha G S
Assignee: Jaydeep Vishwakarma
Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch
OOZIE-1722 ensures that child job is killed at launcher restart. But this
doesn't work for java actions as the tag is not passed to the child job.
In case of coord action rerun, new workflow is created and hence new tag. So,
it doesn't ensure old child job is killed at launcher start
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[
https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319653#comment-14319653
]
Jaydeep Vishwakarma commented on OOZIE-2129:
[~rkanter]
My replies are inline
# Why do we get rid of {code}gar.setStartRange(startTime,
System.currentTimeMillis());{code}? IIRC, I put this in to help lower the cost
on YARN's side of looking up apps by tags. We don't have to look any earlier
than startTime.
#* As my understanding yarn do not manage any index, So doing the filter based
on time is just adding another level of complexcity. Correct me if I am wrong.
# I'm not sure we should add support for this to the Java action. While you can
run MR jobs from it, and many users do, we've typically avoided adding
MR-specific features to it to encourage users to use the MR action. The Java
action's primary purpose is to run Java programs.
#* I feel we cannot go with this assumption. I am observed many user including
me using java action for running map-reduce job. Find easy to have controls.
# I'm not sure if this is a problem, but if the user upgrades to a version of
Oozie with this patch, any actions running during the upgrade won't be able to
properly use the tags, right? Even if that's a problem, it's not likely to
occur.
#* Even now also {code}LauncherMapper{code} does not pass tag to child jobs, It
is basically a fix which will resolve the issue for all the jobs which will be
launch after applying this patch.
# In {code}LauncherMapper{code}, don't replace the {code}java.io{code} import
with {code}import java.io.;{code} we don't typically use {} for imports
#* I have checked my default settings of idea, It is brining this imports. I
will fix it.
#I'm not sure I understand how the {code}propagation-conf.xml{code} is used;
aren't all of the necessary configs appropriately set in the launcher and
action confs already?
#* I am using {code}propagation-conf.xml{code} for java action and for others
{code}mapreduce.job.tags{code} will be just duplicate value. There is no such
contract available in JavaMain to pass the conf value.
Duplicate child jobs per instance
-
Key: OOZIE-2129
URL: https://issues.apache.org/jira/browse/OOZIE-2129
Project: Oozie
Issue Type: Bug
Reporter: Shwetha G S
Assignee: Jaydeep Vishwakarma
Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch
OOZIE-1722 ensures that child job is killed at launcher restart. But this
doesn't work for java actions as the tag is not passed to the child job.
In case of coord action rerun, new workflow is created and hence new tag. So,
it doesn't ensure old child job is killed at launcher start
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[ https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319659#comment-14319659 ] Jaydeep Vishwakarma commented on OOZIE-2129: [~rkanter] Ops, I have screwed up with the format, Please let me know if you have any doubt. Duplicate child jobs per instance - Key: OOZIE-2129 URL: https://issues.apache.org/jira/browse/OOZIE-2129 Project: Oozie Issue Type: Bug Reporter: Shwetha G S Assignee: Jaydeep Vishwakarma Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch OOZIE-1722 ensures that child job is killed at launcher restart. But this doesn't work for java actions as the tag is not passed to the child job. In case of coord action rerun, new workflow is created and hence new tag. So, it doesn't ensure old child job is killed at launcher start -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[ https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14315878#comment-14315878 ] Jaydeep Vishwakarma commented on OOZIE-2129: Jira for hadoop-3 profile. https://issues.apache.org/jira/browse/OOZIE-2138 Duplicate child jobs per instance - Key: OOZIE-2129 URL: https://issues.apache.org/jira/browse/OOZIE-2129 Project: Oozie Issue Type: Bug Reporter: Shwetha G S Assignee: Jaydeep Vishwakarma Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch OOZIE-1722 ensures that child job is killed at launcher restart. But this doesn't work for java actions as the tag is not passed to the child job. In case of coord action rerun, new workflow is created and hence new tag. So, it doesn't ensure old child job is killed at launcher start -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[
https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316074#comment-14316074
]
Hadoop QA commented on OOZIE-2129:
--
Testing JIRA OOZIE-2129
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:red}-1{color} the patch contains 1 line(s) with trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1577
.Tests failed: 2
.Tests errors: 1
.The patch failed the following testcases:
.
testBundlePauseExtendMaterializesCoordinator(org.apache.oozie.command.bundle.TestBundleChangeXCommand)
. testSparkAction(org.apache.oozie.action.hadoop.TestSparkActionExecutor)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/2217/
Duplicate child jobs per instance
-
Key: OOZIE-2129
URL: https://issues.apache.org/jira/browse/OOZIE-2129
Project: Oozie
Issue Type: Bug
Reporter: Shwetha G S
Assignee: Jaydeep Vishwakarma
Attachments: OOZIE-2129-v0.patch, OOZIE-2129-v1.patch
OOZIE-1722 ensures that child job is killed at launcher restart. But this
doesn't work for java actions as the tag is not passed to the child job.
In case of coord action rerun, new workflow is created and hence new tag. So,
it doesn't ensure old child job is killed at launcher start
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[
https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14313710#comment-14313710
]
Shwetha G S commented on OOZIE-2129:
ActionStartXCommand:
{code}
+} else if (wfJob.getParentId() != null) {
+context.setVar(COORD_ACTION_TAG, wfJob.getParentId() + @
+ wfAction.getName());
+}
{code}
If coord action id is not set and there is no parent, set it to workflow action
id. I know that it defaults to workflow action id in Java Action Executor, but
it won't work for non-java derived actions.
The changes for launcher are just in hadoop-2
(hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java).
We need to replicate the same for hadoop-3 as well. Instead, can we make only
one module for hadoop-utils which can be shared for both hadoop-2 and hadoop-3.
You can set oozie.hadoop-utils.version in main pom.xml for hadoop-2 and
hadoop-3 profiles to be the same
LauncherMainHadoopUtils.getChildYarnJobs():
String tag = actionConf.get(mapreduce.job.tags);
should read from child.mapreduce.job.tags
+System.out.println(childYarnJobs.size() + Child yarn jobs are found);
Print all the child job ids?
Hive2Main:
I don't think we are passing the action conf to child jobs, so job tags might
not work
LauncherMapper.propagateConf():
Use loadActionConf()
Duplicate child jobs per instance
-
Key: OOZIE-2129
URL: https://issues.apache.org/jira/browse/OOZIE-2129
Project: Oozie
Issue Type: Bug
Reporter: Shwetha G S
Assignee: Jaydeep Vishwakarma
Attachments: OOZIE-2129-v0.patch
OOZIE-1722 ensures that child job is killed at launcher restart. But this
doesn't work for java actions as the tag is not passed to the child job.
In case of coord action rerun, new workflow is created and hence new tag. So,
it doesn't ensure old child job is killed at launcher start
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[ https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14313732#comment-14313732 ] Shwetha G S commented on OOZIE-2129: [~rkanter], extending what you had done in OOZIE-1722, this jira changes the tag to contain coord action id so that it can handle cases where there are multiple workflows per coord action. Also, fixes the tags propagation to child jobs. Can you take a look once please? Thanks Duplicate child jobs per instance - Key: OOZIE-2129 URL: https://issues.apache.org/jira/browse/OOZIE-2129 Project: Oozie Issue Type: Bug Reporter: Shwetha G S Assignee: Jaydeep Vishwakarma Attachments: OOZIE-2129-v0.patch OOZIE-1722 ensures that child job is killed at launcher restart. But this doesn't work for java actions as the tag is not passed to the child job. In case of coord action rerun, new workflow is created and hence new tag. So, it doesn't ensure old child job is killed at launcher start -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[ https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311871#comment-14311871 ] Jaydeep Vishwakarma commented on OOZIE-2129: TestSparkActionExecutor is failing, It is failing without apply this patch as well. Duplicate child jobs per instance - Key: OOZIE-2129 URL: https://issues.apache.org/jira/browse/OOZIE-2129 Project: Oozie Issue Type: Bug Reporter: Shwetha G S Assignee: Jaydeep Vishwakarma Attachments: OOZIE-2129-v0.patch OOZIE-1722 ensures that child job is killed at launcher restart. But this doesn't work for java actions as the tag is not passed to the child job. In case of coord action rerun, new workflow is created and hence new tag. So, it doesn't ensure old child job is killed at launcher start -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[ https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310913#comment-14310913 ] Jaydeep Vishwakarma commented on OOZIE-2129: I have done the test in following scenarios: +Workflow with pig/java action+ - Kill Launcher mapper attempt - Restart node manager on which app-master is running - Kill launcher job - Follow up with workflow retry +Coord job +Direct workflow with pig/java action+ - Kill Launcher mapper attempt - Restart node manager on which app-master is running - Kill launcher job - Follow up with workflow retry - Kill launcher job - Follow up with coord action retry +Coord job +Subflow workflow with pig/java action+ - Kill Launcher mapper attempt - Restart node manager on which app-master is running - Kill launcher job - Follow up with workflow retry - Kill launcher job - Follow up with coord action retry Duplicate child jobs per instance - Key: OOZIE-2129 URL: https://issues.apache.org/jira/browse/OOZIE-2129 Project: Oozie Issue Type: Bug Reporter: Shwetha G S Assignee: Jaydeep Vishwakarma Attachments: OOZIE-2129-v0.patch OOZIE-1722 ensures that child job is killed at launcher restart. But this doesn't work for java actions as the tag is not passed to the child job. In case of coord action rerun, new workflow is created and hence new tag. So, it doesn't ensure old child job is killed at launcher start -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[ https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310916#comment-14310916 ] Jaydeep Vishwakarma commented on OOZIE-2129: Some issue is going on with review board unable to upload patch in review board. I will add it in review board once it will be fine. Duplicate child jobs per instance - Key: OOZIE-2129 URL: https://issues.apache.org/jira/browse/OOZIE-2129 Project: Oozie Issue Type: Bug Reporter: Shwetha G S Assignee: Jaydeep Vishwakarma Attachments: OOZIE-2129-v0.patch OOZIE-1722 ensures that child job is killed at launcher restart. But this doesn't work for java actions as the tag is not passed to the child job. In case of coord action rerun, new workflow is created and hence new tag. So, it doesn't ensure old child job is killed at launcher start -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[
https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310950#comment-14310950
]
Hadoop QA commented on OOZIE-2129:
--
Testing JIRA OOZIE-2129
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:red}-1{color} the patch contains 1 line(s) with trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1574
.Tests failed: 1
.Tests errors: 1
.The patch failed the following testcases:
. testSparkAction(org.apache.oozie.action.hadoop.TestSparkActionExecutor)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/2214/
Duplicate child jobs per instance
-
Key: OOZIE-2129
URL: https://issues.apache.org/jira/browse/OOZIE-2129
Project: Oozie
Issue Type: Bug
Reporter: Shwetha G S
Assignee: Jaydeep Vishwakarma
Attachments: OOZIE-2129-v0.patch
OOZIE-1722 ensures that child job is killed at launcher restart. But this
doesn't work for java actions as the tag is not passed to the child job.
In case of coord action rerun, new workflow is created and hence new tag. So,
it doesn't ensure old child job is killed at launcher start
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2129) Duplicate child jobs per instance
[ https://issues.apache.org/jira/browse/OOZIE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308824#comment-14308824 ] Jaydeep Vishwakarma commented on OOZIE-2129: This is happening with other actions also. Earlier fix does not propagates tags information to child jobs which launch by launcher. Duplicate child jobs per instance - Key: OOZIE-2129 URL: https://issues.apache.org/jira/browse/OOZIE-2129 Project: Oozie Issue Type: Bug Reporter: Shwetha G S Assignee: Jaydeep Vishwakarma OOZIE-1722 ensures that child job is killed at launcher restart. But this doesn't work for java actions as the tag is not passed to the child job. In case of coord action rerun, new workflow is created and hence new tag. So, it doesn't ensure old child job is killed at launcher start -- This message was sent by Atlassian JIRA (v6.3.4#6332)
