[jira] Subscription: Oozie Patch Available
Issue Subscription Filter: Oozie Patch Available (90 issues) Subscriber: ooziedaily Key Summary OOZIE-3486 duplicate code in ControlNodeHandler https://issues.apache.org/jira/browse/OOZIE-3486 OOZIE-3482 Fix bug in CoordSubmitXCommand#validateCoordinatorJob https://issues.apache.org/jira/browse/OOZIE-3482 OOZIE-3480 Add windowactionstatus metrics in DBLiteWorkflowStoreService https://issues.apache.org/jira/browse/OOZIE-3480 OOZIE-3468 Use modernizer plugin https://issues.apache.org/jira/browse/OOZIE-3468 OOZIE-3461 CoordMaterializeTriggerService code cleanup https://issues.apache.org/jira/browse/OOZIE-3461 OOZIE-3449 Make spark-2 as the default profile https://issues.apache.org/jira/browse/OOZIE-3449 OOZIE-3447 Run test case in local : It shows oozie-hsqldb-orm.xml exception https://issues.apache.org/jira/browse/OOZIE-3447 OOZIE-3418 Upgrade to Guava 27 https://issues.apache.org/jira/browse/OOZIE-3418 OOZIE-3404 The env variable of SPARK_HOME needs to be set when running pySpark https://issues.apache.org/jira/browse/OOZIE-3404 OOZIE-3375 Can't use empty in coordinator https://issues.apache.org/jira/browse/OOZIE-3375 OOZIE-3367 Using && in EL expressions in oozie bundle.xml files generates parse errors https://issues.apache.org/jira/browse/OOZIE-3367 OOZIE-3366 Update workflow status and subworkflow status on suspend command https://issues.apache.org/jira/browse/OOZIE-3366 OOZIE-3364 Rerunning Oozie bundle jobs starts the coordinators in indeterminate order https://issues.apache.org/jira/browse/OOZIE-3364 OOZIE-3362 When killed, SSH action should kill the spawned processes on target host https://issues.apache.org/jira/browse/OOZIE-3362 OOZIE-3335 Cleanup parseFilter methods https://issues.apache.org/jira/browse/OOZIE-3335 OOZIE-3328 Create Hive compatibility action executor to run hive actions using beeline https://issues.apache.org/jira/browse/OOZIE-3328 OOZIE-3320 Oozie ShellAction should support absolute bash file path https://issues.apache.org/jira/browse/OOZIE-3320 OOZIE-3319 Log SSH action callback error output https://issues.apache.org/jira/browse/OOZIE-3319 OOZIE-3301 Update NOTICE file https://issues.apache.org/jira/browse/OOZIE-3301 OOZIE-3274 Remove slf4j https://issues.apache.org/jira/browse/OOZIE-3274 OOZIE-3266 Coord action rerun support RERUN_SKIP_NODES option https://issues.apache.org/jira/browse/OOZIE-3266 OOZIE-3256 refactor OozieCLI class https://issues.apache.org/jira/browse/OOZIE-3256 OOZIE-3199 Let system property restriction configurable https://issues.apache.org/jira/browse/OOZIE-3199 OOZIE-3196 Authorization: restrict world readability by user https://issues.apache.org/jira/browse/OOZIE-3196 OOZIE-3179 Adding a configurable config-default.xml location to a workflow https://issues.apache.org/jira/browse/OOZIE-3179 OOZIE-3170 Oozie Diagnostic Bundle tool fails with NPE due to missing service class https://issues.apache.org/jira/browse/OOZIE-3170 OOZIE-3137 Add support for log4j2 in HiveMain https://issues.apache.org/jira/browse/OOZIE-3137 OOZIE-3135 Configure log4j2 in SqoopMain https://issues.apache.org/jira/browse/OOZIE-3135 OOZIE-3091 Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroWrapper" https://issues.apache.org/jira/browse/OOZIE-3091 OOZIE-3071 Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than Spark 2.2.0 https://issues.apache.org/jira/browse/OOZIE-3071 OOZIE-3063 Sanitizing variables that are part of openjpa.ConnectionProperties https://issues.apache.org/jira/browse/OOZIE-3063 OOZIE-3062 Set HADOOP_CONF_DIR for spark action https://issues.apache.org/jira/browse/OOZIE-3062 OOZIE-2952 Fix Findbugs warnings in oozie-sharelib-oozie https://issues.apache.org/jira/browse/OOZIE-2952 OOZIE-2834 ParameterVerifier logging non-useful warning for workflow definition https://issues.apache.org/jira/browse/OOZIE-2834 OOZIE-2812 SparkConfigurationService should support loading configurations from multiple Spark versions https://issues.apache.org/jira/browse/OOZIE-2812 OOZIE-2795 Create lib directory or symlink for Oozie CLI during packaging https://issues.apache.org/jira/browse/OOZIE-2795 OOZIE-2784 Include WEEK as a parameter in the Coordinator Expression Language Evaulator https://issues.apache.org/jira/browse/OOZIE-2784 OOZIE-2779 Mask Hive2 action Beeline JDBC password https://issues.apache.org/jira/browse/OOZIE-2779 OOZIE-2736 Reduce the number of threads during test execution https://issues.apache.or
[jira] [Comment Edited] (OOZIE-3512) Flaky test TestActionStartXCommand.testActionWithEscapedStringAndCDATA
[
https://issues.apache.org/jira/browse/OOZIE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890154#comment-16890154
]
Peter Bacsko edited comment on OOZIE-3512 at 7/22/19 3:37 PM:
--
Usually an application stays in ACCEPTED state if there are not enough
resources (vcores / memory). Another problem is when a node manager becomes
UNHEALTHY - we set up the Mini YARN cluster with a single NM cluster, so if
that happens, we can't run any applications. This happened many times before
due to disk space issues and the disk checker, having detected a low amount of
free space, marked the NM as "unhealthy" so we ended up with 0 NMs. But this
was addressed and the threshold was raised to 99% or 100%. Anyway I'd examine
the RM or NM output from the Mini cluster to see why this wasn't scheduled.
was (Author: pbacsko):
Usually an application stays in ACCEPTED state if there are not enough
resources (vcores / memory). Another problem is when a node manager becomes
UNHEALTHY - we set up the Mini YARN cluster with a single NM cluster, so if
that happens, we can't run any applications. This happened many times before
due to disk space issues and the disk checker, having detected a low amount of
free space, marked the NM as "unhalthy" so we ended up with 0 NMs. But this was
addressed and the threashold was raised to 99% or 100%. Anyway I'd examine the
RM or NM output from the Mini cluster to see why this wasn't scheduled.
> Flaky test TestActionStartXCommand.testActionWithEscapedStringAndCDATA
> --
>
> Key: OOZIE-3512
> URL: https://issues.apache.org/jira/browse/OOZIE-3512
> Project: Oozie
> Issue Type: Sub-task
> Components: tests
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: duan xiong
>Priority: Major
>
> {{TestActionStartXCommand.testActionWithEscapedStringAndCDATA}} is flaky,
> sometimes (for instance:
> https://issues.apache.org/jira/browse/OOZIE-3470?focusedCommentId=16817901&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16817901
> ) it fails with the following error message:
> {noformat}junit.framework.AssertionFailedError: YARN App state for app
> application_1559489642789_0018 expected: but was:
> at junit.framework.Assert.fail(Assert.java:57)
> at junit.framework.Assert.failNotEquals(Assert.java:329)
> at junit.framework.Assert.assertEquals(Assert.java:78)
> at junit.framework.TestCase.assertEquals(TestCase.java:244)
> at
> org.apache.oozie.test.XTestCase.waitUntilYarnAppDoneAndAssertSuccess(XTestCase.java:1358)
> at
> org.apache.oozie.command.wf.TestActionStartXCommand.testActionWithEscapedStringAndCDATA(TestActionStartXCommand.java:235)
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (OOZIE-3529) Oozie not supported for s3 as filesystem
[
https://issues.apache.org/jira/browse/OOZIE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890243#comment-16890243
]
Steve Loughran commented on OOZIE-3529:
---
This error message/stack trace is *not* from hadoop-common. Does Oozie have a
special binding to block access to the local FS?
The S3A connector needs a local directory to buffer data before uploading it
multi-MB blocks; it uses the LocalDirAllocator to take the list of paths in
"fs.s3a.buffer.dir" (falling back to hadoop.tmp.dir), and expects to be given a
path in a local FS to which it can save temp files.
That stack trace implies something in in Oozie is stopping applications get at
that local filesystem, so we can't allocate data
As a short term fix, set {{fs.s3a.fast.upload.buffer}} to {{bytebuffer}}. This
will switch to using off-heap byte buffers for buffering data. Provided Oozie
doesn't write data faster than it can be uploaded to S3 at such a rate that
you run out of memory, this should work
> Oozie not supported for s3 as filesystem
>
>
> Key: OOZIE-3529
> URL: https://issues.apache.org/jira/browse/OOZIE-3529
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.1, 5.1.0
>Reporter: Denes Bodo
>Priority: Critical
> Labels: S3
> Attachments: id.pig, job.properties, workflow.xml
>
>
> Many customer who uses s3 file system as secondary one experiences the
> following error when Oozie tries to submit the Yarn application:
> {noformat}
> 2019-04-29 13:02:53,770 WARN ForkedActionStartXCommand:523 -
> SERVER[hwnode1.puretec.purestorage.com] USER[hrt_qa] GROUP[-] TOKEN[]
> APP[demo-wf] JOB[001-190423141707256-oozie-oozi-W]
> ACTION[001-190423141707256-oozie-oozi-W@streaming-node] Error starting
> action [streaming-node]. ErrorType [ERROR], ErrorCode
> [UnsupportedOperationException], Message [UnsupportedOperationException:
> Accessing local file system is not allowed]
> org.apache.oozie.action.ActionExecutorException:
> UnsupportedOperationException: Accessing local file system is not allowed
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.createLauncherConf(JavaActionExecutor.java:1092)
> at
> org.apache.oozie.action.hadoop.MapReduceActionExecutor.createLauncherConf(MapReduceActionExecutor.java:309)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1197)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1472)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:41)
> at
> org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:30)
> at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.UnsupportedOperationException: Accessing local file
> system is not allowed
> at
> org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:48)
> at
> org.apache.hadoop.fs.LocalFileSystem.initialize(LocalFileSystem.java:47)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
> at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:433)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:301)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:378)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForWrite(LocalDirAllocator.java:461)
> at
> org.apache.hadoop.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocator.java:200)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.createTmpFileForWrite(S3AFileSystem.java:572)
> at
> org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlock
[jira] [Commented] (OOZIE-3528) Migrate to PowerMock 2 and Mockito2
[ https://issues.apache.org/jira/browse/OOZIE-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890233#comment-16890233 ] Andras Salamon commented on OOZIE-3528: --- Thanks for the contribution [~kmarton], +1, committed to master. > Migrate to PowerMock 2 and Mockito2 > --- > > Key: OOZIE-3528 > URL: https://issues.apache.org/jira/browse/OOZIE-3528 > Project: Oozie > Issue Type: Improvement >Reporter: Julia Kinga Marton >Assignee: Julia Kinga Marton >Priority: Major > Attachments: OOZIE-3528-001.patch, OOZIE-3528-002.patch, > OOZIE-3528-003.patch, WIP_OOZIE-3528.patch > > > PowerMock 1 does not support JDK11. If we want to . support it we will need > to migrate to PoweMock 2 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (OOZIE-3528) Migrate to PowerMock 2 and Mockito2
[
https://issues.apache.org/jira/browse/OOZIE-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890218#comment-16890218
]
Hadoop QA commented on OOZIE-3528:
--
Testing JIRA OOZIE-3528
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch adds/modifies 16 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1{color} There are no new bugs found in total.
.{color:green}+1{color} There are no new bugs found in [webapp].
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [sharelib/git].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [tools].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3175
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:green}*+1 Overall result, good!, no -1s*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/1191/
> Migrate to PowerMock 2 and Mockito2
> ---
>
> Key: OOZIE-3528
> URL: https://issues.apache.org/jira/browse/OOZIE-3528
> Project: Oozie
> Issue Type: Improvement
>Reporter: Julia Kinga Marton
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-3528-001.patch, OOZIE-3528-002.patch,
> OOZIE-3528-003.patch, WIP_OOZIE-3528.patch
>
>
> PowerMock 1 does not support JDK11. If we want to . support it we will need
> to migrate to PoweMock 2
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
Failed: OOZIE-3528 PreCommit Build #1191
Jira: https://issues.apache.org/jira/browse/OOZIE-3528
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/1191/
###
## LAST 100 LINES OF THE CONSOLE
###
[...truncated 2.03 MB...]
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/hive].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/spark].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [fluent-job/fluent-job-api].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [core].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [client].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [docs].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [examples].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [tools].
[INFO] There are no new bugs found totally].
[TRACE] SpotBugs diffs checked and reports created
[TRACE] Summary file size is 2536 bytes
[TRACE] Full summary file size is 1525 bytes
[TRACE] File
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/SPOTBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
removed
[TRACE] File
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/SPOTBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5]
removed
Running test-patch task BACKWARDS_COMPATIBILITY
Running test-patch task TESTS
Running test-patch task DISTRO
Testing JIRA OOZIE-3528
Cleaning local git workspace
+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any star imports
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 16 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 Javadoc generation succeeded with the patch
+1 the patch does not seem to introduce new Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
+1 There are no new bugs found in [webapp].
+1 There are no new bugs found in [server].
+1 There are no new bugs found in [sharelib/git].
+1 There are no new bugs found in [sharelib/hive2].
+1 There are no new bugs found in [sharelib/pig].
+1 There are no new bugs found in [sharelib/oozie].
+1 There are no new bugs found in [sharelib/streaming].
+1 There are no new bugs found in [sharelib/sqoop].
+1 There are no new bugs found in [sharelib/hcatalog].
+1 There are no new bugs found in [sharelib/distcp].
+1 There are no new bugs found in [sharelib/hive].
+1 There are no new bugs found in [sharelib/spark].
+1 There are no new bugs found in [fluent-job/fluent-job-api].
+1 There are no new bugs found in [core].
+1 There are no new bugs found in [client].
+1 There are no new bugs found in [docs].
+1 There are no new bugs found in [examples].
+1 There are no new bugs found in [tools].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 3175
+1 DISTRO
+1 distro tarball builds with the patch
+1 Overall result, good!, no -1s
The full output of the test-patch run is available at
https://builds.apache.org/job/PreCommit-OOZIE-Build/1191/
Adding comment to JIRA
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0100
31750 0 100 3175 0 2285 0:00:01 0:00:01 --:--:--
2284{"self":"https://issues.apache.org/jira/rest/api/2/issue/13244598/comment/16890218","id":"16890218","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa&avatarId=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small&ownerId=hadoopqa&a
[jira] [Commented] (OOZIE-2566) TestCoordActionInputCheckXCommand.testCoordActionInputCheckXCommandUniqueness() is flaky
[
https://issues.apache.org/jira/browse/OOZIE-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890156#comment-16890156
]
Peter Bacsko commented on OOZIE-2566:
-
[~asalamon74] I don't have a definite answer to these questions right now. It
could be that there's no added value, but we have to double-check this. If it's
unnecessary, then we can just remove it. Let's examine this together next week.
> TestCoordActionInputCheckXCommand.testCoordActionInputCheckXCommandUniqueness()
> is flaky
>
>
> Key: OOZIE-2566
> URL: https://issues.apache.org/jira/browse/OOZIE-2566
> Project: Oozie
> Issue Type: Sub-task
> Components: core
>Reporter: Peter Bacsko
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-2566-01.patch
>
>
> The testcase testCoordActionInputCheckXCommandUniqueness is unstable.
> We add three XCommands with the same actionId (entityKeys are different) into
> the CallableQueueService. Only the first XCommand is expected to run.
> The reason why sometimes either the 2nd or 3rd XCommand executes is because
> as soon as the first starts to run, its removed from the {{uniqueCallables}}
> map immediately. If the first scheduled task runs quickly, then either the
> 2nd or 3rd XCommand has the chance to get scheduled.
> Step by step:
> 1. Schedule first XCommand
> 2. XCommand is added to {{uniqueCallables}}
> 3. Schedule second XCommand
> 4. First XCommand starts to run in the thread pool and removes itself from
> {{uniqueCallables}} (see {{CallableWrapper.run()}})
> 5. Second XCommand can successfully add itself to {{uniqueCallables}}
> 6. Second XCommand starts to run
> Please clarify whether this is the expected behavior of CallableQueueService.
> If not, then moving {{removeFromUniqueCallables()}} to the finally block
> solves the problem.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (OOZIE-3512) Flaky test TestActionStartXCommand.testActionWithEscapedStringAndCDATA
[
https://issues.apache.org/jira/browse/OOZIE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890154#comment-16890154
]
Peter Bacsko commented on OOZIE-3512:
-
Usually an application stays in ACCEPTED state if there are not enough
resources (vcores / memory). Another problem is when a node manager becomes
UNHEALTHY - we set up the Mini YARN cluster with a single NM cluster, so if
that happens, we can't run any applications. This happened many times before
due to disk space issues and the disk checker, having detected a low amount of
free space, marked the NM as "unhalthy" so we ended up with 0 NMs. But this was
addressed and the threashold was raised to 99% or 100%. Anyway I'd examine the
RM or NM output from the Mini cluster to see why this wasn't scheduled.
> Flaky test TestActionStartXCommand.testActionWithEscapedStringAndCDATA
> --
>
> Key: OOZIE-3512
> URL: https://issues.apache.org/jira/browse/OOZIE-3512
> Project: Oozie
> Issue Type: Sub-task
> Components: tests
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: duan xiong
>Priority: Major
>
> {{TestActionStartXCommand.testActionWithEscapedStringAndCDATA}} is flaky,
> sometimes (for instance:
> https://issues.apache.org/jira/browse/OOZIE-3470?focusedCommentId=16817901&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16817901
> ) it fails with the following error message:
> {noformat}junit.framework.AssertionFailedError: YARN App state for app
> application_1559489642789_0018 expected: but was:
> at junit.framework.Assert.fail(Assert.java:57)
> at junit.framework.Assert.failNotEquals(Assert.java:329)
> at junit.framework.Assert.assertEquals(Assert.java:78)
> at junit.framework.TestCase.assertEquals(TestCase.java:244)
> at
> org.apache.oozie.test.XTestCase.waitUntilYarnAppDoneAndAssertSuccess(XTestCase.java:1358)
> at
> org.apache.oozie.command.wf.TestActionStartXCommand.testActionWithEscapedStringAndCDATA(TestActionStartXCommand.java:235)
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (OOZIE-3528) Migrate to PowerMock 2 and Mockito2
[ https://issues.apache.org/jira/browse/OOZIE-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890127#comment-16890127 ] Hadoop QA commented on OOZIE-3528: -- PreCommit-OOZIE-Build started > Migrate to PowerMock 2 and Mockito2 > --- > > Key: OOZIE-3528 > URL: https://issues.apache.org/jira/browse/OOZIE-3528 > Project: Oozie > Issue Type: Improvement >Reporter: Julia Kinga Marton >Assignee: Julia Kinga Marton >Priority: Major > Attachments: OOZIE-3528-001.patch, OOZIE-3528-002.patch, > OOZIE-3528-003.patch, WIP_OOZIE-3528.patch > > > PowerMock 1 does not support JDK11. If we want to . support it we will need > to migrate to PoweMock 2 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (OOZIE-2566) TestCoordActionInputCheckXCommand.testCoordActionInputCheckXCommandUniqueness() is flaky
[
https://issues.apache.org/jira/browse/OOZIE-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890122#comment-16890122
]
Andras Salamon commented on OOZIE-2566:
---
Thanks for checking the patch [~pbacsko].
To be honest I'm not sure about the whole purpose of this specific test and I
collected to many questions, so I uploaded a simple fix.
* It seems to me there are two similar tests:
{{TestCoordActionInputCheckXCommand.testCoordActionInputCheckXCommandUniqueness}}
and {{TestCoordKillXCommand.testCoordKillXCommandUniqueness}}. Why do we test
these two methods specifically? During the tests we replace the {{execute()}}
method with a simple sleep, so it's more like a {{CallableQueue}} test. We have
several {{TestCallableQueue.testQueueUniqueness*}} tests, what is the added
value here?
* If we only want to test the uniqueness then we could simplify the tests. We
don't really need to execute the tests, we could just queue them and check the
{{getUniqueDump() }} but again why is it better than the {{TestCallableQueue}}
tests?
> TestCoordActionInputCheckXCommand.testCoordActionInputCheckXCommandUniqueness()
> is flaky
>
>
> Key: OOZIE-2566
> URL: https://issues.apache.org/jira/browse/OOZIE-2566
> Project: Oozie
> Issue Type: Sub-task
> Components: core
>Reporter: Peter Bacsko
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-2566-01.patch
>
>
> The testcase testCoordActionInputCheckXCommandUniqueness is unstable.
> We add three XCommands with the same actionId (entityKeys are different) into
> the CallableQueueService. Only the first XCommand is expected to run.
> The reason why sometimes either the 2nd or 3rd XCommand executes is because
> as soon as the first starts to run, its removed from the {{uniqueCallables}}
> map immediately. If the first scheduled task runs quickly, then either the
> 2nd or 3rd XCommand has the chance to get scheduled.
> Step by step:
> 1. Schedule first XCommand
> 2. XCommand is added to {{uniqueCallables}}
> 3. Schedule second XCommand
> 4. First XCommand starts to run in the thread pool and removes itself from
> {{uniqueCallables}} (see {{CallableWrapper.run()}})
> 5. Second XCommand can successfully add itself to {{uniqueCallables}}
> 6. Second XCommand starts to run
> Please clarify whether this is the expected behavior of CallableQueueService.
> If not, then moving {{removeFromUniqueCallables()}} to the finally block
> solves the problem.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (OOZIE-3528) Migrate to PowerMock 2 and Mockito2
[ https://issues.apache.org/jira/browse/OOZIE-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890119#comment-16890119 ] Julia Kinga Marton commented on OOZIE-3528: --- testActionKillCommandDate:org.apache.oozie.command.coord.TestCoordActionsKillXCommand passed locally, so I am retriggering the precommit build > Migrate to PowerMock 2 and Mockito2 > --- > > Key: OOZIE-3528 > URL: https://issues.apache.org/jira/browse/OOZIE-3528 > Project: Oozie > Issue Type: Improvement >Reporter: Julia Kinga Marton >Assignee: Julia Kinga Marton >Priority: Major > Attachments: OOZIE-3528-001.patch, OOZIE-3528-002.patch, > OOZIE-3528-003.patch, WIP_OOZIE-3528.patch > > > PowerMock 1 does not support JDK11. If we want to . support it we will need > to migrate to PoweMock 2 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (OOZIE-2566) TestCoordActionInputCheckXCommand.testCoordActionInputCheckXCommandUniqueness() is flaky
[
https://issues.apache.org/jira/browse/OOZIE-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890117#comment-16890117
]
Hadoop QA commented on OOZIE-2566:
--
Testing JIRA OOZIE-2566
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1{color} There are no new bugs found in total.
.{color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/git].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [webapp].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3175
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:green}*+1 Overall result, good!, no -1s*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/1190/
> TestCoordActionInputCheckXCommand.testCoordActionInputCheckXCommandUniqueness()
> is flaky
>
>
> Key: OOZIE-2566
> URL: https://issues.apache.org/jira/browse/OOZIE-2566
> Project: Oozie
> Issue Type: Sub-task
> Components: core
>Reporter: Peter Bacsko
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-2566-01.patch
>
>
> The testcase testCoordActionInputCheckXCommandUniqueness is unstable.
> We add three XCommands with the same actionId (entityKeys are different) into
> the CallableQueueService. Only the first XCommand is expected to run.
> The reason why sometimes either the 2nd or 3rd XCommand executes is because
> as soon as the first starts to run, its removed from the {{uniqueCallables}}
> map immediately. If the first scheduled task runs quickly, then either the
> 2nd or 3rd XCommand has the chance to get scheduled.
> Step by step:
> 1. Schedule first XCommand
> 2. XCommand is added to {{uniqueCallables}}
> 3. Schedule second XCommand
> 4. First XCommand starts to run in the thread pool and removes itself from
> {{uniqueCallables}} (see {{CallableWrapper.run()}})
> 5. Second XCommand can successfully add itself to {{uniqueCallables}}
> 6. Second XCommand starts to run
> Please clarify whether this is the expected behavior of CallableQueueService.
> If not, then moving {{removeFromUniqueCallables()}} to the finally block
> solves the problem.
--
This message was sent by Atlassian JIRA
(v7.6.14#
Failed: OOZIE-2566 PreCommit Build #1190
Jira: https://issues.apache.org/jira/browse/OOZIE-2566
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/1190/
###
## LAST 100 LINES OF THE CONSOLE
###
[...truncated 2.03 MB...]
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/distcp].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/hive2].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [server].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [examples].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [core].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [client].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [docs].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [webapp].
[INFO] There are no new bugs found totally].
[TRACE] SpotBugs diffs checked and reports created
[TRACE] Summary file size is 2535 bytes
[TRACE] Full summary file size is 1525 bytes
[TRACE] File
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/SPOTBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
removed
[TRACE] File
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/SPOTBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5]
removed
Running test-patch task BACKWARDS_COMPATIBILITY
Running test-patch task TESTS
Running test-patch task DISTRO
Testing JIRA OOZIE-2566
Cleaning local git workspace
+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any star imports
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 1 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 Javadoc generation succeeded with the patch
+1 the patch does not seem to introduce new Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
+1 There are no new bugs found in [fluent-job/fluent-job-api].
+1 There are no new bugs found in [tools].
+1 There are no new bugs found in [sharelib/oozie].
+1 There are no new bugs found in [sharelib/streaming].
+1 There are no new bugs found in [sharelib/spark].
+1 There are no new bugs found in [sharelib/pig].
+1 There are no new bugs found in [sharelib/sqoop].
+1 There are no new bugs found in [sharelib/git].
+1 There are no new bugs found in [sharelib/hive].
+1 There are no new bugs found in [sharelib/hcatalog].
+1 There are no new bugs found in [sharelib/distcp].
+1 There are no new bugs found in [sharelib/hive2].
+1 There are no new bugs found in [server].
+1 There are no new bugs found in [examples].
+1 There are no new bugs found in [core].
+1 There are no new bugs found in [client].
+1 There are no new bugs found in [docs].
+1 There are no new bugs found in [webapp].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 3175
+1 DISTRO
+1 distro tarball builds with the patch
+1 Overall result, good!, no -1s
The full output of the test-patch run is available at
https://builds.apache.org/job/PreCommit-OOZIE-Build/1190/
Adding comment to JIRA
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
0 00 00 0 0 0 --:--:-- --:--:-- --:--:--
0{"self":"https://issues.apache.org/jira/rest/api/2/issue/12977618/comment/16890117","id":"16890117","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa&avatarId=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small&ownerId=hadoopqa&avatarId=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall&ownerId=hadoopqa&avatarId=10393","32x32":"https://issues.apache.org/jira/secure/useravatar
[jira] [Commented] (OOZIE-3528) Migrate to PowerMock 2 and Mockito2
[
https://issues.apache.org/jira/browse/OOZIE-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890105#comment-16890105
]
Hadoop QA commented on OOZIE-3528:
--
Testing JIRA OOZIE-3528
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch adds/modifies 16 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:orange}0{color} There are [4] new bugs found in total that would be nice
to have fixed.
.{color:green}+1{color} There are no new bugs found in [client].
.{color:orange}0{color} There are [4] new bugs found in [server] that would
be nice to have fixed.
.You can find the SpotBugs diff here: server/findbugs-new.html
.{color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
.{color:green}+1{color} There are no new bugs found in [webapp].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/git].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2975
.Tests failed : 1
.Tests in error : 0
.Tests timed out : 0
{color:red}-1{color} [ERROR] There are [1] test failures in [core]. Listing
only the first [5] ones
testActionKillCommandDate:org.apache.oozie.command.coord.TestCoordActionsKillXCommand
Check console output for the full list of errors/failures
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/1189/
> Migrate to PowerMock 2 and Mockito2
> ---
>
> Key: OOZIE-3528
> URL: https://issues.apache.org/jira/browse/OOZIE-3528
> Project: Oozie
> Issue Type: Improvement
>Reporter: Julia Kinga Marton
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-3528-001.patch, OOZIE-3528-002.patch,
> OOZIE-3528-003.patch, WIP_OOZIE-3528.patch
>
>
> PowerMock 1 does not support JDK11. If we want to . support it we will need
> to migrate to PoweMock 2
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
Failed: OOZIE-3528 PreCommit Build #1189
Jira: https://issues.apache.org/jira/browse/OOZIE-3528
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/1189/
###
## LAST 100 LINES OF THE CONSOLE
###
[...truncated 2.01 MB...]
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/hive2].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/git].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/streaming].
[WARN] There are [4] new bugs found in total that would be nice to have fixed.
[TRACE] SpotBugs diffs checked and reports created
[TRACE] Summary file size is 2671 bytes
[TRACE] Full summary file size is 1660 bytes
[TRACE] File
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/SPOTBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
removed
[TRACE] File
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/SPOTBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5]
removed
Running test-patch task BACKWARDS_COMPATIBILITY
Running test-patch task TESTS
xargs: WARNING: a NUL character occurred in the input. It cannot be passed
through in the argument list. Did you mean to use the --null option?
Running test-patch task DISTRO
Testing JIRA OOZIE-3528
Cleaning local git workspace
+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any star imports
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 16 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 Javadoc generation succeeded with the patch
+1 the patch does not seem to introduce new Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
0 There are [4] new bugs found in total that would be nice to have fixed.
+1 There are no new bugs found in [client].
0 There are [4] new bugs found in [server] that would be nice to have fixed.
You can find the SpotBugs diff here: server/findbugs-new.html
+1 There are no new bugs found in [fluent-job/fluent-job-api].
+1 There are no new bugs found in [webapp].
+1 There are no new bugs found in [examples].
+1 There are no new bugs found in [core].
+1 There are no new bugs found in [docs].
+1 There are no new bugs found in [tools].
+1 There are no new bugs found in [sharelib/distcp].
+1 There are no new bugs found in [sharelib/oozie].
+1 There are no new bugs found in [sharelib/hive].
+1 There are no new bugs found in [sharelib/spark].
+1 There are no new bugs found in [sharelib/pig].
+1 There are no new bugs found in [sharelib/hcatalog].
+1 There are no new bugs found in [sharelib/sqoop].
+1 There are no new bugs found in [sharelib/hive2].
+1 There are no new bugs found in [sharelib/git].
+1 There are no new bugs found in [sharelib/streaming].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient
annotations
+1 the patch does not modify JPA files
-1 TESTS
Tests run: 2975
Tests failed : 1
Tests in error : 0
Tests timed out : 0
-1 [ERROR] There are [1] test failures in [core]. Listing only the first [5]
ones
testActionKillCommandDate:org.apache.oozie.command.coord.TestCoordActionsKillXCommand
Check console output for the full list of errors/failures
+1 DISTRO
+1 distro tarball builds with the patch
-1 Overall result, please check the reported -1(s)
The full output of the test-patch run is available at
https://builds.apache.org/job/PreCommit-OOZIE-Build/1189/
Adding comment to JIRA
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 36650 00 0 0 0 --:--:-- --:--:-- --:--:--
0{"self":"https://issues.apache.org/jira/rest/api/2/issue/13244598/comment/16890105","id":"16890105","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa&avatarId=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small&ownerId=hadoopqa&avatarId=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall&ownerI
[jira] [Commented] (OOZIE-2566) TestCoordActionInputCheckXCommand.testCoordActionInputCheckXCommandUniqueness() is flaky
[
https://issues.apache.org/jira/browse/OOZIE-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890073#comment-16890073
]
Peter Bacsko commented on OOZIE-2566:
-
[~asalamon74] can't you replace this 200ms delay with some more realiable
wait/notify logic? In the past, these kind of static delays caused a lot of
headaches. I know here it solves the problem, but if there's a better way, we
better try that.
> TestCoordActionInputCheckXCommand.testCoordActionInputCheckXCommandUniqueness()
> is flaky
>
>
> Key: OOZIE-2566
> URL: https://issues.apache.org/jira/browse/OOZIE-2566
> Project: Oozie
> Issue Type: Sub-task
> Components: core
>Reporter: Peter Bacsko
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-2566-01.patch
>
>
> The testcase testCoordActionInputCheckXCommandUniqueness is unstable.
> We add three XCommands with the same actionId (entityKeys are different) into
> the CallableQueueService. Only the first XCommand is expected to run.
> The reason why sometimes either the 2nd or 3rd XCommand executes is because
> as soon as the first starts to run, its removed from the {{uniqueCallables}}
> map immediately. If the first scheduled task runs quickly, then either the
> 2nd or 3rd XCommand has the chance to get scheduled.
> Step by step:
> 1. Schedule first XCommand
> 2. XCommand is added to {{uniqueCallables}}
> 3. Schedule second XCommand
> 4. First XCommand starts to run in the thread pool and removes itself from
> {{uniqueCallables}} (see {{CallableWrapper.run()}})
> 5. Second XCommand can successfully add itself to {{uniqueCallables}}
> 6. Second XCommand starts to run
> Please clarify whether this is the expected behavior of CallableQueueService.
> If not, then moving {{removeFromUniqueCallables()}} to the finally block
> solves the problem.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (OOZIE-2566) TestCoordActionInputCheckXCommand.testCoordActionInputCheckXCommandUniqueness() is flaky
[
https://issues.apache.org/jira/browse/OOZIE-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890064#comment-16890064
]
Hadoop QA commented on OOZIE-2566:
--
PreCommit-OOZIE-Build started
> TestCoordActionInputCheckXCommand.testCoordActionInputCheckXCommandUniqueness()
> is flaky
>
>
> Key: OOZIE-2566
> URL: https://issues.apache.org/jira/browse/OOZIE-2566
> Project: Oozie
> Issue Type: Sub-task
> Components: core
>Reporter: Peter Bacsko
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-2566-01.patch
>
>
> The testcase testCoordActionInputCheckXCommandUniqueness is unstable.
> We add three XCommands with the same actionId (entityKeys are different) into
> the CallableQueueService. Only the first XCommand is expected to run.
> The reason why sometimes either the 2nd or 3rd XCommand executes is because
> as soon as the first starts to run, its removed from the {{uniqueCallables}}
> map immediately. If the first scheduled task runs quickly, then either the
> 2nd or 3rd XCommand has the chance to get scheduled.
> Step by step:
> 1. Schedule first XCommand
> 2. XCommand is added to {{uniqueCallables}}
> 3. Schedule second XCommand
> 4. First XCommand starts to run in the thread pool and removes itself from
> {{uniqueCallables}} (see {{CallableWrapper.run()}})
> 5. Second XCommand can successfully add itself to {{uniqueCallables}}
> 6. Second XCommand starts to run
> Please clarify whether this is the expected behavior of CallableQueueService.
> If not, then moving {{removeFromUniqueCallables()}} to the finally block
> solves the problem.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (OOZIE-3528) Migrate to PowerMock 2 and Mockito2
[ https://issues.apache.org/jira/browse/OOZIE-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890061#comment-16890061 ] Hadoop QA commented on OOZIE-3528: -- PreCommit-OOZIE-Build started > Migrate to PowerMock 2 and Mockito2 > --- > > Key: OOZIE-3528 > URL: https://issues.apache.org/jira/browse/OOZIE-3528 > Project: Oozie > Issue Type: Improvement >Reporter: Julia Kinga Marton >Assignee: Julia Kinga Marton >Priority: Major > Attachments: OOZIE-3528-001.patch, OOZIE-3528-002.patch, > OOZIE-3528-003.patch, WIP_OOZIE-3528.patch > > > PowerMock 1 does not support JDK11. If we want to . support it we will need > to migrate to PoweMock 2 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (OOZIE-2566) TestCoordActionInputCheckXCommand.testCoordActionInputCheckXCommandUniqueness() is flaky
[
https://issues.apache.org/jira/browse/OOZIE-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890027#comment-16890027
]
Andras Salamon commented on OOZIE-2566:
---
I've encountered the same problem, hope you don't mind [~pbacsko] I'm taking
this over.
> TestCoordActionInputCheckXCommand.testCoordActionInputCheckXCommandUniqueness()
> is flaky
>
>
> Key: OOZIE-2566
> URL: https://issues.apache.org/jira/browse/OOZIE-2566
> Project: Oozie
> Issue Type: Sub-task
> Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> The testcase testCoordActionInputCheckXCommandUniqueness is unstable.
> We add three XCommands with the same actionId (entityKeys are different) into
> the CallableQueueService. Only the first XCommand is expected to run.
> The reason why sometimes either the 2nd or 3rd XCommand executes is because
> as soon as the first starts to run, its removed from the {{uniqueCallables}}
> map immediately. If the first scheduled task runs quickly, then either the
> 2nd or 3rd XCommand has the chance to get scheduled.
> Step by step:
> 1. Schedule first XCommand
> 2. XCommand is added to {{uniqueCallables}}
> 3. Schedule second XCommand
> 4. First XCommand starts to run in the thread pool and removes itself from
> {{uniqueCallables}} (see {{CallableWrapper.run()}})
> 5. Second XCommand can successfully add itself to {{uniqueCallables}}
> 6. Second XCommand starts to run
> Please clarify whether this is the expected behavior of CallableQueueService.
> If not, then moving {{removeFromUniqueCallables()}} to the finally block
> solves the problem.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Assigned] (OOZIE-2566) TestCoordActionInputCheckXCommand.testCoordActionInputCheckXCommandUniqueness() is flaky
[
https://issues.apache.org/jira/browse/OOZIE-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Salamon reassigned OOZIE-2566:
-
Assignee: Andras Salamon (was: Peter Bacsko)
> TestCoordActionInputCheckXCommand.testCoordActionInputCheckXCommandUniqueness()
> is flaky
>
>
> Key: OOZIE-2566
> URL: https://issues.apache.org/jira/browse/OOZIE-2566
> Project: Oozie
> Issue Type: Sub-task
> Components: core
>Reporter: Peter Bacsko
>Assignee: Andras Salamon
>Priority: Major
>
> The testcase testCoordActionInputCheckXCommandUniqueness is unstable.
> We add three XCommands with the same actionId (entityKeys are different) into
> the CallableQueueService. Only the first XCommand is expected to run.
> The reason why sometimes either the 2nd or 3rd XCommand executes is because
> as soon as the first starts to run, its removed from the {{uniqueCallables}}
> map immediately. If the first scheduled task runs quickly, then either the
> 2nd or 3rd XCommand has the chance to get scheduled.
> Step by step:
> 1. Schedule first XCommand
> 2. XCommand is added to {{uniqueCallables}}
> 3. Schedule second XCommand
> 4. First XCommand starts to run in the thread pool and removes itself from
> {{uniqueCallables}} (see {{CallableWrapper.run()}})
> 5. Second XCommand can successfully add itself to {{uniqueCallables}}
> 6. Second XCommand starts to run
> Please clarify whether this is the expected behavior of CallableQueueService.
> If not, then moving {{removeFromUniqueCallables()}} to the finally block
> solves the problem.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Comment Edited] (OOZIE-3528) Migrate to PowerMock 2 and Mockito2
[ https://issues.apache.org/jira/browse/OOZIE-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890016#comment-16890016 ] Julia Kinga Marton edited comment on OOZIE-3528 at 7/22/19 9:16 AM: [~asalamon74] below you can find my answers to your questions: * I will check with JDK11 during OOZIE-3504. In this fix my purpose was to do the upgrade, and to make everything with the new versions using JDK8 * You are right, we should use the newer version from branch 2. I have run some tests manually with 2.28.2, they passed, let's see what the precommit will show. * variable creation: done was (Author: kmarton): [~asalamon74] below you can find my answers to your questions: * I will check with JDK11 during OOZIE-3504. In this fix my purpose was to do the upgrade, and to make everything with the new versions using JDK8 * You are right, we should use the newer version from line2. I have run some tests manually with 2.28.2, they passed, let's see what the precommit will show. * variable creation: done > Migrate to PowerMock 2 and Mockito2 > --- > > Key: OOZIE-3528 > URL: https://issues.apache.org/jira/browse/OOZIE-3528 > Project: Oozie > Issue Type: Improvement >Reporter: Julia Kinga Marton >Assignee: Julia Kinga Marton >Priority: Major > Attachments: OOZIE-3528-001.patch, OOZIE-3528-002.patch, > OOZIE-3528-003.patch, WIP_OOZIE-3528.patch > > > PowerMock 1 does not support JDK11. If we want to . support it we will need > to migrate to PoweMock 2 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (OOZIE-3528) Migrate to PowerMock 2 and Mockito2
[ https://issues.apache.org/jira/browse/OOZIE-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890016#comment-16890016 ] Julia Kinga Marton commented on OOZIE-3528: --- [~asalamon74] below you can find my answers to your questions: * I will check with JDK11 during OOZIE-3504. In this fix my purpose was to do the upgrade, and to make everything with the new versions using JDK8 * You are right, we should use the newer version from line2. I have run some tests manually with 2.28.2, they passed, let's see what the precommit will show. * variable creation: done > Migrate to PowerMock 2 and Mockito2 > --- > > Key: OOZIE-3528 > URL: https://issues.apache.org/jira/browse/OOZIE-3528 > Project: Oozie > Issue Type: Improvement >Reporter: Julia Kinga Marton >Assignee: Julia Kinga Marton >Priority: Major > Attachments: OOZIE-3528-001.patch, OOZIE-3528-002.patch, > OOZIE-3528-003.patch, WIP_OOZIE-3528.patch > > > PowerMock 1 does not support JDK11. If we want to . support it we will need > to migrate to PoweMock 2 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (OOZIE-3528) Migrate to PowerMock 2 and Mockito2
[ https://issues.apache.org/jira/browse/OOZIE-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julia Kinga Marton updated OOZIE-3528: -- Attachment: OOZIE-3528-003.patch > Migrate to PowerMock 2 and Mockito2 > --- > > Key: OOZIE-3528 > URL: https://issues.apache.org/jira/browse/OOZIE-3528 > Project: Oozie > Issue Type: Improvement >Reporter: Julia Kinga Marton >Assignee: Julia Kinga Marton >Priority: Major > Attachments: OOZIE-3528-001.patch, OOZIE-3528-002.patch, > OOZIE-3528-003.patch, WIP_OOZIE-3528.patch > > > PowerMock 1 does not support JDK11. If we want to . support it we will need > to migrate to PoweMock 2 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (OOZIE-3200) Drop Instrumentation in favor of Metrics
[
https://issues.apache.org/jira/browse/OOZIE-3200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Salamon reassigned OOZIE-3200:
-
Assignee: (was: Andras Salamon)
> Drop Instrumentation in favor of Metrics
>
>
> Key: OOZIE-3200
> URL: https://issues.apache.org/jira/browse/OOZIE-3200
> Project: Oozie
> Issue Type: Improvement
> Components: monitoring
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Priority: Blocker
> Fix For: 6.0
>
>
> OOZIE-1817 added the option to use DropWizard Metrics instead of our
> homegrown Instrumentation. We left the Instrumentation as the default for
> compatibility; in Oozie 5.1.0, we should drop Instrumentation and only have
> Metrics.
> OOZIE-2645 deprecated {{Instrumentation}} and made {{Metrics}} the default
> choice, but no code cleanup happened there.
> We can also use this opportunity to clean up the code and interface for
> Metrics, which currently has to conform to Instrumentation for pluggability.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (OOZIE-3408) Create new unit test class for PurgeXCommand
[
https://issues.apache.org/jira/browse/OOZIE-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1688#comment-1688
]
Andras Salamon commented on OOZIE-3408:
---
I've created a few benchmarks back then but it was slow anyways so i'm closing
this issue.
> Create new unit test class for PurgeXCommand
>
>
> Key: OOZIE-3408
> URL: https://issues.apache.org/jira/browse/OOZIE-3408
> Project: Oozie
> Issue Type: Sub-task
> Components: tests
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
>
> There is one unit test class for {{PurgeXCommand}}, {{TestPurgeXCommand}}
> which needs to be cleaned up. This class tests the selection of the purgeable
> records. Most of the test cases are rather small, only a few workflows,
> coordinators, bundles have to be checked.
> It is also very important that purge service works correctly if large number
> of records has to be checked. Purge service should not only select the
> purgeable records correctly, it has to do that in a fast and memory efficient
> ways.
> Since {{TestPurgeXCommand}} is already very large, a new class should be
> introduced to contain these tests. Probably the two classes will have a
> common superclass.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Resolved] (OOZIE-3408) Create new unit test class for PurgeXCommand
[
https://issues.apache.org/jira/browse/OOZIE-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Salamon resolved OOZIE-3408.
---
Resolution: Won't Do
> Create new unit test class for PurgeXCommand
>
>
> Key: OOZIE-3408
> URL: https://issues.apache.org/jira/browse/OOZIE-3408
> Project: Oozie
> Issue Type: Sub-task
> Components: tests
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
>
> There is one unit test class for {{PurgeXCommand}}, {{TestPurgeXCommand}}
> which needs to be cleaned up. This class tests the selection of the purgeable
> records. Most of the test cases are rather small, only a few workflows,
> coordinators, bundles have to be checked.
> It is also very important that purge service works correctly if large number
> of records has to be checked. Purge service should not only select the
> purgeable records correctly, it has to do that in a fast and memory efficient
> ways.
> Since {{TestPurgeXCommand}} is already very large, a new class should be
> introduced to contain these tests. Probably the two classes will have a
> common superclass.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Resolved] (OOZIE-2882) Rerun workflow fails Error: E0404
[
https://issues.apache.org/jira/browse/OOZIE-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julia Kinga Marton resolved OOZIE-2882.
---
Resolution: Fixed
Fix Version/s: 5.2.0
This issue was fixed with OOZIE-3265
> Rerun workflow fails Error: E0404
> -
>
> Key: OOZIE-2882
> URL: https://issues.apache.org/jira/browse/OOZIE-2882
> Project: Oozie
> Issue Type: Improvement
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.2.0
>
>
> Only one of the properties are allowed [oozie.wf.rerun.skip.nodes OR
> oozie.wf.rerun.failnodes]
> Reproduction:
> 1. Create a workflow with more than 1 node. Eg: Fork - with three parallel
> shell actions. Make sure one of them fails
> 2. Rerun with 'oozie.wf.rerun.failnodes' set.
> 3. Rerun again with 'oozie.wf.rerun.skip.nodes' and check 'Skip all
> successful nodes'.
> You will get the following error.
> Error: E0404 : E0404: Only one of the properties are allowed
> [oozie.wf.rerun.skip.nodes OR oozie.wf.rerun.failnodes]
> When a user reruns a workflow job with oozie.wf.rerun.failnode=true and if
> the job fails in subsequent steps, we do not have an option to resubmit the
> workflow using oozie.wf.rerun.skip.node=action1,action2 to allow submission
> from predecessor steps.
> Currently, once the workflow fails and one of the rerun options is used for
> job rerun it gets merged and there is no way to override like regular oozie
> configurations or variables.
> We have a few options:
> 1. If fail.nodes and skip.nodes are specified at the same time (or one of
> them was carried over from a previous wf run), we can add {generate
> skip.nodes by discovering nodes that did not fail} union {skip.nodes}
> 2. Add a way to remove properties (this is also is potentially helpful for
> other use cases)
> 3. The "newest" property (oozie.wf.rerun.skip.nodes or
> oozie.wf.rerun.failnodes) takes priority and the previous is ignored
> 4. Make oozie.wf.rerun.skip.nodes or oozie.wf.rerun.failnodes somehow not
> persist in the DB
> Part of this JIRA would be to figure out which is the best option.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
