[ https://issues.apache.org/jira/browse/OOZIE-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16316599#comment-16316599 ]
Hadoop QA commented on OOZIE-1401: ---------------------------------- Testing JIRA OOZIE-1401 Cleaning local git workspace ---------------------------- {color:green}+1 PATCH_APPLIES{color} {color:green}+1 CLEAN{color} {color:red}-1 RAW_PATCH_ANALYSIS{color} . {color:green}+1{color} the patch does not introduce any @author tags . {color:green}+1{color} the patch does not introduce any tabs . {color:green}+1{color} the patch does not introduce any trailing spaces . {color:red}-1{color} the patch contains 2 line(s) longer than 132 characters . {color:green}+1{color} the patch adds/modifies 1 testcase(s) {color:green}+1 RAT{color} . {color:green}+1{color} the patch does not seem to introduce new RAT warnings {color:green}+1 JAVADOC{color} . {color:green}+1{color} the patch does not seem to introduce new Javadoc warnings {color:green}+1 COMPILE{color} . {color:green}+1{color} HEAD compiles . {color:green}+1{color} patch compiles . {color:green}+1{color} the patch does not seem to introduce new javac warnings {color:green}+1{color} There are no new bugs found in total. . {color:green}+1{color} There are no new bugs found in [docs]. . {color:green}+1{color} There are no new bugs found in [sharelib/distcp]. . {color:green}+1{color} There are no new bugs found in [sharelib/hive]. . {color:green}+1{color} There are no new bugs found in [sharelib/spark]. . {color:green}+1{color} There are no new bugs found in [sharelib/hive2]. . {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog]. . {color:green}+1{color} There are no new bugs found in [sharelib/streaming]. . {color:green}+1{color} There are no new bugs found in [sharelib/pig]. . {color:green}+1{color} There are no new bugs found in [sharelib/sqoop]. . {color:green}+1{color} There are no new bugs found in [sharelib/oozie]. . {color:green}+1{color} There are no new bugs found in [examples]. . {color:green}+1{color} There are no new bugs found in [client]. . {color:green}+1{color} There are no new bugs found in [core]. . {color:green}+1{color} There are no new bugs found in [tools]. . {color:green}+1{color} There are no new bugs found in [server]. {color:green}+1 BACKWARDS_COMPATIBILITY{color} . {color:green}+1{color} the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations . {color:green}+1{color} the patch does not modify JPA files {color:green}+1 TESTS{color} . Tests run: 2080 . {color:orange}Tests failed at first run:{color} TestJavaActionExecutor#testCredentialsSkip TestCoordMaterializeTriggerService#testCoordMaterializeTriggerService3 . For the complete list of flaky tests, see TEST-SUMMARY-FULL files. {color:green}+1 DISTRO{color} . {color:green}+1{color} distro tarball builds with the patch ---------------------------- {color:red}*-1 Overall result, please check the reported -1(s)*{color} The full output of the test-patch run is available at . https://builds.apache.org/job/PreCommit-OOZIE-Build/312/ > PurgeCommand should purge the workflow jobs w/o end_time > -------------------------------------------------------- > > Key: OOZIE-1401 > URL: https://issues.apache.org/jira/browse/OOZIE-1401 > Project: Oozie > Issue Type: Sub-task > Components: bundle, coordinator, workflow > Affects Versions: trunk > Reporter: Mona Chitnis > Assignee: Attila Sasvari > Fix For: 5.0.0b1 > > Attachments: OOZIE-1401-001.patch, amend-OOZIE-1401-001.patch, > amend-OOZIE-1401-002.patch > > > Currently, {{PurgeXCommand}} logic is not working with those workflow jobs > with {{end_time=null}}. This command needs to take care of those jobs as > well. This happens in the case of long stuck jobs after Hadoop restarts or DB > failures. It could be done by checking {{last_modified_time}} instead, if > {{end_time}} is not available. > The current query: > {code:sql} > select w from WorkflowJobBean w where w.endTimestamp < :endTime > {code} > There is also an issue when: > * there is a parent workflow that has its {{end_time}} set > * is otherwise eligible for {{PurgeXCommand}}: {{end_time}} is older than > configured number of days, and has {{status}} either {{KILLED}}, or > {{FAILED}}, or {{SUCCEEDED}} > * has a child workflow that has the {{parent_id}} set to the {{id}} of the > parent workflow > * child workflow has its {{end_time = NULL}} > In this case, > [*{{PurgeXCommand#fetchTerminatedWorkflow()}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/command/PurgeXCommand.java#L249] > throws a {{NullPointerException}} like this: > {noformat} > 2017-09-29 07:59:46,365 DEBUG org.apache.oozie.command.PurgeXCommand: > SERVER[host-10-17-101-90.coe.cloudera.com] USER[-] GROUP[-] TOKEN[-] APP[-] > JOB[-] ACTION[-] Purging workflows of long running coordinators is turned on > 2017-09-29 07:59:46,371 DEBUG org.apache.oozie.command.PurgeXCommand: > SERVER[host-10-17-101-90.coe.cloudera.com] USER[-] GROUP[-] TOKEN[-] APP[-] > JOB[-] ACTION[-] Execute command [purge] key [null] > 2017-09-29 07:59:46,371 INFO org.apache.oozie.command.PurgeXCommand: > SERVER[host-10-17-101-90.coe.cloudera.com] USER[-] GROUP[-] TOKEN[-] APP[-] > JOB[-] ACTION[-] STARTED Purge to purge Workflow Jobs older than [1] days, > Coordinator Jobs older than [1] days, and Bundlejobs older than [1] days. > 2017-09-29 07:59:46,375 ERROR org.apache.oozie.command.PurgeXCommand: > SERVER[host-10-17-101-90.coe.cloudera.com] USER[-] GROUP[-] TOKEN[-] APP[-] > JOB[-] ACTION[-] Exception, > java.lang.NullPointerException > at > org.apache.oozie.command.PurgeXCommand.fetchTerminatedWorkflow(PurgeXCommand.java:249) > at > org.apache.oozie.command.PurgeXCommand.processWorkflowsHelper(PurgeXCommand.java:227) > at > org.apache.oozie.command.PurgeXCommand.processWorkflows(PurgeXCommand.java:199) > at > org.apache.oozie.command.PurgeXCommand.execute(PurgeXCommand.java:150) > at org.apache.oozie.command.PurgeXCommand.execute(PurgeXCommand.java:53) > at org.apache.oozie.command.XCommand.call(XCommand.java:286) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)