[
https://issues.apache.org/jira/browse/OOZIE-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17632216#comment-17632216
]
Hadoop QA commented on OOZIE-3669:
----------------------------------
Testing JIRA OOZIE-3669
Cleaning local git workspace
----------------------------
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
. {color:green}+1{color} the patch does not introduce any @author tags
. {color:green}+1{color} the patch does not introduce any tabs
. {color:green}+1{color} the patch does not introduce any trailing spaces
. {color:green}+1{color} the patch does not introduce any star imports
. {color:green}+1{color} the patch does not introduce any line longer than
132
. {color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
. {color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
. {color:green}+1{color} Javadoc generation succeeded with the patch
. {color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
{color:green}+1 COMPILE{color}
. {color:green}+1{color} HEAD compiles
. {color:green}+1{color} patch compiles
. {color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [5] new bugs found below threshold in total that
must be fixed.
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/git].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:orange}0{color} There are [4] new bugs found in [server] that would
be nice to have fixed.
. You can find the SpotBugs diff here: server/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:red}-1{color} There are [5] new bugs found below threshold in
[core] that must be fixed.
. You can find the SpotBugs diff here (look for the red and orange ones):
core/findbugs-new.html
. The most important SpotBugs errors are:
. At BulkJPAExecutor.java:[line 206]: This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection
. At BulkJPAExecutor.java:[line 176]: At BulkJPAExecutor.java:[line 175]
. At BulkJPAExecutor.java:[line 205]: At BulkJPAExecutor.java:[line 199]
. This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection: At BulkJPAExecutor.java:[line 206]
. At BulkJPAExecutor.java:[line 111]: At BulkJPAExecutor.java:[line 127]
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
. {color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
. {color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
. Tests run: 3224
{color:green}+1 DISTRO{color}
. {color:green}+1{color} distro tarball builds with the patch
{color:green}+1 MODERNIZER{color}
----------------------------
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://ci-hadoop.apache.org/job/PreCommit-OOZIE-Build/98/
> Fix purge process for bundles to prevent orphan coordinators
> ------------------------------------------------------------
>
> Key: OOZIE-3669
> URL: https://issues.apache.org/jira/browse/OOZIE-3669
> Project: Oozie
> Issue Type: Bug
> Components: core
> Affects Versions: 5.2.1
> Reporter: Janos Makai
> Assignee: Janos Makai
> Priority: Major
> Attachments: OOZIE-3669-001.patch, OOZIE-3669-002.patch
>
>
> The Oozie purge process for bundles is creating orphan coordinators. When
> purging bundle jobs and bundle actions, it does not always purge coordinator
> jobs, etc. This causes orphaned coordinators, meaning neither they nor their
> children will ever be purged due to the purge logic.
>
> ----
>
> When purging bundles, it first compiles a list of any coordinators which are
> not ready to purge [0]. It checks the coord list for status and
> coordOlderThan. If the no child coordinator meets these criteria, it adds it
> to the coordsToPurge list.
> Being added to the list does not guarantee that the coordinator will be
> purged however. The processCoordinators method also has logic to check if the
> children workflows are older than wfOlderThan [1]. If a purge command is
> started where wfOlderThan is much higher than coordOlderThan (for example the
> default values are 30 days for workflows and 7 days for coordinators), then
> the bundle will be purged, but the coordinator will not.
> Since the bundle is now purged, the child coordinator will never be purged
> because only parentless coordinators will be checked, since coordinators with
> parents will only be purged when the bundle is purged
> [0]
> {code:java}
> PurgeXCommand
> 380 long numChildrenNotReady = jpaService.execute(
> 381 new CoordJobsCountNotForPurgeFromParentIdJPAExecutor(coordOlderThan,
> bundleId));
> CoordinatorJobBean
> 192 @NamedQuery(name = "GET_COORD_COUNT_WITH_PARENT_ID_NOT_READY_FOR_PURGE",
> query = "select count(w) from CoordinatorJobBean"
> 193 + " w where w.bundleId = :parentId and (w.statusStr NOT IN ('SUCCEEDED',
> 'FAILED', 'KILLED', 'DONEWITHERROR') "
> 194 + "OR w.lastModifiedTimestamp >= :lastModTime)"),
> {code}
>
> [1]
> {code:java}
> PurgeXCommand
> 343 List<String> workflowChildren = fetchTerminatedWorkflow(wfjBeanList);
> 344
> private boolean isWorkflowPurgeable(WorkflowJobBean wfjBean, long
> wfOlderThanMS) {
> 308 final Date wfEndTime = wfjBean.getEndTime();
> 309 final boolean isFinished = wfjBean.inTerminalState();
> 310 if (isFinished && wfEndTime != null && wfEndTime.getTime() <
> wfOlderThanMS)
> { 311 return true; 312 }
> 313 else {
> 314 final Date lastModificationTime = wfjBean.getLastModifiedTime();
> 315 if (isFinished && lastModificationTime != null &&
> lastModificationTime.getTime() < wfOlderThanMS)
> { 316 return true; 317 }
> 318 }
> 319 return false;
> 345 // if all workflow are ready to purge add them and add the coordinator
> and their actions
> 346 if(workflowChildren.size() == wfjBeanList.size()) {
> 347 LOG.debug("Purging coordinator " + coordId);
> 348 wfsToPurge.addAll(workflowChildren);
> 349 coordsToPurge.add(coordId);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
