GitHub user narendly opened a pull request:
https://github.com/apache/helix/pull/288
PR
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/narendly/helix master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/helix/pull/288.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #288
----
commit 3844ad60034b029f3bbd916f629a7969117c1b26
Author: narendly <narendly@...>
Date: 2018-11-01T23:54:48Z
[HELIX-782] TASK: Make TaskDriver use ZKClient's create when creating
workflows
TaskDriver should use create() but currently is using set(), which just
overwrites ZNodes that are in ZK. This is undesirable and we need to fix it,
especially in the wake of ZNode restructuring.
AC:
1. Make TaskDriver use create() instead of set()
2. Add an integration test:
TestWorkflowCreation:testWorkflowCreationNoDuplicates()
commit 3d9c03064a5c26a9ed9ad674567674f2d9eca160
Author: narendly <narendly@...>
Date: 2018-11-01T23:55:59Z
TASK: Fix JobQueue's job state-related bug
The bug was observed in
TestTaskRebalancerStopResume:stopAndResumeNamedQueue(), which was being
unstable.
It was observed that for JobQueues with multiple jobs, the second job would
get marked as IN_PROGRESS even though the first job hadn't completed/failed,
especially when the queue was being stopped and resumed. This was due to a bug
in getIncompleteJobCount() because it was not counting jobs in STOPPING state.
This was fixed and another check was added right before JobDispatcher marks a
job as STOPPED so that it would not mark it STOPPED if the job state is
NOT_STARTED.
Changelist:
1. Fix getIncompleteJobCount()
2. Add a check so that we don't mark NOT_STARTED jobs as STOPPED
commit befb1036f8d8be2729a800d3dde88fc1362a6489
Author: narendly <narendly@...>
Date: 2018-11-01T23:57:33Z
[HELIX-784] TASK: Fix a bug in getExpiredJobs
getExpiredJobs(), when the job config is null, would just continue instead
of adding it to expiredJobs so that the job cleanup/purge would be re-tried.
This could possibly cause purge failures to leave a lot of jobs un-purged with
just the job config missing in ZK. This RB fixes this.
Changelist:
1. Add the job name to expiredJobs if the job config does not exist in ZK
2. Add a more detailed description in the error log
3. Add an integration test for two task-related stages:
TaskPersistDataStage and TaskGarbageCollectionStage in TestTaskStage.java
----
---
