Lei Xia created HELIX-617:
-----------------------------
Summary: Job IdealState is generated even the job is not running
and not removed when it is completed.
Key: HELIX-617
URL: https://issues.apache.org/jira/browse/HELIX-617
Project: Apache Helix
Issue Type: Task
Reporter: Lei Xia
Assignee: Lei Xia
-- Problem:
In current task framework implementation, one IdealState and one ExternalView
znode are generated for each job once the job is queued (not when it is
started). These znodes will stay in the zookeeper until the specified data
expiry time (usually one day in Espresso backup setting). This is even worse
for recurrent job queue. For recurrent job queue, one idealstate/externalview
is created for original job template, and one idealstate/externalview will be
created for each scheduled run of the job.
This usually generates a significant amounts of znodes in IdealState. For
example, to schedule a queue with 100 jobs, 100 "template" idealstates are
created and will stay there for ever. While a new set of 100 "scheduling"
idealstates are created for every time all these jobs are scheduled to run,
which will stay until the expiry time setup in the job configuration has passed
(usually a relative long time).
-- Proposed Changes:
The IdealState of a job will be generated only when it is scheduled to run, and
will be removed immediately once the job is completed. In practice, there will
be only one job running at the same time for each queue, thus only one
IdealState will exist for each queue at any given time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
