[
https://issues.apache.org/jira/browse/KYLIN-6087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18104041#comment-18104041
]
huangsheng edited comment on KYLIN-6087 at 9/1/26 11:34 AM:
------------------------------------------------------------
h1. Dev Design
h2. Background
Kylin supports manual incremental segment builds, but users otherwise need an
external scheduler when models must build fixed logical-date ranges
periodically.
This feature provides built-in automatic segment scheduling with two levels of
control:
* A project-level switch determines whether the dispatcher processes the
project.
* Each model retains its own schedule and target-range configuration.
The dispatcher only submits standard Kylin build jobs. It does not execute
Spark workloads directly.
h2. Configuration
h3. Project-Level Switch
Automatic segment dispatch is disabled by default and must be enabled for each
project:
{{{
"auto_segment_build_enabled": true
}}}
The switch is available under {*}Project Settings > Basic Settings > Segment
Settings{*}.
Disabling the project switch stops automatic dispatch without deleting or
modifying model-level configurations.
h3. Model-Level Configuration
Each model stores its own {{auto_segment_build}} configuration:
{{{
"enabled": true,
"trigger_time": "01:00:00",
"logical_date_offset_days": 1,
"data_range_start_time": "00:00:00",
"data_range_end_time": "24:00:00"
}}}
Field semantics:
||Field||Description||
|{{enabled}}|Enables automatic segment builds for this model.|
|{{trigger_time}}|Daily trigger time in the project timezone, formatted as
{{{}HH:mm:ss{}}}.|
|{{logical_date_offset_days}}|Number of days subtracted from the scheduled
trigger date. It must be at least {{{}1{}}}.|
|{{data_range_start_time}}|Start time on the calculated logical date.|
|{{data_range_end_time}}|End time on the logical date. {{24:00:00}} represents
midnight at the end of that date.|
{{24:00:00}} is accepted only for {{{}data_range_end_time{}}}. Other time
values must be valid {{HH:mm:ss}} values.
Automatic segment builds are not supported for:
* Broken models
* Streaming models
* Full-load models without a partition column
* Multi-partition models
Offline batch models remain eligible. Model online status does not control
automatic segment scheduling.
h3. Dispatcher Interval
The dispatcher uses a server-level fixed-delay configuration:
{{kylin.model.auto-segment-build.dispatcher-interval-ms=30000}}
The default interval is 30 seconds. The next scan starts after the previous
scan completes and the configured delay elapses.
This interval controls how frequently Kylin checks schedules. It does not
replace the model-level {{{}trigger_time{}}}.
h2. Scheduling Architecture
A single global dispatcher scans metadata instead of registering one in-memory
cron task for every model.
For each execution, the dispatcher:
# Verifies that the current node is the job-scheduler master.
# Uses an in-process atomic guard to prevent overlapping dispatcher executions.
# Enumerates all projects.
# Skips projects whose {{auto_segment_build_enabled}} switch is disabled.
# Enumerates healthy underlying models, including both online and offline
models.
# Filters models that are unsupported or whose model-level schedule is
disabled or invalid.
# Calculates the latest scheduled occurrence that is not later than the
current time.
# Calculates and normalizes the target segment range.
# Evaluates existing segments and build jobs against the target range.
# Calls {{ModelBuildService.incrementBuildSegmentsByScheduler}} when
submission is allowed.
{{ModelBuildService}} creates the segment and persists a normal {{INC_BUILD}}
job using the scheduler submitter identity.
h2. Schedule Calculation
The scheduled occurrence is calculated in the project timezone:
{{if today's trigger time <= current time:
scheduled occurrence = today's trigger
else:
scheduled occurrence = yesterday's trigger}}
The logical date and target range are then calculated from that scheduled
occurrence:
{{logical date = scheduled local date - logical_date_offset_days
target start = logical date + data_range_start_time
target end = logical date + data_range_end_time}}
For example, with:
{{Project timezone: Asia/Shanghai
Current time: 2026-09-01 10:00:00
Trigger time: 01:00:00
Offset: 1 day
Data range: 00:00:00 - 24:00:00}}
The latest scheduled occurrence is {{{}2026-09-01 01:00:00{}}}, and the target
range is:
{{[2026-08-31 00:00:00, 2026-09-01 00:00:00)}}
The calculated timestamps are normalized with the model partition-date format
before being compared with existing segment ranges. This keeps dispatcher
decisions consistent with {{{}ModelBuildService{}}}.
h2. Trigger Reliability
The dispatcher no longer relies on a {{(previousScanTime, currentScanTime]}}
trigger window.
Every scan evaluates the latest due schedule cycle. This provides the following
behavior:
* A scan delayed beyond {{trigger_time}} still processes the cycle.
* A process restart after {{trigger_time}} still processes the cycle.
* A master takeover after {{trigger_time}} still processes the cycle.
* Enabling a project or model after {{trigger_time}} processes the latest due
cycle.
Only the latest due cycle is evaluated. The dispatcher does not replay every
historical cycle after a multi-day outage.
Each cycle is identified by:
{{project / model / scheduled occurrence / normalized target range}}
An in-memory cache tracks up to 10,000 cycle states and expires entries after
two days of inactivity.
Only fully covered cycles are treated as complete and skipped without further
metadata checks. Other states are re-evaluated during subsequent scans.
The cache is an optimization rather than the source of correctness. After
restart, failover, or cache eviction, existing segment metadata is used to
determine whether the cycle has already been completed.
h2. Range-Aware Conflict Handling
The scheduler compares the normalized target range with existing {{{}READY{}}},
{{{}WARNING{}}}, and {{NEW}} segments.
||State||Condition||Action||
|{{COVERED}}|One segment, a superset segment, or a contiguous union of
{{READY}} segments completely covers the target.|Mark the cycle complete and do
not submit.|
|{{COVERED_WITH_WARNING}}|The target is completely covered, but required
coverage includes a {{WARNING}} segment.|Mark the cycle complete, do not
submit, and log a warning.|
|{{IN_PROGRESS}}|An overlapping {{NEW}} segment is associated with a
{{{}READY{}}}, {{{}PENDING{}}}, or {{RUNNING}} build job.|Wait and re-evaluate
on the next dispatcher scan.|
|{{BLOCKED}}|An overlapping {{NEW}} segment is associated with a
non-progressing job, such as {{ERROR}} or {{{}PAUSED{}}}, or has no
corresponding job.|Do not submit and continue re-evaluating.|
|{{PARTIAL_OVERLAP}}|Existing {{READY}} or {{WARNING}} segments overlap the
target but do not completely cover it.|Do not automatically replace or expand
the existing segments. Continue re-evaluating.|
|{{NO_OVERLAP}}|No existing segment overlaps the target.|Submit the automatic
build.|
|{{SUBMITTED}}|{{ModelBuildService}} accepted the request.|Re-evaluate metadata
during the next scan until coverage is complete.|
|{{RETRYABLE_FAILURE}}|Submission failed with an exception.|Retry during the
next dispatcher scan.|
Completed coverage is evaluated before in-progress segments. Therefore, if
usable segments already cover the complete target, no additional automatic
build is submitted.
h2. Boundary Behavior
The following cases are handled explicitly:
||Existing condition||Result||
|The exact target range was manually built successfully.|Skip because the
target is covered.|
|A larger successful segment contains the target range.|Skip because the target
is covered.|
|Multiple adjacent successful segments collectively cover the target.|Skip
because the target is covered.|
|An exact-range segment is currently building.|Wait and re-evaluate.|
|A larger segment containing the target is currently building.|Wait; after
success, the target becomes covered.|
|A smaller segment inside the target is currently building.|Wait; after
success, the result becomes a partial-overlap conflict.|
|A segment partially intersects the target and is currently building.|Wait;
after success, the result becomes a partial-overlap conflict.|
|Another segment build is running on a disjoint range.|Allow the scheduled
target to be submitted.|
|An overlapping build job is paused or failed.|Block submission until metadata
or job state is resolved.|
|An overlapping {{NEW}} segment has no corresponding job.|Treat it as an orphan
and block submission.|
|Submission fails temporarily.|Retry during the next dispatcher scan.|
The scheduler intentionally does not delete existing segments, split the target
range, build only missing holes, or replace manually created partial ranges.
h2. Concurrency Control
Concurrency control is range-aware rather than model-wide.
A progressing build job blocks automatic submission only when its target
segment overlaps the scheduled target range. Build jobs on disjoint ranges do
not block submission.
Correctness is protected at multiple levels:
* Only the job-scheduler master runs the dispatcher.
* The in-process atomic guard serializes dispatcher scans.
* Existing segment metadata provides idempotency.
* Kylin job submission validates overlapping running jobs.
* Segment creation and job persistence run inside existing metadata
transactions.
* Failed submissions remain retryable.
These existing transactional and overlap checks remain the final consistency
guard during master transitions or concurrent manual submissions.
h2. Observability
The dispatcher logs state transitions with the project, model, target range,
and related segments.
Repeated unchanged states do not repeatedly emit the same warning. Repeated
submission failures are downgraded to debug logging after the initial error.
Invalid model configurations are skipped, and identical validation warnings are
suppressed for one day.
h2. Compatibility
The existing model-level {{auto_segment_build}} metadata, REST API, and UI
configuration remain unchanged.
The updated implementation adds:
* A project-level {{auto_segment_build_enabled}} switch
* A fixed-delay dispatcher interval
* Latest-cycle catch-up behavior
* Range-aware conflict detection
* Retry of unresolved or transient states
Existing projects default to project-level automatic dispatch being disabled.
Existing model configurations remain stored and become active when the project
switch is enabled.
No new Spark execution path or job type is introduced. Automatic builds
continue to use the standard {{INC_BUILD}} workflow.
was (Author: JIRAUSER297438):
h3. Dev Design
h3.
Background
Kylin supports manual incremental segment builds, but users need an external
scheduler when a model must build a fixed logical-day range periodically. This
feature adds built-in automatic segment build scheduling while preserving
model-level configuration and workload isolation.
h3. Configuration
Each model stores its own {{auto_segment_build}} configuration:
{code:java}
{
"enabled": true,
"trigger_time": "01:00:00",
"logical_date_offset_days": 1,
"data_range_start_time": "00:00:00",
"data_range_end_time": "24:00:00"
} {code}
The dispatcher frequency is a server-level configuration:
{code:java}
kylin.model.auto-segment-build.dispatcher-cron=*/30 * * * * ? {code}
The dispatcher cron controls how often Kylin checks model schedules. It does
not replace the model-level {{{}trigger_time{}}}.
h3. Scheduling Architecture
A single dispatcher periodically scans projects instead of registering one
in-memory cron task for every model.
For each project, the dispatcher:
# Runs only on the job-scheduler master.
# Enumerates online and healthy models.
# Filters out disabled, broken, offline, streaming, full-load, and
multi-partition models.
# Evaluates each model’s {{trigger_time}} using the project timezone.
# Submits the model build only when its trigger falls in
{{{}(previousScanTime, currentScanTime]{}}}.
# Calls {{ModelBuildService}} to create the segment and persist a normal
{{INC_BUILD}} job.
The dispatcher does not execute Spark build work directly.
h3. Trigger Reliability
Dispatcher executions are serialized inside one process. The previous scan
timestamp creates an exclusive/inclusive trigger window, preventing duplicate
submission when the dispatcher runs multiple times around the same trigger.
On startup, clock rollback, or master takeover, the dispatcher uses a
one-minute lookback. This avoids replaying historical builds while still
covering normal scheduling jitter.
The segment range is calculated from the scheduled trigger date in the project
timezone, rather than from the server’s current local date. This keeps delayed
execution and timezone behavior deterministic.
h3. Concurrency Control
Before submission, the scheduler checks all progressing build-category jobs for
the same model.
If any model build job is progressing, the scheduled build is skipped. This is
intentionally model-level rather than segment-level because concurrent build
jobs may mutate shared model and segment metadata.
The current design does not retry that daily trigger after the progressing job
finishes. Existing transaction and segment-overlap validation remain the final
consistency guards against concurrent master transitions or duplicate
submissions.
h3. Compatibility
The existing model-level REST API, metadata structure, and UI configuration
remain unchanged. The change only replaces the internal per-model
{{ScheduledFuture}} implementation with a dispatcher.
h3.
> Support built-in auto scheduled segment build with model-level config
> ---------------------------------------------------------------------
>
> Key: KYLIN-6087
> URL: https://issues.apache.org/jira/browse/KYLIN-6087
> Project: Kylin
> Issue Type: New Feature
> Components: Job Engine
> Affects Versions: 5.0.3
> Reporter: huangsheng
> Assignee: huangsheng
> Priority: Minor
>
> Currently, in kylin, incremental segment build job can only be triggered
> manually. I would like to add an automatic timed trigger for segment build
> job function because now we all rely on external timing tools (such as
> crontab) to call the kylin incremental build api at regular intervals to
> start segment construction. For example, the build job of yesterday's segment
> is triggered at 1 a.m. every day
> I want to directly incorporate this function into kylin. My idea is to add a
> new entry for scheduled scheduling in the model configuration pop-up box. It
> is necessary to configure the daily trigger time and the time range for
> building this segment. There are two parameters to configure for the time
> range. One is the logical date, such as yesterday
> The day before yesterday, etc., one is a specific time, for example, from
> 0:00 to 24:00. Then the back end schedules build tasks at regular intervals
> based on this configuration.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
