[
https://issues.apache.org/jira/browse/KYLIN-6087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18104041#comment-18104041
]
huangsheng edited comment on KYLIN-6087 at 8/12/26 12:42 PM:
-------------------------------------------------------------
h3. Dev Design
h3.
Background
Kylin supports manual incremental segment builds, but users need an external
scheduler when a model must build a fixed logical-day range periodically. This
feature adds built-in automatic segment build scheduling while preserving
model-level configuration and workload isolation.
h3. Configuration
Each model stores its own {{auto_segment_build}} configuration:
{code:java}
{
"enabled": true,
"trigger_time": "01:00:00",
"logical_date_offset_days": 1,
"data_range_start_time": "00:00:00",
"data_range_end_time": "24:00:00"
} {code}
The dispatcher frequency is a server-level configuration:
{code:java}
kylin.model.auto-segment-build.dispatcher-cron=*/30 * * * * ? {code}
The dispatcher cron controls how often Kylin checks model schedules. It does
not replace the model-level {{{}trigger_time{}}}.
h3. Scheduling Architecture
A single dispatcher periodically scans projects instead of registering one
in-memory cron task for every model.
For each project, the dispatcher:
# Runs only on the job-scheduler master.
# Enumerates online and healthy models.
# Filters out disabled, broken, offline, streaming, full-load, and
multi-partition models.
# Evaluates each model’s {{trigger_time}} using the project timezone.
# Submits the model build only when its trigger falls in
{{{}(previousScanTime, currentScanTime]{}}}.
# Calls {{ModelBuildService}} to create the segment and persist a normal
{{INC_BUILD}} job.
The dispatcher does not execute Spark build work directly.
h3. Trigger Reliability
Dispatcher executions are serialized inside one process. The previous scan
timestamp creates an exclusive/inclusive trigger window, preventing duplicate
submission when the dispatcher runs multiple times around the same trigger.
On startup, clock rollback, or master takeover, the dispatcher uses a
one-minute lookback. This avoids replaying historical builds while still
covering normal scheduling jitter.
The segment range is calculated from the scheduled trigger date in the project
timezone, rather than from the server’s current local date. This keeps delayed
execution and timezone behavior deterministic.
h3. Concurrency Control
Before submission, the scheduler checks all progressing build-category jobs for
the same model.
If any model build job is progressing, the scheduled build is skipped. This is
intentionally model-level rather than segment-level because concurrent build
jobs may mutate shared model and segment metadata.
The current design does not retry that daily trigger after the progressing job
finishes. Existing transaction and segment-overlap validation remain the final
consistency guards against concurrent master transitions or duplicate
submissions.
h3. Compatibility
The existing model-level REST API, metadata structure, and UI configuration
remain unchanged. The change only replaces the internal per-model
{{ScheduledFuture}} implementation with a dispatcher.
h3.
was (Author: JIRAUSER297438):
*Dev Design*
*Background*
Kylin currently supports manual incremental segment builds, but users must
trigger them externally when they need a model to build a fixed logical-day
range every day. The feature adds model-level auto segment build configuration
while keeping scheduling lightweight and safe for heavy build workloads.
*Design*
Each model stores its own {{auto_segment_build}} configuration:
{{{}enabled{}}}, {{{}trigger_time{}}}, {{{}logical_date_offset_days{}}},
{{{}data_range_start_time{}}}, and {{{}data_range_end_time{}}}. A single
configurable dispatcher runs on each server
({{{}kylin.model.auto-segment-build.dispatcher-cron{}}}, default every 30
seconds), but only the job-scheduler master performs dispatching. For every
project, the dispatcher enumerates online and healthy batch models, evaluates
each model’s trigger time in the project timezone, and submits only models
whose daily trigger falls into the current dispatch window. The build range is
calculated from the scheduled logical date rather than the server timezone.
The dispatcher does not register one in-memory cron task per model and does not
execute Spark build work itself. It calls {{ModelBuildService}} to create the
segment and persist the normal {{INC_BUILD}} job. Before submission, it
conservatively skips the model when any build-category job for that model is
still progressing, preventing concurrent model mutations. Existing transaction
and segment-overlap validation remain the final consistency guard.
*Reliability and Compatibility*
Dispatcher executions are serialized locally. The previous scan timestamp
defines an exclusive/inclusive trigger window {{{}(previousScan,
currentScan]{}}}, preventing duplicate dispatch within one process while
allowing delayed scans to catch the latest trigger. On startup or master
takeover, only a one-minute lookback is used, matching cron behavior without
replaying historical builds. Offline, broken, streaming, full-load, and
multi-partition models are skipped. Existing model configuration and REST/UI
contracts remain unchanged
> Support built-in auto scheduled segment build with model-level config
> ---------------------------------------------------------------------
>
> Key: KYLIN-6087
> URL: https://issues.apache.org/jira/browse/KYLIN-6087
> Project: Kylin
> Issue Type: New Feature
> Components: Job Engine
> Affects Versions: 5.0.3
> Reporter: huangsheng
> Assignee: huangsheng
> Priority: Minor
>
> Currently, in kylin, incremental segment build job can only be triggered
> manually. I would like to add an automatic timed trigger for segment build
> job function because now we all rely on external timing tools (such as
> crontab) to call the kylin incremental build api at regular intervals to
> start segment construction. For example, the build job of yesterday's segment
> is triggered at 1 a.m. every day
> I want to directly incorporate this function into kylin. My idea is to add a
> new entry for scheduled scheduling in the model configuration pop-up box. It
> is necessary to configure the daily trigger time and the time range for
> building this segment. There are two parameters to configure for the time
> range. One is the logical date, such as yesterday
> The day before yesterday, etc., one is a specific time, for example, from
> 0:00 to 24:00. Then the back end schedules build tasks at regular intervals
> based on this configuration.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
