[
https://issues.apache.org/jira/browse/KYLIN-6087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18104041#comment-18104041
]
huangsheng commented on KYLIN-6087:
-----------------------------------
*Dev Design*
*Background*
Kylin currently supports manual incremental segment builds, but users must
trigger them externally when they need a model to build a fixed logical-day
range every day. The feature adds model-level auto segment build configuration
while keeping scheduling lightweight and safe for heavy build workloads.
*Design*
Each model stores its own {{auto_segment_build}} configuration:
{{{}enabled{}}}, {{{}trigger_time{}}}, {{{}logical_date_offset_days{}}},
{{{}data_range_start_time{}}}, and {{{}data_range_end_time{}}}. A single
configurable dispatcher runs on each server
({{{}kylin.model.auto-segment-build.dispatcher-cron{}}}, default every 30
seconds), but only the job-scheduler master performs dispatching. For every
project, the dispatcher enumerates online and healthy batch models, evaluates
each model’s trigger time in the project timezone, and submits only models
whose daily trigger falls into the current dispatch window. The build range is
calculated from the scheduled logical date rather than the server timezone.
The dispatcher does not register one in-memory cron task per model and does not
execute Spark build work itself. It calls {{ModelBuildService}} to create the
segment and persist the normal {{INC_BUILD}} job. Before submission, it
conservatively skips the model when any build-category job for that model is
still progressing, preventing concurrent model mutations. Existing transaction
and segment-overlap validation remain the final consistency guard.
*Reliability and Compatibility*
Dispatcher executions are serialized locally. The previous scan timestamp
defines an exclusive/inclusive trigger window {{{}(previousScan,
currentScan]{}}}, preventing duplicate dispatch within one process while
allowing delayed scans to catch the latest trigger. On startup or master
takeover, only a one-minute lookback is used, matching cron behavior without
replaying historical builds. Offline, broken, streaming, full-load, and
multi-partition models are skipped. Existing model configuration and REST/UI
contracts remain unchanged
> Support built-in auto scheduled segment build with model-level config
> ---------------------------------------------------------------------
>
> Key: KYLIN-6087
> URL: https://issues.apache.org/jira/browse/KYLIN-6087
> Project: Kylin
> Issue Type: New Feature
> Components: Job Engine
> Affects Versions: 5.0.3
> Reporter: huangsheng
> Assignee: huangsheng
> Priority: Minor
>
> Currently, in kylin, incremental segment build job can only be triggered
> manually. I would like to add an automatic timed trigger for segment build
> job function because now we all rely on external timing tools (such as
> crontab) to call the kylin incremental build api at regular intervals to
> start segment construction. For example, the build job of yesterday's segment
> is triggered at 1 a.m. every day
> I want to directly incorporate this function into kylin. My idea is to add a
> new entry for scheduled scheduling in the model configuration pop-up box. It
> is necessary to configure the daily trigger time and the time range for
> building this segment. There are two parameters to configure for the time
> range. One is the logical date, such as yesterday
> The day before yesterday, etc., one is a specific time, for example, from
> 0:00 to 24:00. Then the back end schedules build tasks at regular intervals
> based on this configuration.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
