[
https://issues.apache.org/jira/browse/KYLIN-6087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18110155#comment-18110155
]
ASF GitHub Bot commented on KYLIN-6087:
---------------------------------------
Mrhs121 commented on code in PR #2347:
URL: https://github.com/apache/kylin/pull/2347#discussion_r3900777472
##########
src/data-loading-service/src/main/java/org/apache/kylin/rest/scheduler/AutoBuildSegmentScheduler.java:
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kylin.rest.scheduler;
+
+import static
org.apache.kylin.metadata.model.AutoSegmentBuildConfig.END_OF_DAY;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;
+import java.time.ZoneId;
+import java.time.ZonedDateTime;
+import java.time.format.DateTimeFormatter;
+import java.time.format.DateTimeParseException;
+import java.util.List;
+import java.util.Locale;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicReference;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.kylin.common.KylinConfig;
+import org.apache.kylin.common.util.Pair;
+import org.apache.kylin.guava30.shaded.common.annotations.VisibleForTesting;
+import org.apache.kylin.guava30.shaded.common.collect.Lists;
+import org.apache.kylin.job.execution.AbstractExecutable;
+import org.apache.kylin.job.execution.ExecutableManager;
+import org.apache.kylin.job.execution.ExecutableState;
+import org.apache.kylin.job.execution.JobTypeEnum;
+import org.apache.kylin.job.util.JobContextUtil;
+import org.apache.kylin.metadata.cube.model.NDataflowManager;
+import org.apache.kylin.metadata.model.AutoSegmentBuildConfig;
+import org.apache.kylin.metadata.model.NDataModel;
+import org.apache.kylin.metadata.model.PartitionDesc;
+import org.apache.kylin.metadata.project.NProjectManager;
+import org.apache.kylin.metadata.project.ProjectInstance;
+import org.apache.kylin.rest.service.ModelBuildService;
+import org.apache.kylin.rest.service.params.IncrementBuildSegmentParams;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.beans.factory.annotation.Qualifier;
+import org.springframework.scheduling.annotation.Scheduled;
+import org.springframework.stereotype.Component;
+
+import lombok.val;
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+@Component
+public class AutoBuildSegmentScheduler {
+ private static final Duration INITIAL_TRIGGER_LOOKBACK =
Duration.ofMinutes(1);
+ private static final DateTimeFormatter TIME_FORMATTER =
DateTimeFormatter.ofPattern("HH:mm:ss", Locale.ROOT);
+
+ @Autowired
+ @Qualifier("modelBuildService")
+ private ModelBuildService modelBuildService;
+
+ private final AtomicBoolean dispatching = new AtomicBoolean(false);
+ private final AtomicReference<Instant> lastDispatchTime = new
AtomicReference<>();
+
+ @Scheduled(cron = "${kylin.model.auto-segment-build.dispatcher-cron:*/30 *
* * * ?}")
Review Comment:
This system-level parameter controls only the internal dispatcher scan
interval, rather than the actual model build schedule.
The actual `trigger_time` is stored in model metadata and read on every
scan, so changes to a model's schedule take effect immediately without
restarting Kylin. Keeping a single system-level dispatcher also avoids
maintaining separate scheduler lifecycles for every project or model.
To make this distinction clearer, I changed `dispatcher-cron` to
`dispatcher-interval-ms` and added documentation explaining that it is only the
internal scan cadence. The startup lookback window is also derived from the
same interval.
> Support built-in auto scheduled segment build with model-level config
> ---------------------------------------------------------------------
>
> Key: KYLIN-6087
> URL: https://issues.apache.org/jira/browse/KYLIN-6087
> Project: Kylin
> Issue Type: New Feature
> Components: Job Engine
> Affects Versions: 5.0.3
> Reporter: huangsheng
> Assignee: huangsheng
> Priority: Minor
>
> Currently, in kylin, incremental segment build job can only be triggered
> manually. I would like to add an automatic timed trigger for segment build
> job function because now we all rely on external timing tools (such as
> crontab) to call the kylin incremental build api at regular intervals to
> start segment construction. For example, the build job of yesterday's segment
> is triggered at 1 a.m. every day
> I want to directly incorporate this function into kylin. My idea is to add a
> new entry for scheduled scheduling in the model configuration pop-up box. It
> is necessary to configure the daily trigger time and the time range for
> building this segment. There are two parameters to configure for the time
> range. One is the logical date, such as yesterday
> The day before yesterday, etc., one is a specific time, for example, from
> 0:00 to 24:00. Then the back end schedules build tasks at regular intervals
> based on this configuration.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
