[
https://issues.apache.org/jira/browse/KYLIN-6011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17928278#comment-17928278
]
Guoliang Sun commented on KYLIN-6011:
-------------------------------------
h3. Dev Design
In `org.apache.kylin.job.scheduler.JdbcJobScheduler#produceJob` and
`org.apache.kylin.job.scheduler.JdbcJobScheduler#consumeJob`, the SQL query for
releasing `job_lock` and the sorting for adding `job_lock` are inconsistent.
This causes the release process to prioritize locks with earlier creation
times, while the addition process prioritizes locks for higher-priority tasks.
To resolve this, unify the queries for releasing and adding locks by sorting
them based on priority.
> After the build task is completed, it is not cleared from the lock table,
> causing the remaining build tasks to fail to start
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: KYLIN-6011
> URL: https://issues.apache.org/jira/browse/KYLIN-6011
> Project: Kylin
> Issue Type: Bug
> Affects Versions: 5.0.0
> Reporter: Guoliang Sun
> Priority: Major
>
> h3. Root cause
> META and ROUTINE type tasks are set as the highest priority, see:
> `org.apache.kylin.job.execution.ExecutableManager#checkAndSubmitCronJob`.
> The job lock table prioritizes checking and cleaning the earliest created
> records when clearing expired locks, and by default retrieves only 30 entries
> at a time (`kylin.job.master-pull-batch-size`).
> If a large number of regular build tasks (> 30) are submitted first, followed
> by a large number of META or ROUTINE type tasks, the job lock table will fail
> to clean up completed META or ROUTINE tasks.
> The job scheduler will then continuously read completed META or ROUTINE tasks
> from the job lock table, preventing new tasks from being scheduled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
