0AyanamiRei opened a new pull request, #67681:
URL: https://github.com/apache/doris/pull/67681
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Routine load creation published a mutable job to the scheduler before
serializing its create journal. The scheduler could change the job from
`NEED_SCHEDULE` to `RUNNING` before serialization, while runtime task
membership was not persisted. Replaying such a record restored a `RUNNING` job
with no tasks. With a stable source topology, the job could remain stuck
without consuming data.
This PR makes the create journal durable before publishing the job to the
scheduler. It also restores persisted `RUNNING` jobs as `NEED_SCHEDULE`,
preserving their source topology and progress so normal scheduling rebuilds the
missing tasks. The recovery behavior applies to both Kafka and Kinesis routine
load jobs.
Before:
```text
publish job -> scheduler changes it to RUNNING -> serialize create journal
replay -> RUNNING + no runtime tasks
```
After:
```text
serialize create journal in NEED_SCHEDULE -> publish job -> scheduler
creates tasks
replay old RUNNING metadata -> NEED_SCHEDULE -> scheduler rebuilds tasks
```
#### Cloud Kafka follow-up discovered during review
This change exposes a pre-existing ordering issue in cloud Kafka
automatic-partition recovery that is intentionally left for a separate fix.
`isKafkaPartitionsChanged()` can update `currentKafkaPartitions` before
`updateCloudProgress()` successfully fetches the authoritative offset from Meta
Service. If that fetch fails, a later scheduler round can see a stable topology
and skip the Meta Service retry. Once the restored job is `NEED_SCHEDULE`, it
may then create a task from stale FE metadata, which can repeat already
committed data and regress the stored routine-load progress.
A follow-up should make authoritative cloud progress synchronization a
prerequisite for task creation and preserve the retry requirement across
scheduler rounds. It should cover automatic and explicit partitions, the first
Meta Service fetch failing and later succeeding, and verification that no task
starts from the stale serialized offset.
### Release note
Routine load jobs restored after FE restart or failover can rebuild missing
runtime tasks and continue consuming.
### Check List (For Author)
- Test
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [x] No need to test or manual test. Explain why:
- [x] Other reason: Added Kafka and Kinesis persistence/replay unit
coverage, but did not compile or run it locally per review request. Targeted FE
Checkstyle passed.
- Behavior changed:
- [ ] No.
- [x] Yes. Persisted `RUNNING` routine load jobs are restored as
`NEED_SCHEDULE` because runtime tasks are not persisted.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
