JNSimba opened a new pull request, #69498:
URL: https://github.com/apache/doris/pull/69498
### What problem does this PR solve?
Issue Number: N/A
Related PR: #57264
Problem Summary:
Cloud streaming jobs query Meta Service for progress while replaying each
job update journal, making FE replay depend on Meta Service availability and
RPC latency. ALTER already persists the updated offset provider state and job
statistics in the journal.
Restore the provider state and cloud statistics locally during replay
instead of querying Meta Service. A paused job therefore retains its altered
offset and journaled statistics after an FE restart, before RESUME. PENDING
scheduling still fetches the authoritative committed progress from Meta Service
before dispatching tasks, and pauses the job if that fetch fails.
### Release note
Cloud streaming jobs restore journaled offsets and statistics during FE
replay without querying Meta Service. Progress is still synchronized from Meta
Service before consumption resumes.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Validation on the ByteDance development machine:
- `run-fe-ut.sh`: all 18 `StreamingInsertJobOffsetPersistenceTest` cases
passed in the final targeted run, including a serialized journal round trip,
paused offset/statistics recovery without an MS query, and failed PENDING
recovery preventing task dispatch.
- The remaining 131 streaming-package tests passed in the preceding
package run against the same production change.
- Checkstyle: zero violations in the two changed Java files.
- Existing ALTER-offset FE restart regression expectations are preserved.
Docker FE restart regression suites have not been run.
- Behavior changed:
- [ ] No.
- [x] Yes. Cloud job update replay restores local journal state instead
of querying Meta Service; PENDING progress synchronization is preserved.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
