0AyanamiRei opened a new pull request, #67696:
URL: https://github.com/apache/doris/pull/67696
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
A failed `ALTER ROUTINE LOAD` can leave a Kinesis job partially modified in
memory, even though the statement returns an error and no ALTER journal entry
is written.
For example, pause a job whose progress contains `shard-0`, then alter its
region, endpoint, or default position together with an explicit shard list
containing an unknown shard. Omit `kinesis_stream` so that the
existing-progress membership check applies. Before this change,
`modifyPropertiesInternal()` updates the live data-source properties and
replaces `customKinesisShards` before `KinesisProgress.checkShards()` rejects
the unknown shard. The exception releases the job lock without restoring those
fields.
The leader can therefore retain configuration that journal replay does not
apply to other FEs. A residual explicit shard list can also bypass remote shard
discovery after resume. There is another failure point in the same ALTER path:
common-property validation for flexible partial updates can fail after Kinesis
state and some common properties have already changed.
This change orders validation before mutation:
1. Read the requested Kinesis changes into local variables and check shard
membership before modifying any job state. Preserve the existing progress-reset
behavior when a stream is provided.
2. Validate the requested unique-key update mode and flexible partial-update
constraints before assigning any common job properties in `RoutineLoadJob`.
3. Apply the common properties before updating Kinesis configuration, shard
lists, and progress. The existing successful ALTER path then writes its journal
entry.
The guarantee covers shard-membership and common-property validation
failures. These failures now occur before any live state changes, so they
require no rollback. The shared `RoutineLoadJob` adjustment is necessary to
keep common-property validation from becoming another source of partial Kinesis
ALTER updates.
### Release note
Kinesis Routine Load preserves its previous configuration, shard selection,
and progress when ALTER fails shard-membership or common-property validation.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- Behavior changed:
- [ ] No.
- [x] Yes. Failed validation leaves the Kinesis job state unchanged and
does not write an ALTER journal entry.
- Does this need documentation?
- [x] No.
- [ ] Yes.
Validation:
- Before the fix, the four new failure-atomicity test cases fail because job
state changes after the expected ALTER error. The other 11 Kinesis job tests
pass.
- After the fix, all 47 tests pass, with no failures, errors, or skipped
tests:
```bash
./run-fe-ut.sh --run
org.apache.doris.load.routineload.KinesisRoutineLoadJobTest,org.apache.doris.load.routineload.RoutineLoadJobTest,org.apache.doris.load.routineload.KafkaRoutineLoadJobTest,org.apache.doris.load.routineload.kinesis.KinesisDataSourcePropertiesTest
```
- New tests cover dynamic and explicit shard selection, shared open/explicit
shard list instances, unchanged state and no journal entry on failure, shard
discovery after the resume state transition, common-property validation
failures with and without a stream change, and successful ALTER journal
serialization/replay.
- FE build via `build.sh --fe -j48` passes with `DISABLE_BUILD_UI=ON` and
`DISABLE_JAVA_CHECK_STYLE=OFF`; output is directed to a separate local build
directory. Checkstyle reports zero violations. UI building is disabled because
the local npm dependency installation fails.
- `git diff --check` passes.
- Real AWS Kinesis consumption and multi-FE failover have not been tested.
Shard discovery is mocked in the unit tests.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
