0AyanamiRei opened a new pull request, #66634:
URL: https://github.com/apache/doris/pull/66634
### What problem does this PR solve?
Issue Number: N/A
Related PR: #64878
Problem Summary:
Routine Load historically persists the original CREATE statement in
`origStmt` and reparses it when an image is loaded. However, `ALTER ROUTINE
LOAD` changes the effective load definition stored in `RoutineLoadJob` without
changing the original CREATE statement. After ALTER followed by follower
replay, checkpoint, or FE restart, the job could therefore recover the original
CREATE semantics instead of the definition currently used to create tasks.
This PR treats the existing `RoutineLoadJob` runtime fields as the
authoritative effective state and fixes both persistence paths:
- **Image/checkpoint:** persist partitions, column mappings,
preceding/where/delete filters, separators, sequence column, merge type,
execution memory limit, and `memtable_on_sink_node` directly on
`RoutineLoadJob`.
- **ALTER journal:** persist a nullable `RoutineLoadDesc` delta in
`AlterRoutineLoadJobOperationLog`, apply it under the job write lock on the
leader, and apply properties and the delta in the same order during follower
replay.
The design deliberately does not introduce a `LoadDefinition` wrapper or
duplicate job/data-source snapshots:
- The fields consumed by Kafka/Kinesis task construction are the image
source of truth.
- `jobProperties` remains authoritative for cached properties such as max
filter ratio, batch parallelism, single-tablet loading, partial-update
mode/policy, and CSV `enclose/escape/empty_field_as_null`.
- New-format images restore direct state without creating a SQL context or
accessing the catalog.
- `origStmt` remains persisted for downgrade readability and legacy image
migration; ALTER does not rewrite it.
- Old images are identified by the absence of the newly persisted merge
type. The new FE reparses `origStmt` once, restores load clauses and
`exec_mem_limit`, and the next checkpoint writes the direct-state format.
- Legacy `memtable_on_sink_node` cannot be reconstructed and retains its
historical post-restart value of `false`.
The ALTER journal stores only clauses changed by that ALTER. Nullable
`RoutineLoadDesc` fields preserve clauses not modified by the delta. Old logs
do not contain the new field and therefore leave the current load definition
unchanged during replay.
CSV ALTER properties are validated only on the leader before mutation.
Replay trusts persisted journal values so validation added by a newer FE cannot
block a journal written by an older FE. Runtime CSV caches are updated together
with `jobProperties` and rebuilt from that map after image recovery.
### Expr serialization dependency
Routine Load directly persists Expr objects from column mappings, preceding
filters, where filters, and delete conditions. These expressions are converted
back to SQL before each Nereids task is built, so every field that affects
`ExprToSqlVisitor` output must survive Gson round-trip.
This PR hardens the shared legacy Expr serialization contract:
- Adds stable serialization for SQL-relevant state in `PlaceHolderExpr`,
`TimeV2Literal`, `MatchPredicate`, `SearchPredicate`, `SlotRef`, and
`VariableExpr`.
- Persists `FunctionCallExpr` ORDER BY metadata and nested `OrderByElement`
state.
- Requires all concrete Expr subtypes to be present in both Gson registries
and in a non-default sample set.
- Reflectively requires every Expr instance field to have `@SerializedName`
or be explicitly classified as non-durable derived state.
- Verifies that Gson round-trip preserves both
`ExprToSqlVisitor(WITH_TABLE)` and `ExprToSqlVisitor(WITHOUT_TABLE)` output for
every registered concrete subtype.
- Adds `analysis/AGENTS.md` review guidance so future Expr changes account
for metadata persistence.
Compatibility boundary:
- **Old image -> new FE:** supported through one-time `origStmt` migration.
ALTER clauses that were never recorded by the old image or old journal are not
recoverable; the original CREATE semantics are restored.
- **New image/log -> old FE:** unknown JSON fields are structurally
ignorable and `origStmt` is still available, but semantic downgrade is not
guaranteed. An old FE may restore the original CREATE clauses and ignore new
ALTER deltas.
- Avoid ALTERing Routine Load load clauses until every FE has been upgraded.
During a mixed-version rolling upgrade, an old follower can ignore the new
journal delta, while a new follower cannot reconstruct a delta omitted by an
old leader.
- Rolling back after a new-version checkpoint can lose post-upgrade ALTER
semantics.
Out of scope:
- Target-table ALTER.
- Full failed-ALTER/edit-log-write atomicity refactoring.
- Pause/cancel reason persistence, automatic-recovery backoff,
state-transition timestamps, and unrelated Routine Load lifecycle issues.
Tests added or extended:
- Complete direct-state image round-trip with invalid `origStmt`,
non-default task configuration, all load clauses, and SQL-sensitive Expr values.
- Empty load-definition recovery proving new images do not fall back to SQL
parsing.
- Fixed image and ALTER-log fixtures generated by the real serializer at
merge base `a8928245`.
- Kafka/Kinesis durable-versus-derived state recovery.
- ALTER journal serialization for every load-clause category and complex
nested Expr values.
- Kafka/Kinesis leader-to-replay delta application, CSV cache
synchronization, legacy journal replay, and checkpoint parity.
- Full concrete Expr subtype coverage, field-classification enforcement, and
SQL-semantic Gson round-trip.
- A three-FE Docker case covering follower journal replay before checkpoint,
leader failover, a subsequent ALTER on the new leader, checkpoint creation, and
FE restart recovery.
Validation status for the final head:
- `git diff --check`: passed.
- FE compile, testCompile, and checkstyle: passed.
- Focused FE unit tests: 36 passed, 0 failures, 0 errors.
- The three-FE Docker regression case was added but has not been executed
locally.
### Release note
Routine Load jobs now preserve the effective load definition after ALTER
across follower replay, leader failover, checkpoint, and FE restart.
During a rolling upgrade, do not ALTER Routine Load load clauses until all
FEs are running the new version. Rolling back after a new-version checkpoint
can restore the original CREATE clauses instead of post-upgrade ALTER semantics.
### Check List (For Author)
- Test
- [x] Regression test
- Added a three-FE Routine Load journal replay, leader failover,
checkpoint, and restart case.
- Not executed locally.
- [x] Unit Test
- 36 focused FE tests passed.
- FE compile, testCompile, and checkstyle passed.
- [ ] Manual test
- [ ] No need to test or manual test
- Behavior changed:
- [ ] No.
- [x] Yes. New images and ALTER journals retain the current effective
Routine Load definition; legacy images migrate once from `origStmt`.
- Does this need documentation?
- [ ] No.
- [x] Yes. Rolling-upgrade and rollback limitations must be documented.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
