Doris-Breakwater commented on issue #66177: URL: https://github.com/apache/doris/issues/66177#issuecomment-5104112515
Breakwater-GitHub-Analysis-Slot: slot_e7b2388728fd This content is generated by AI for reference only. Initial assessment: **valid, high-confidence FE metadata-state bug**. I checked the live issue metadata and the relevant code on current upstream `master` (`2215dc476a1f9a1f6edba8c7a056e450165f7001`). The issue is open and currently has no labels, assignee, milestone, or comments. PR #66178 is linked and open; it has no reviews yet and its CI is still pending. The affected files are unchanged between upstream `master` and the local checkout used for inspection. The reported state transition is deterministic: - A schema-change job is created in `PENDING` while the table is initially set to `SCHEMA_CHANGE`. - `AlterJobV2.checkTableStable()` changes the table to `WAITING_STABLE` and returns `false` when the table is unstable; the job itself remains `PENDING`. - `SchemaChangeJobV2.cancelImpl()` runs `cancelInternal()`, then calls `changeTableState(..., NORMAL)`. That helper only performs the transition when the current table state is `SCHEMA_CHANGE`; it silently ignores `WAITING_STABLE`. - The job is nevertheless marked `CANCELLED` and journaled. The first `CANCEL ALTER TABLE COLUMN` therefore leaves the table in `WAITING_STABLE` with no unfinished schema-change job. - A subsequent cancel reaches `SchemaChangeHandler.cancelColumnJob()`, finds no unfinished job, and throws the reported “could not find related job” error. `OlapTable.checkNormalStateForAlter()` then rejects later ALTER, partition, rename, and truncate paths while the state remains non-`NORMAL`. Git history supports treating this as a regression. `WAITING_STABLE` was introduced as a temporary repair-permitting state, and cancellation while unstable was explicitly enabled later. A subsequent refactor moved table-state cleanup into `SchemaChangeJobV2`; the new helper retained only the `SCHEMA_CHANGE` predicate and omitted `WAITING_STABLE`. Linked PR assessment: **the direction is correct, but #66178 needs an ownership guard before merge**. - The PR's orphan branch checks only `SchemaChangeHandler.getUnfinishedAlterJobV2ByTableId()`. Because `WAITING_STABLE` is also used by `RollupJobV2`, `CANCEL ... COLUMN` against a table with a live unstable rollup job would find no schema-change job and durably force the table to `NORMAL` via `setTableStatusInternal()`. The rollup job is still live. The recovery path must also check `MaterializedViewHandler` (and preserve the state if any backing alter job exists). - The PR broadens the generic `SchemaChangeJobV2.changeTableState()` helper, but that helper is called after normal finish, cancel, replay-finish, and replay-cancel—not only the cancellation path. Allowing every caller to reset `WAITING_STABLE` can let a stale/replayed schema-change job reset a `WAITING_STABLE` currently owned by a rollup job. Limit the new source-state transition to cancellation/replay-cancellation with an ownership invariant, or make callers supply the expected source states. - The PR releases the table lock before calling `setTableStatusInternal()`, which performs an unconditional state change under a newly acquired lock. The durable recovery operation should re-check that the table is still `WAITING_STABLE` and still has no backing schema-change or rollup job at the point of mutation. Recommended fix/review direction: 1. Make cancellation of the live schema-change job transition its own table from `WAITING_STABLE` to `NORMAL`, without broadening unrelated finish/replay paths. 2. Self-heal only a genuinely orphaned `WAITING_STABLE`: verify that neither the schema-change handler nor the materialized-view/rollup handler has an unfinished job, and revalidate the state atomically when applying the durable reset. 3. Reconcile persistence semantics. `OlapTable` documents `WAITING_STABLE` as in-memory-only, but the `state` field is Gson-serialized and no `WAITING_STABLE` normalization exists in `gsonPostProcess()` or elsewhere in the current source. An image can therefore retain an orphaned state. Either enforce the intended non-persistence or reconcile table states against loaded alter jobs after metadata load. Required tests: - Extend `SchemaChangeJobV2Test.testSchemaChangeWhileTabletNotStable`: after forcing a replica to `DECOMMISSION`, assert table state `WAITING_STABLE`, cancel through `SchemaChangeHandler` (the user-facing path), and assert job state `CANCELLED` plus table state `NORMAL`. PR #66178 currently calls `schemaChangeJob.cancel()` directly. - Set a table to `WAITING_STABLE` with no unfinished alter job, run `CANCEL ALTER TABLE COLUMN`, and assert successful recovery to `NORMAL`. - Keep an unfinished rollup/MV job for a `WAITING_STABLE` table and verify the schema-change orphan-recovery path does not reset the table. - Add cancelled-job replay/image-load coverage so `WAITING_STABLE` cannot survive without a backing job. No additional logs, runtime profile, or reproduction details are required to confirm the bug on `master`; the current code establishes it directly. For deciding release-branch backports, the first affected release/commit should still be identified from branch history. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
