hanke580 opened a new issue, #69412: URL: https://github.com/apache/doris/issues/69412
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version - master 15eb3f4e432b1d4a39f89457956135da80b90940 (2026-10-18), built from source (`build.sh --fe --be`), 1 FE + 1 BE, `replication_num = 1` - 4.1.5 (build 917b20be8): same result ### What's Wrong? A schema change that creates new tablets (`ALTER TABLE ... ORDER BY`, `MODIFY COLUMN`) on a merge-on-write UNIQUE KEY table can deadlock with ordinary loads: - The ALTER job waits in `WAITING_TXN` for the transactions that started before it. - One of those transactions commits after a load that started once the job was in WAITING_TXN (that load is double-written to the shadow tablet). The older transaction gets a partition version that has no rowset on the shadow tablet. - The BE then refuses to publish the next double-written version on the shadow (`TABLET_NOTREADY`) tablet: `[E-3115] version not continuous for mow`. Versions become visible in order, so the older transaction stays COMMITTED behind it, and the ALTER job keeps waiting for that transaction. Nothing breaks the cycle automatically. The job timeout is 30 days (2592000 s). Loads that were acknowledged as committed stay invisible, and so does every later load into the partition (a further INSERT also ended COMMITTED and stayed invisible). The state survives an FE and BE restart. `CANCEL ALTER TABLE COLUMN` releases everything at once. With the same steps on a merge-on-read table the job finishes, and so do the steps without the ALTER. ### What You Expected? A load that returns COMMITTED becomes visible eventually, and the schema change finishes (or fails with an error). Concurrent loads should not hang the partition. ### How to Reproduce? Three MySQL sessions (S, A1, A2), run in this order: ```sql -- S CREATE DATABASE scdl; USE scdl; CREATE TABLE t (k INT, a INT, b VARCHAR(16)) UNIQUE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ('replication_num' = '1', 'enable_unique_key_merge_on_write' = 'true'); INSERT INTO t VALUES (1, 1, 'x'), (2, 2, 'y'); -- A1 USE scdl; BEGIN; INSERT INTO t VALUES (101, 101, 'a1'); -- A2 USE scdl; BEGIN; INSERT INTO t VALUES (102, 102, 'a2'); -- S ALTER TABLE t ORDER BY (k, b, a); -- or: ALTER TABLE t MODIFY COLUMN a BIGINT SHOW ALTER TABLE COLUMN WHERE TableName = 't'; -- repeat until State = WAITING_TXN (well under a second) INSERT INTO t VALUES (201, 201, 'b'); -- B: started after the watershed, double-written -- A1 COMMIT; -- S INSERT INTO t VALUES (202, 202, 'c'); -- C -- A2 COMMIT; -- S, any time later SELECT k FROM t ORDER BY k; SHOW ALTER TABLE COLUMN WHERE TableName = 't'; ``` Observed on master: ``` S> SHOW ALTER TABLE COLUMN -> State WAITING_TXN, TransactionId 178 S> INSERT (201) -> {'status':'VISIBLE', 'txnId':'180'} (0.2s) A1> COMMIT -> {'status':'VISIBLE', 'txnId':'171'} (1.0s) S> INSERT (202) -> {'status':'COMMITTED', 'txnId':'183'} (60.1s) A2> COMMIT -> ERROR 1105: transaction commit successfully, BUT data will be visible later. (30.0s) SELECT k FROM t ORDER BY k; -> 1, 2, 101, 201 (102 and 202 are missing) SHOW ALTER TABLE COLUMN -> ... TransactionId 178, State WAITING_TXN, Timeout 2592000 SHOW TRANSACTION WHERE id = 172 (A2) -> COMMITTED "wait for publishing partition 1792376644359 version 5. self version: 6. table 1792376644360 ..." SHOW TRANSACTION WHERE id = 183 (C) -> COMMITTED "publish on tablet 1792376644382 failed. succeed replica num 0 < load required replica num 1. table: 1792376644360, partition: 1792376644359, publish version: 5" ``` BE log, repeating every few seconds: ``` W20261019 02:59:59.002902 481652 task_worker_pool.cpp:2188] failed to publish version|signature=183|transaction_id=183|error_tablets_num=1|error=[E-3115]version not continuous for mow, tablet_id=1792376644382, tablet_max_version=3, txn_version=5, commit_tso=-1 ``` If B starts before the job reaches WAITING_TXN, B gets a transaction id below the watershed, nothing is double-written, and the job finishes normally. After restarting the FE and the BE (about 10 minutes after the steps), nothing had changed (job WAITING_TXN, keys 1, 2, 101, 201). `CANCEL ALTER TABLE COLUMN FROM t` makes both transactions VISIBLE at once, and all keys appear. The same steps on a merge-on-read table: all six keys are visible and the job is FINISHED. ### Anything Else? Likely cause (master 15eb3f4e43): - BE, `be/src/storage/task/engine_publish_version_task.cpp:224-233`: for a merge-on-write tablet in `TABLET_NOTREADY`, a version is not published when the versions below it are not continuous (`_handle_publish_version_not_continuous`, `:332`, E-3115). The shadow tablet holds only the double-written versions of loads that started after the ALTER (B = v3). A1 started before the ALTER, so its version v4 has no rowset on the shadow tablet. C's version v5 can never be published there until the history conversion fills in v4. - FE, `fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java:3175` (`checkQuorumReplicas`): the shadow tablet's replica has to succeed, so C stays COMMITTED. `:1517`: versions become visible in order, so A2 (v6) waits for C (v5). - FE, `fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java:478` (`runWaitingTxnJob`) and `:900` (`checkFailedPreviousLoadAndAbort`): the job waits until every transaction below the watershed is finished, including A2. The history conversion therefore never starts. `enable_auto_clone_on_mow_publish_missing_version` (BE, default false) and `mow_publish_max_discontinuous_version_num` do not apply here: the missing version does not exist on any replica of the shadow tablet. The FE config `publish_version_check_alter_replica` (default true; its description says to turn it off only temporarily "if publish version and schema change encounter a deadlock") is a manual way out: `ADMIN SET FRONTEND CONFIG ('publish_version_check_alter_replica' = 'false')` on a stuck run made C and A2 VISIBLE, and the job FINISHED with all six rows. (On runs where the FE and the BE had been restarted while the job waited, the released jobs failed instead: `schema change tasks failed, error reason: task exec error: Required field 'now_string' was not present! Struct: TQueryGlobals(now_string:null)`; the rows became visible.) It was added by #26428 for a similar cycle (a transaction below the watershed that published to the new replica after an FE restart). Here the blocked transaction C is above the watershed, so the FE checks the shadow replica, and the BE refuses the publish because of the gap left by A1. Related but different: #60953 (version-gap replicas excluded from quorum counting, no schema change). #41527 and #28634 (E-3115 under concurrent or Spark writes, no schema change). #63443 (merged; cloud mode, fills version holes after the alter version before the new tablet becomes RUNNING). #60746 and #29708 (alter jobs stuck in WAITING_TXN for other reasons: an exception in IndexChangeJob, a PREPARE transaction left open for days). #66177 (stuck in WAITING_STABLE after cancel). #30834 (closed, not merged: publish should not check the new tablets of a schema change, in regression tests). #67247 (open: reserve the watershed before exposing the shadow indexes, for transactions above the watershed that miss the shadow index; here A1 and A2 are below the watershed). ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
