owenchenxy opened a new issue, #69724: URL: https://github.com/apache/doris/issues/69724
### Search before asking - [x] I had searched in the issues and found no similar issues. ### Version `doris-4.1.5-release-917b20be838` (both FE and BE; reported by `SELECT @@version_comment`) ### What happens With single-replica load enabled on both sides: - FE: `experimental_enable_single_replica_load = true` - BE: `enable_single_replica_load = true` **every** Stream Load into a `replication_num = 3` table hangs forever. The client receives `HTTP/1.1 100 Continue`, then no response at all (no headers, no body) until the client timeout. The table stays at 0 rows. ### Minimal discrimination (same cluster, same credentials) | Target | Result | | --- | --- | | table with `replication_num = 1` | **Success**, 1 row, `LoadTimeMs=138` | | table with `replication_num = 3` | **Hangs** (100-continue then silence) | | `INSERT INTO ... VALUES` (incl. AUTO_INCREMENT table) | **Success** (thrift path) | So the failure is specific to the multi-replica (BE-to-BE) load data path, not to auth, schema, partitions or the tables themselves. ### Logs FE begins the transaction normally: ``` begin transaction: txn id 158382 with label probe-... from coordinator BE: 10.201.160.204 ``` Coordinator BE opens the channel and then the channel is closed with a pending batch: ``` new income streaming load request ... db=sims_nonprod_logs, tbl=test_noai begin to execute stream load ... txn_id=158382 VNodeChannel[...], load_id=..., txn_id=158382, node=10.201.160.218:8060 mark closed, left pending batch size: 1 hang_wait: 0 All node channels are stopped(maybe finished/offending/cancelled), sender thread exit. ``` Peer BE (`10.201.160.218`) has **no log at all** for that txn id - the data never arrives. Some loads eventually surface: ``` failed to send brpc batch, error=RPC call is timed out, error_text=[E1008]Reached timeout=599998ms @10.201.160.218:8060 fragment execute failed, err_msg=[INTERNAL_ERROR]rpc failed, error code:1008 ``` ### Ruled out - Network: TCP `22/8030/8040/8060/9060` reachable pairwise between all 3 BEs; brpc `/status` returns 200 in ~2ms from every peer; MTU verified with `ping -M do -s 8972` (jumbo frames pass). - Thread pools: after a full restart there are no `fail to offer request to the work pool` entries; `heavy_work_active_threads=0`, queue 0. - Metadata: running transaction count is 0 for the affected DBs; `ADMIN SHOW REPLICA STATUS` all `NORMAL/OK`; `SHOW PROC '/backends'` all `Alive=true`, `isLoadDisabled=false`; today's partitions exist (`VisibleVersion=1`). - FE: MySQL port and thrift fine, `beginTransaction` logs normal. - Restarting all 3 BE processes (SIGTERM did not even terminate the old ones - they ignored it and needed SIGKILL) does **not** fix it. ### Config matrix (same table, same cluster) | FE `experimental_enable_single_replica_load` | BE `enable_single_replica_load` | Result | | --- | --- | --- | | true | true | hangs forever | | true | false | fast failure: `[INTERNAL_ERROR]single replica load is disabled on BE.` | | false | false | **works**, `LoadTimeMs=39` | ### Regression vs 4.1.4 The **identical** configuration (FE `experimental_enable_single_replica_load=true`, BE `enable_single_replica_load=true`) works on `doris-4.1.4-rc04-ad35a140c7f` (a sibling cluster on the same platform, hundreds of millions of rows ingested). So this looks like a 4.1.5 regression rather than a general property of single replica load. ### Possibly related - #64708 BE write path wedges: a BE-to-BE load-stream brpc socket (8060) goes "Broken" and is never revived - #68973 Stream Load blocks its libevent worker in on_header / full pipe - #65582 INSERT OVERWRITE ... PARTITION(*) hangs forever due to a receiver-side load-stream close race ### Expectation Either the single-replica-load path should not hang for `replication_num >= 2` stream load in 4.1.5, or stream load should reject/ignore `experimental_enable_single_replica_load` with a clear error instead of hanging indefinitely. Happy to provide BE/FE logs, metrics, pstack and exact reproduce steps. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
