GJ100 opened a new issue, #68169: URL: https://github.com/apache/doris/issues/68169
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 3.0.x / 4.1.x / master ### What's Wrong? When a client application (e.g., a microservice or web backend with a 3–5 second query timeout) disconnects due to client-side timeout while the query is still waiting on the BE (e.g. waiting for rowset/delete-bitmap lock or waiting in queue during heavy ingestion/compaction), the following cascade failure occurs: 1. **FE Network Layer Removes Connection Silently**: FE's `AcceptListener` detects the TCP FIN/RST from the client and triggers `connection.setCloseListener(...)` -> `connectScheduler.getConnectPoolMgr().unregisterConnection(context)`. The connection is removed from `connectionMap`. 2. **No Cancel Signal Is Dispatched**: `ConnectPoolMgr.unregisterConnection()` removes the context from `connectionMap`, but **does NOT call `context.cancelQuery()`**. No cancel signal is sent to the `Coordinator`, and no `cancel_plan_fragment` RPC is dispatched to BE nodes. 3. **Query Escapes `TimeoutChecker` Entirely (Ghost Query)**: FE's background `TimeoutChecker` (`checkTimer`) iterates strictly over `connectionMap.values()`. Because the connection was already removed in step 1, `checkTimeout()` is **never called again** for this context. The query bypasses `query_timeout` (e.g. 300s) and hangs indefinitely in `QeProcessorImpl` / `information_schema.active_queries` (observed running for >3400 seconds / 57 minutes as `RUNNING`). 4. **FE Worker Thread Stalls in `coordBase.getNext()`**: Because the query was actively executing and `ReadListener.suspendAcceptQuery()` had already suspended reading on the socket, the worker thread remains blocked waiting for BE results. Since no data is written to the closed socket, no `IOException` / `EPIPE` is raised to break the loop. 5. **Workload Group Queue Slot Leak & Cluster Stall**: Because `Coordinator.close()` is never executed, the query's `QueueToken` is never returned to the `QueryQueue`. When all slots (`max_concurrency`) in the workload group are occupied by these orphaned queries, **all subsequent queries in that workload group are stuck in `WAIT_IN_QUEUE` forever**, until they fail with `query queue timeout`. ### What You Expected? When a client disconnects or closes the connection: 1. `unregisterConnection()` must immediately cancel any active query on that connection. 2. The `Coordinator` must abort BE fragment execution via cancel RPCs, release the workload group's `QueueToken`, and unblock the worker thread. 3. The query must be immediately removed from `QeProcessorImpl` and `information_schema.active_queries`. 4. Workload group slots must be promptly freed for queued queries. ### How to Reproduce? 1. Create a workload group with strict concurrency: ```sql CREATE WORKLOAD GROUP wg_test PROPERTIES ('max_concurrency'='1', 'max_queue_size'='10', 'queue_timeout'='60000'); ``` 2. Assign `wg_test` to user `test`. 3. Submit a query that takes several seconds (or simulate a lock wait / slow scan on BE). 4. Abruptly kill the client process (send TCP FIN/RST) within 2 seconds before any result row is returned. 5. Inspect `information_schema.active_queries`: - The query remains `RUNNING` long after `query_timeout`. 6. Submit a second query from a new connection: - The new query stays in `WAIT_IN_QUEUE` and eventually times out. ### Anything Else? Observed in production where point-lookups (`SELECT ... FROM tbl WHERE user_id = ...`) encountering lock wait were abandoned by client microservices after 3s. The orphaned queries lived for >57 minutes in `active_queries` while subsequent queries piled up in `WAIT_IN_QUEUE`. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
