zhijunn opened a new issue, #69509: URL: https://github.com/apache/doris/issues/69509
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version Doris 2.1.11-rc01 (97b77e6cda), production and test clusters run the same build, 3 FE + 5 BE for test clusters. ### What's Wrong? `fetch_result()` reads the queue status once, **before** `blocking_get()`, and never again: ```cpp RETURN_IF_ERROR(queue->status()); // the only check bool success = queue->blocking_get(result); if (success && *result == nullptr) { *eos = true; queue->blocking_put(nullptr); } else if (!success) { *eos = true; } return Status::OK(); // OK on every path ``` What wakes a blocked waiter is the `nullptr` sentinel pushed by `MemoryScratchSink::close()`, which does not look at `exec_status`. A failed or cancelled fragment therefore reaches the client as a normal end-of-stream. Two version shapes: | Version | Is the fragment error recorded in the queue? | Client outcome | |---|---|---| | 2.1.x | yes (`PlanFragmentExecutor::cancel()`, failed `open()`), but never re-checked | race: silent only if the client is blocked at cancellation time | | 3.0 / 4.0 / master | **no** — external scan runs on the pipeline engine (`TPipelineFragmentParams`) and `PipelineFragmentContext::cancel()` has no `update_queue_status()` | every cancellation during the scan is silent | A second silent path: `ResultQueueMgr::cancel()` (scan-context close or GC) shuts the queue down and erases the map entry without a status, so a blocked caller gets `blocking_get() == false`, which becomes `eos = true` + `OK`. Production impact: on 2026-10-16 a Spark job reading a 168M-row dimension table received `OK` + `eos=true` from a fragment cancelled mid-scan. Job, write and export all reported success, and 7,936 rows of a join key were written as NULL. Two attempts on the same tablet, one second apart: ``` attempt 0 (started 13:46:23): get_next → CANCELLED 13:46:27 → task FAILED (loud) attempt 1 (started 13:46:28): get_next → OK, eos=true 13:46:32 → task SUCCESS (truncated) ``` `fetch_result()` has one caller, `BackendService::get_next`, so the affected surface is the thrift external-scan protocol (the Spark connector and any other client of that API). ### What You Expected? - A failed or cancelled fragment reaches the client as a non-OK status on every path, including when the queue is shut down. - `fetch_result()` never returns `OK` + `eos=true` while the fragment's terminal state is an error. - A client already blocked when the status is set observes it on wake-up, not on its next call. ### How to Reproduce? 1) Deterministic unit test, no cluster (add to `be/test/runtime/result_queue_mgr_test.cpp`; not executed by me — I did not build BE from source): ```cpp std::thread fetcher([&] { fetch_status = queue_mgr.fetch_result(id, &batch, &eos); }); sleep(200ms); // fetcher blocked queue_mgr.update_queue_status(id, Status::Cancelled("cancelled")); queue->blocking_put(nullptr); // what the sink does fetcher.join(); EXPECT_FALSE(fetch_status.ok()); // current code: OK, eos = true ``` 2) End-to-end on 2.1.11 with `doris.request.query.timeout.s` = 1 s, a 500M-row single-bucket table and a predicate matching 5 rows: 22 runs gave 9 silent short reads (job SUCCESS, zero errors, 4 or 2 rows against a baseline of 5), 6 full reads and 7 unrelated connect failures — **60% of the successful runs were silent**. 3) Under memory and disk pressure, production parameters unchanged: 154,616,498 rows read against a 156,170,436 baseline, 1,553,938 missing, job SUCCESS, no error, no timeout. `be.INFO` at that moment: ``` [MemoryGC] ... seek 103 tasks. among them, 103 tasks will be canceled ... tablet=13358 res=[CANCELLED] ``` ### Anything Else? **Suggested fix** (proposed, not built or run): 1. `MemoryScratchSink(LocalState)::close()`: `if (!exec_status.ok()) _queue->update_status(exec_status);` before the sentinel — the only place that still holds the terminal status on 3.0+. 2. `fetch_result()`: re-check `queue->status()` after `blocking_get()`; treat `!success` (queue shut down) as a failure, not as eos. 3. `ResultQueueMgr::cancel()`: record a status before `shutdown()`, or expose "shut down" on `RecordBatchQueue`. 4. Optional: restore the queue notification in the pipeline cancel path (it disappeared in the pipeline rewrite and is still commented on 2.1.x as `TODO pipeline incomp`). **Relation to #67604.** It implements 1 and 2 for the sentinel path. This issue is separate because the defect had no issue at all, because the shut-down path above stays silent after that PR, and because 2.1/3.0/4.0 need the change backported (`fetch_result()` is identical on all four, so 2.1.x needs only item 2 — one hunk). **Related, not different.** #64172 (an operator sets `eos` early and drops rows), #65762, #63609, #67503; connector `#314` (an unchanged table counted 68,684 / 97,918 / 99,903 on three runs) and `#42421`, which fixed a *cause of cancellation*, not the swallowing. **Not verified, and why.** The production cancellation trigger is not recovered (the BE log was rotated; it is inferred from the reproduction). Nothing here is validated at the C++ level: the unit test above was not executed and the patch was not compiled, for the reason given in the last field below. The evidence therefore comes from unmodified 2.1.11 clusters and from source reading. I also did not grep the whole tree for `update_queue_status()`; the files in the cancel/close path were read one by one. **How the analysis was done.** The anomaly was found by data reconciliation. The source analysis, the reproduction harness and the draft patch were produced with an AI coding assistant on a checkout of tag `2.1.11-rc01`, then checked by a human against the source and against the logs; the assistant had no access to the production cluster. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
