lide-reed opened a new issue, #67997: URL: https://github.com/apache/doris/issues/67997
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.3 ### What's Wrong? Description / What happened RuntimeFilter::serialize() marks the outgoing filter as disabled for any state that is not READY: // be/src/exec/runtime_filter/runtime_filter.h auto state = _wrapper->get_state(); if (state != RuntimeFilterWrapper::State::READY) { request->set_disabled(true); // <-- UNINITED is treated as DISABLED here return Status::OK(); } UNINITED does not mean "this filter was cancelled", it means "no filter content has been produced yet". The consumer, however, cannot tell the difference: // be/src/exec/runtime_filter/runtime_filter_wrapper.cpp (RuntimeFilterWrapper::signal) if (other->_wrapper->is_disabled()) { _wrapper->set_state(RuntimeFilterWrapper::State::DISABLED, "get disabled from remote"); } DISABLED is an absorbing state. Once it is set the consumer permanently gives up the filter, and the probe side runs with no runtime filter at all -> full scan of the (large) probe table. So a filter that was simply not produced yet is turned into a filter that is permanently cancelled. Root cause: readiness is counted per producer, not per content RuntimeFilterMerger::merge_from() decides readiness only by the number of producers that have reported: // be/src/exec/runtime_filter/runtime_filter_merger.h *ready = _received_producer_num == _expected_producer_num; if (_received_producer_num == _expected_producer_num) { _rf_state = State::READY; // merger's own flag, NOT the wrapper state } if (_wrapper->get_state() == RuntimeFilterWrapper::State::UNINITED) { _wrapper = other->_wrapper; // adopt; may still be UNINITED return Status::OK(); } The merger becomes "ready" as soon as the producer count is reached, independently of whether any producer actually produced filter content. If all producers finish without producing content (e.g. the build side yields no rows, or producers are finalized early), the adopted _wrapper is still UNINITED while merge_from() has already returned ready == true. The caller then publishes the filter, serialize() sees state != READY, and broadcasts disabled=true to every consumer. Affected code paths RuntimeFilterProducer::publish() -> do_merge -> _send_to_local_targets() / _send_to_remote_targets() RuntimeFilterMergeControllerEntity::_send_rf_to_target() (broadcast join / merge controller path) Impact Sudden, non-deterministic performance regression on join queries: the probe side (usually the big fact table) performs a full scan instead of being filtered, and query latency can regress by an order of magnitude. Silent: the query still succeeds with correct results, so the regression is only visible in profiles / latency, not as an error. Timing dependent (whether producers report before or after producing content), which makes it hard to reproduce deterministically and hard to diagnose in production. ### What You Expected? Align with the legacy (branch-3.1) runtime filter semantics: Only a really disabled filter (hit max_in_num, join spill, RPC error -> State::DISABLED) should be published as disabled. A filter that is merely not ready (UNINITED) must not be published as disabled. It should be skipped so that consumers keep waiting until runtime_filter_wait_time_ms and then time out normally, instead of being told "disabled from remote". ### How to Reproduce? Not deterministic (timing dependent). Conceptual steps: Run a join whose runtime filter is pushed down to a large probe table. Make the build side produce no filter content, so all producers report to the merger without any content. The merger reaches the expected producer count with _wrapper still UNINITED, and publishes the filter. Observe in the query profile that the filter state is DISABLED with reason get disabled from remote, while nothing ever really disabled it, and that the probe side no longer applies the filter (full scan). ### Anything Else? _No response_ ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
