haoyangeng-db opened a new pull request, #58435: URL: https://github.com/apache/spark/pull/58435
### What changes were proposed in this pull request? This PR makes `SparkConnectClient.execute_command_as_iterator` yield every decoded response item instead of requiring every item to be a dictionary. Streaming-query listener and pipeline consumers ignore `PlanMetrics` and `ObservedMetrics` while continuing to reject unrelated malformed responses. It also creates Python Data Source profiler accumulators only when performance or memory profiling is enabled. ### Why are the changes needed? Python Data Source workers currently register profiler accumulators even when profiling is disabled. Their empty accumulator updates can arrive alongside command results as observed metrics. The Spark Connect command iterator treats those valid non-dictionary responses as unknown and terminates long-lived command streams, including streaming listeners. The low-level iterator already decodes several legitimate response types, so it should preserve them and let each command-specific consumer decide which types it accepts. ### Does this PR introduce _any_ user-facing change? Yes. Spark Connect command streams no longer fail when valid execution metrics accompany command results. Python Data Source workers also stop sending empty profiler updates when profiling is disabled. Unexpected response shapes still raise an error. ### How was this patch tested? Added focused tests covering: - command results followed by plan and observed metrics; - protobuf extension responses; - streaming-listener metadata before and after registration; - malformed listener and pipeline responses; - disabled, unsupported, performance, and memory profiler modes; - preservation of ordinary accumulator updates. All 7 focused tests and all 44 tests in the mocked Spark Connect client suite passed. Ruff check/format and `git diff --check` passed. A full SBT build could not resolve external dependencies in this environment because DNS was unavailable. ### Was this patch authored or co-authored using generative AI tooling? Co-authored with Codex -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
