Shixiong Zhu created SPARK-28605:
------------------------------------
Summary: Performance regression in SS's foreach
Key: SPARK-28605
URL: https://issues.apache.org/jira/browse/SPARK-28605
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 2.4.3
Reporter: Shixiong Zhu
When "ForeachWriter.open" return "false", ForeachSink v1 will skip the whole
partition without reading data. But in ForeachSink v2, due to the API
limitation, it needs to read the whole partition even if all data just gets
dropped.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
