Gengliang Wang created SPARK-62035:
--------------------------------------
Summary: Add spark.sql.execution.maxScanBytes to cancel queries
that read too much data
Key: SPARK-62035
URL: https://issues.apache.org/jira/browse/SPARK-62035
Project: Spark
Issue Type: New Feature
Components: SQL
Affects Versions: 5.0.0
Reporter: Gengliang Wang
A query with a missing filter can read terabytes of data before anyone notices.
A timeout does not help: it fires only after the compute has been spent, and
the query returns nothing.
This adds an optional setting, off by default:
spark.sql.execution.maxScanBytes: the most bytes a query may read from its data
sources.
A query that reads more is cancelled and fails with SCAN_BYTES_LIMIT_EXCEEDED.
This is similar to maximum_bytes_billed in BigQuery and
query_max_scan_physical_bytes in Trino.
The limit is a best-effort guard rather than a hard cap:
* Spark learns how much the tasks of a query have read from the executor
heartbeats and when a task ends, so a query can read more than the limit before
it is cancelled.
* Only data sources that report the bytes they read are limited. File-based
sources report the bytes they read from storage, while sources such as JDBC
report none.
* Reading cached data counts the size of the cached blocks.
* Streaming queries are not limited.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]