Gengliang Wang created SPARK-62035:
--------------------------------------

             Summary: Add spark.sql.execution.maxScanBytes to cancel queries 
that read too much data
                 Key: SPARK-62035
                 URL: https://issues.apache.org/jira/browse/SPARK-62035
             Project: Spark
          Issue Type: New Feature
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Gengliang Wang


A query with a missing filter can read terabytes of data before anyone notices. 
A timeout does not help: it fires only after the compute has been spent, and 
the query returns nothing.

This adds an optional setting, off by default:

spark.sql.execution.maxScanBytes: the most bytes a query may read from its data 
sources.

A query that reads more is cancelled and fails with SCAN_BYTES_LIMIT_EXCEEDED. 
This is similar to maximum_bytes_billed in BigQuery and 
query_max_scan_physical_bytes in Trino.

The limit is a best-effort guard rather than a hard cap:
* Spark learns how much the tasks of a query have read from the executor 
heartbeats and when a task ends, so a query can read more than the limit before 
it is cancelled.
* Only data sources that report the bytes they read are limited. File-based 
sources report the bytes they read from storage, while sources such as JDBC 
report none.
* Reading cached data counts the size of the cached blocks.
* Streaming queries are not limited.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to