[
https://issues.apache.org/jira/browse/SPARK-62035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-62035:
-----------------------------------
Labels: pull-request-available (was: )
> Add spark.sql.execution.maxScanBytes to cancel queries that read too much data
> ------------------------------------------------------------------------------
>
> Key: SPARK-62035
> URL: https://issues.apache.org/jira/browse/SPARK-62035
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Gengliang Wang
> Priority: Major
> Labels: pull-request-available
>
> A query with a missing filter can read terabytes of data before anyone
> notices. A timeout does not help: it fires only after the compute has been
> spent, and the query returns nothing.
> This adds an optional setting, off by default:
> spark.sql.execution.maxScanBytes: the most bytes a query may read from its
> data sources.
> A query that reads more is cancelled and fails with
> SCAN_BYTES_LIMIT_EXCEEDED. This is similar to maximum_bytes_billed in
> BigQuery and query_max_scan_physical_bytes in Trino.
> The limit is a best-effort guard rather than a hard cap:
> * Spark learns how much the tasks of a query have read from the executor
> heartbeats and when a task ends, so a query can read more than the limit
> before it is cancelled.
> * Only data sources that report the bytes they read are limited. File-based
> sources report the bytes they read from storage, while sources such as JDBC
> report none.
> * Reading cached data counts the size of the cached blocks.
> * Streaming queries are not limited.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]