[
https://issues.apache.org/jira/browse/SPARK-60610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60610:
-----------------------------------
Labels: pull-request-available (was: )
> FileSourceScanExec.bucketedScan re-reads spark.sql.sources.bucketing.enabled
> on the copy made at stage creation
> ---------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60610
> URL: https://issues.apache.org/jira/browse/SPARK-60610
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.1.0
> Reporter: Naveen Kumar Puppala
> Priority: Major
> Labels: pull-request-available
>
> {{FileSourceScanExec.bucketedScan}} is a lazy val that reads the live session
> config via {{{}getSqlConf(relation.sparkSession).bucketingEnabled{}}}, and
> {{outputPartitioning}} depends on it. When the node is copied after planning
> (for example the copy {{PlanAdaptiveDynamicPruningFilters}} makes at stage
> creation), the lazy val re-evaluates on the copy and reads the config again.
> If {{spark.sql.sources.bucketing.enabled}} is turned off between planning and
> execution, the rebuilt scan reports a different output partitioning than the
> plan was built with.
> This is the V1 file-source analogue of SPARK-60260 (DataSource V2
> storage-partitioned join) and SPARK-59252. Found by inspection during the
> SPARK-60260 review ([apache/spark#59789 · [SPARK-60260][SQL] Keep
> storage…|https://github.com/apache/spark/pull/59789], Peter Toth). A
> reproduction confirming the resulting wrong results or failure is still to be
> written.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
