[
https://issues.apache.org/jira/browse/SPARK-60931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60931:
-----------------------------------
Labels: pull-request-available (was: )
> Batch TransformWithState resolves an executor-local RocksDB path on HDFS
> ------------------------------------------------------------------------
>
> Key: SPARK-60931
> URL: https://issues.apache.org/jira/browse/SPARK-60931
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 5.0.0
> Reporter: Adam Binford
> Priority: Major
> Labels: pull-request-available
>
> h3. Description
> Batch {{TransformWithState}} fails when Hadoop's {{fs.defaultFS}} is HDFS.
> The batch operator creates an executor-local temporary directory and uses its
> bare absolute path as the state store checkpoint root. RocksDB treats that
> path as its DFS root, resolves it through Hadoop's default filesystem, and
> attempts to create it on HDFS.
> h3. Regression
> This appears to regress with
> [SPARK-54984|https://issues.apache.org/jira/browse/SPARK-54984] (commit
> [acc80fd1b205|https://github.com/apache/spark/commit/acc80fd1b205792a43cc352a8e492e34bbe880da]),
> from [PR #53747|https://github.com/apache/spark/pull/53747]. The commit
> added root-directory creation to RocksDB version-0 loading. Batch
> {{TransformWithState}} initializes its store at version 0, so it now reaches
> that creation path. The corresponding call is absent from Spark 4.1's
> version-0 load path; batch completion aborts the store instead of committing
> it.
> The PR discussion says this handles newly repartitioned partitions with
> missing checkpoint directories during {{loadEmpty}}, but does not discuss the
> HDFS default-filesystem effect or batch {{TransformWithState}}: [review
> response|https://github.com/apache/spark/pull/53747#discussion_r2677386642].
> h3. Reproduction
> Configure Hadoop with {{fs.defaultFS}} pointing to HDFS, then run
> {{TransformWithState}} in batch mode. The executor-local temporary path is
> treated as an HDFS path, and directory creation fails.
> h3. Expected behavior
> Batch-local RocksDB temporary paths should be handled on the local filesystem.
> h3. Actual behavior
> Hadoop resolves the bare local path against {{fs.defaultFS}} and attempts to
> create it on HDFS.
> h3. GitHub issue
> https://github.com/apache/spark/issues/60097
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
