Adam Binford created SPARK-60931:
------------------------------------
Summary: Batch TransformWithState resolves an executor-local
RocksDB path on HDFS
Key: SPARK-60931
URL: https://issues.apache.org/jira/browse/SPARK-60931
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Adam Binford
h3. Description
Batch {{TransformWithState}} fails when Hadoop's {{fs.defaultFS}} is HDFS. The
batch operator creates an executor-local temporary directory and uses its bare
absolute path as the state store checkpoint root. RocksDB treats that path as
its DFS root, resolves it through Hadoop's default filesystem, and attempts to
create it on HDFS.
h3. Regression
This appears to regress with
[SPARK-54984|https://issues.apache.org/jira/browse/SPARK-54984] (commit
[acc80fd1b205|https://github.com/apache/spark/commit/acc80fd1b205792a43cc352a8e492e34bbe880da]),
from [PR #53747|https://github.com/apache/spark/pull/53747]. The commit added
root-directory creation to RocksDB version-0 loading. Batch
{{TransformWithState}} initializes its store at version 0, so it now reaches
that creation path. The corresponding call is absent from Spark 4.1's version-0
load path; batch completion aborts the store instead of committing it.
The PR discussion says this handles newly repartitioned partitions with missing
checkpoint directories during {{loadEmpty}}, but does not discuss the HDFS
default-filesystem effect or batch {{TransformWithState}}: [review
response|https://github.com/apache/spark/pull/53747#discussion_r2677386642].
h3. Reproduction
Configure Hadoop with {{fs.defaultFS}} pointing to HDFS, then run
{{TransformWithState}} in batch mode. The executor-local temporary path is
treated as an HDFS path, and directory creation fails.
h3. Expected behavior
Batch-local RocksDB temporary paths should be handled on the local filesystem.
h3. Actual behavior
Hadoop resolves the bare local path against {{fs.defaultFS}} and attempts to
create it on HDFS.
h3. GitHub issue
https://github.com/apache/spark/issues/60097
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
