cloud-fan commented on PR #59333:
URL: https://github.com/apache/spark/pull/59333#issuecomment-6210825735
Agreed, the fix looks right to me. Before SPARK-56919, FileFormatWriter set
spark.sql.sources.writeJobUUID before writeAndCommit called setupJob.
SPARK-56919 moved setupJob ahead of materializeAdaptiveSparkPlan but left the
UUID assignment inside the try. Since then, setupJob (and abortJob after a
materialization failure) runs without the key. This PR moves the assignment
back ahead of setupJob, the same order SparkHadoopWriter uses, and the value is
still description.uuid. On severity: in Hadoop 3.5.0, which Spark builds
against, mapred-default.xml maps abfs and gs to the manifest committer
factories, and ManifestCommitter derives its job UUID from this key in
setupJob. So in 4.2.0, file writes to abfs:// or gs:// that use the scheme's
default Hadoop committer fail at setupJob. Parquet's default
ParquetOutputCommitter and the streaming sink's default
ManifestFileCommitProtocol don't read the key, so not every write is affected,
but this is still a common cloud path. SPARK-56919 i
s on branch-4.2, branch-4.3 and branch-4.x, so this fix should be backported
to the same branches.
<!-- SPARK_DEV_REVIEW_REPLY
{"feedback_id":"conversation:6159560585","thread_id":"conversation:6159560585","verdict_sha256":"94a57059c643b0b8a0e34e070ad814bdc72a190f9c60952a1c723c442a67e310"}
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
