[
https://issues.apache.org/jira/browse/SPARK-38015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-38015:
------------------------------------
Assignee: Apache Spark
> Mark legacy file naming functions as deprecated in FileCommitProtocol
> ---------------------------------------------------------------------
>
> Key: SPARK-38015
> URL: https://issues.apache.org/jira/browse/SPARK-38015
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 3.3.0
> Reporter: Cheng Su
> Assignee: Apache Spark
> Priority: Minor
>
> [FileCommitProtocol|https://github.com/apache/spark/blob/6bbfb45ffe75aa6c27a7bf3c3385a596637d1822/core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala]
> is the class to commit Spark job output (staging file & directory renaming,
> etc). During Spark 3.2 development, we added new functions into this class to
> allow more flexible output file naming (the PR detail is
> [here|https://github.com/apache/spark/pull/33012]). We didn’t delete the
> existing file naming functions (newTaskTempFile(ext) &
> newTaskTempFileAbsPath(ext)), because we were aware of many other downstream
> projects or codebases already implemented their own custom implementation for
> FileCommitProtocol. Delete the existing functions would be a breaking change
> for them when upgrading Spark version, and we would like to avoid this
> unpleasant surprise for anyone if possible. But we also need to clean up
> legacy as we evolve our codebase.
> So for next step, I would like to propose:
> * Spark 3.3 (now): Add @deprecate annotation to legacy functions in
> FileCommitProtocol -
> [newTaskTempFile(ext)|https://github.com/apache/spark/blob/6bbfb45ffe75aa6c27a7bf3c3385a596637d1822/core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala#L98]
> &
> [newTaskTempFileAbsPath(ext)|https://github.com/apache/spark/blob/6bbfb45ffe75aa6c27a7bf3c3385a596637d1822/core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala#L135].
> * Next Spark major release (or whenever people feel comfortable): delete the
> legacy functions mentioned above from our codebase.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
