[ https://issues.apache.org/jira/browse/SPARK-38102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Keunhyun Oh updated SPARK-38102: -------------------------------- Description: There is no way to apply spark-hadoop-cloud's commitProtocolClass when using saveAsNewAPIHadoopDataset that are to avoid object storage's problem. [https://spark.apache.org/docs/latest/cloud-integration.html] It is needed to support custom commitProtocolClass class when using saveAsNewAPIHadoopDataset by an option. For example, {code:java} spark.hadoop.mapreduce.sources.commitProtocolClass org.apache.spark.internal.io.cloud.PathOutputCommitProtocol{code} was: It is needed to support custom commitProtocolClass class when using saveAsNewAPIHadoopDataset. It's because of no way to apply spark-hadoop-cloud's commitProtocolClass when using saveAsNewAPIHadoopDataset that are to avoid object storage's problem. > Supporting custom commitProtocolClass when using saveAsNewAPIHadoopDataset > -------------------------------------------------------------------------- > > Key: SPARK-38102 > URL: https://issues.apache.org/jira/browse/SPARK-38102 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 3.2.0 > Reporter: Keunhyun Oh > Priority: Major > > There is no way to apply spark-hadoop-cloud's commitProtocolClass when using > saveAsNewAPIHadoopDataset that are to avoid object storage's problem. > [https://spark.apache.org/docs/latest/cloud-integration.html] > > It is needed to support custom commitProtocolClass class when using > saveAsNewAPIHadoopDataset by an option. For example, > {code:java} > spark.hadoop.mapreduce.sources.commitProtocolClass > org.apache.spark.internal.io.cloud.PathOutputCommitProtocol{code} > > -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org