[ https://issues.apache.org/jira/browse/SPARK-44884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17757036#comment-17757036 ]
Steve Loughran commented on SPARK-44884: ---------------------------------------- this is created in the committer; for hadoop-mapreduce ones "mapreduce.fileoutputcommitter.marksuccessfuljobs"; is the flag to enable this; if it is not being created then it'll be down to how saveAsTable commits work > Spark doesn't create SUCCESS file when external path is passed > -------------------------------------------------------------- > > Key: SPARK-44884 > URL: https://issues.apache.org/jira/browse/SPARK-44884 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 3.3.0 > Reporter: Dipayan Dev > Priority: Critical > Attachments: image-2023-08-20-18-08-38-531.png, > image-2023-08-20-18-46-53-342.png > > > The issue is not happening in Spark 2.x (I am using 2.4.0), but only in 3.3.0 > Code to reproduce the issue. > > {code:java} > scala> spark.conf.set("spark.sql.orc.char.enabled", true) > scala> val DF = Seq(("test1", 123)).toDF("name", "num") > scala> DF.write.option("path", > "gs://test_dd123/").mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("test_schema.table_name") > 23/08/20 12:31:43 WARN SessionState: METASTORE_FILTER_HOOK will be ignored, > since hive.security.authorization.manager is set to instance of > HiveAuthorizerFactory. {code} > The above code succeeds and creates the External Hive table, but {*}there is > no SUCCESS file generated{*}. The same code when running spark 2.4.0, > generating a SUCCESS file. > Adding the content of the bucket after table creation > > !image-2023-08-20-18-08-38-531.png|width=453,height=162! > > But when I don’t pass the external path as following, the SUCCESS file is > generated > {code:java} > scala> > DF.write.mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("us_wm_supply_chain_rcv_pre_prod.test_tb1") > {code} > !image-2023-08-20-18-46-53-342.png|width=465,height=166! > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org