[ https://issues.apache.org/jira/browse/SPARK-44884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dipayan Dev updated SPARK-44884: -------------------------------- Description: The issue is not happening in Spark 2.x (I am using 2.4.0), but only in 3.3.0 Code to reproduce the issue {code:java} scala> spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic") scala> val DF = Seq(("test1", 123)).toDF("name", "num") scala> DF.write.option("path", "gs://test_bucket/table").mode("overwrite").partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1") {code} The above code succeeds and creates external Hive table, but {*}there is no SUCCESS file generated{*}. Adding the content of the bucket after table creation !image-2023-08-25-13-01-42-137.png|width=500,height=130! The same code when running with spark 2.4.0 (with or without external path), generates the SUCCESS file. {code:java} scala> DF.write.mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1"){code} !image-2023-08-20-18-46-53-342.png|width=465,height=166! was: The issue is not happening in Spark 2.x (I am using 2.4.0), but only in 3.3.0 Code to reproduce the issue. {code:java} scala> spark.conf.set("spark.sql.orc.char.enabled", true) scala> val DF = Seq(("test1", 123)).toDF("name", "num") scala> DF.write.option("path", "gs://test_dd123/").mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("test_schema.table_name") 23/08/20 12:31:43 WARN SessionState: METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory. {code} The above code succeeds and creates the External Hive table, but {*}there is no SUCCESS file generated{*}. The same code when running spark 2.4.0, generating a SUCCESS file. Adding the content of the bucket after table creation !image-2023-08-20-18-08-38-531.png|width=453,height=162! But when I don’t pass the external path as following, the SUCCESS file is generated {code:java} scala> DF.write.mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("us_wm_supply_chain_rcv_pre_prod.test_tb1") {code} !image-2023-08-20-18-46-53-342.png|width=465,height=166! > Spark doesn't create SUCCESS file in Spark 3.3.0+ > ------------------------------------------------- > > Key: SPARK-44884 > URL: https://issues.apache.org/jira/browse/SPARK-44884 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 3.3.0 > Reporter: Dipayan Dev > Priority: Critical > Attachments: image-2023-08-20-18-46-53-342.png, > image-2023-08-25-13-01-42-137.png > > > The issue is not happening in Spark 2.x (I am using 2.4.0), but only in 3.3.0 > Code to reproduce the issue > > {code:java} > scala> spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic") > scala> val DF = Seq(("test1", 123)).toDF("name", "num") > scala> DF.write.option("path", > "gs://test_bucket/table").mode("overwrite").partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1") > {code} > > The above code succeeds and creates external Hive table, but {*}there is no > SUCCESS file generated{*}. > Adding the content of the bucket after table creation > !image-2023-08-25-13-01-42-137.png|width=500,height=130! > The same code when running with spark 2.4.0 (with or without external path), > generates the SUCCESS file. > {code:java} > scala> > DF.write.mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1"){code} > !image-2023-08-20-18-46-53-342.png|width=465,height=166! > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org