[ https://issues.apache.org/jira/browse/SPARK-44884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dipayan Dev updated SPARK-44884: -------------------------------- Description: The issue is not happening in Spark 2.x (I am using 2.4.0), but only in 3.3.0 Code to reproduce the issue. {code:java} scala> spark.conf.set("spark.sql.orc.char.enabled", true) scala> val DF = Seq(("test1", 123)).toDF("name", "num") scala> DF.write.option("path", "gs://test_dd123/").mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("test_schema.table_name") 23/08/20 12:31:43 WARN SessionState: METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory. {code} The above code succeeds and creates the External Hive table, but {*}there is no SUCCESS file generated{*}. The same code when running spark 2.4.0, generating a SUCCESS file. Adding the content of the bucket are table creation !image-2023-08-20-18-08-38-531.png|width=453,height=162! But when I use the following, the SUCCESS file is generated {code:java} scala> DF.write.mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("us_wm_supply_chain_rcv_pre_prod.test_tb1") {code} !image-2023-08-20-18-46-53-342.png|width=465,height=166! was: The issue is not happening in Spark 2.x (I am using 2.4.0), but only in 3.3.0 Code to reproduce the issue. {code:java} scala> spark.conf.set("spark.sql.orc.char.enabled", true) scala> val DF = Seq(("test1", 123)).toDF("name", "num") scala> DF.write.option("path", "gs://test_dd123/").mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("test_schema.table_name") 23/08/20 12:31:43 WARN SessionState: METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory. {code} The above code succeeds and creates the External Hive table, but {*}there is no SUCCESS file generated{*}. The same code when running spark 2.4.0, generating a SUCCESS file. Adding the content of the bucket are table creation !image-2023-08-20-18-08-38-531.png|width=453,height=162! > Spark doesn't create SUCCESS file when external path is passed > -------------------------------------------------------------- > > Key: SPARK-44884 > URL: https://issues.apache.org/jira/browse/SPARK-44884 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 3.3.0 > Reporter: Dipayan Dev > Priority: Critical > Attachments: image-2023-08-20-18-08-38-531.png, > image-2023-08-20-18-46-53-342.png > > > The issue is not happening in Spark 2.x (I am using 2.4.0), but only in 3.3.0 > Code to reproduce the issue. > > {code:java} > scala> spark.conf.set("spark.sql.orc.char.enabled", true) > scala> val DF = Seq(("test1", 123)).toDF("name", "num") > scala> DF.write.option("path", > "gs://test_dd123/").mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("test_schema.table_name") > 23/08/20 12:31:43 WARN SessionState: METASTORE_FILTER_HOOK will be ignored, > since hive.security.authorization.manager is set to instance of > HiveAuthorizerFactory. {code} > The above code succeeds and creates the External Hive table, but {*}there is > no SUCCESS file generated{*}. The same code when running spark 2.4.0, > generating a SUCCESS file. > Adding the content of the bucket are table creation > > !image-2023-08-20-18-08-38-531.png|width=453,height=162! > > But when I use the following, the SUCCESS file is generated > {code:java} > scala> > DF.write.mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("us_wm_supply_chain_rcv_pre_prod.test_tb1") > {code} > !image-2023-08-20-18-46-53-342.png|width=465,height=166! > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org