[jira] [Updated] (SPARK-44884) Spark doesn't create SUCCESS file in Spark 3.3.0+ when partitionOverwriteMode is dynamic
[
https://issues.apache.org/jira/browse/SPARK-44884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dipayan Dev updated SPARK-44884:
Priority: Minor (was: Critical)
> Spark doesn't create SUCCESS file in Spark 3.3.0+ when partitionOverwriteMode
> is dynamic
>
>
> Key: SPARK-44884
> URL: https://issues.apache.org/jira/browse/SPARK-44884
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.3.0
>Reporter: Dipayan Dev
>Priority: Minor
> Attachments: image-2023-08-20-18-46-53-342.png,
> image-2023-08-25-13-01-42-137.png
>
>
> The issue is not happening in Spark 2.x (I am using 2.4.0), but only in 3.3.0
> (tested with 3.4.1 as well)
> Code to reproduce the issue
>
> {code:java}
> scala> spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
> scala> val DF = Seq(("test1", 123)).toDF("name", "num")
> scala> DF.write.option("path",
> "gs://test_bucket/table").mode("overwrite").partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1")
> {code}
>
> The above code succeeds and creates external Hive table, but {*}there is no
> SUCCESS file generated{*}.
> Adding the content of the bucket after table creation
> !image-2023-08-25-13-01-42-137.png|width=500,height=130!
> The same code when running with spark 2.4.0 (with or without external path),
> generates the SUCCESS file.
> {code:java}
> scala>
> DF.write.mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1"){code}
> !image-2023-08-20-18-46-53-342.png|width=465,height=166!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-44884) Spark doesn't create SUCCESS file in Spark 3.3.0+ when partitionOverwriteMode is dynamic
[
https://issues.apache.org/jira/browse/SPARK-44884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dipayan Dev updated SPARK-44884:
Priority: Major (was: Minor)
> Spark doesn't create SUCCESS file in Spark 3.3.0+ when partitionOverwriteMode
> is dynamic
>
>
> Key: SPARK-44884
> URL: https://issues.apache.org/jira/browse/SPARK-44884
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.3.0
>Reporter: Dipayan Dev
>Priority: Major
> Attachments: image-2023-08-20-18-46-53-342.png,
> image-2023-08-25-13-01-42-137.png
>
>
> The issue is not happening in Spark 2.x (I am using 2.4.0), but only in 3.3.0
> (tested with 3.4.1 as well)
> Code to reproduce the issue
>
> {code:java}
> scala> spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
> scala> val DF = Seq(("test1", 123)).toDF("name", "num")
> scala> DF.write.option("path",
> "gs://test_bucket/table").mode("overwrite").partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1")
> {code}
>
> The above code succeeds and creates external Hive table, but {*}there is no
> SUCCESS file generated{*}.
> Adding the content of the bucket after table creation
> !image-2023-08-25-13-01-42-137.png|width=500,height=130!
> The same code when running with spark 2.4.0 (with or without external path),
> generates the SUCCESS file.
> {code:java}
> scala>
> DF.write.mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1"){code}
> !image-2023-08-20-18-46-53-342.png|width=465,height=166!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-44884) Spark doesn't create SUCCESS file in Spark 3.3.0+ when partitionOverwriteMode is dynamic
[
https://issues.apache.org/jira/browse/SPARK-44884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dipayan Dev updated SPARK-44884:
Summary: Spark doesn't create SUCCESS file in Spark 3.3.0+ when
partitionOverwriteMode is dynamic (was: Spark doesn't create SUCCESS file in
Spark 3.3.0+)
> Spark doesn't create SUCCESS file in Spark 3.3.0+ when partitionOverwriteMode
> is dynamic
>
>
> Key: SPARK-44884
> URL: https://issues.apache.org/jira/browse/SPARK-44884
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.3.0
>Reporter: Dipayan Dev
>Priority: Critical
> Attachments: image-2023-08-20-18-46-53-342.png,
> image-2023-08-25-13-01-42-137.png
>
>
> The issue is not happening in Spark 2.x (I am using 2.4.0), but only in 3.3.0
> (tested with 3.4.1 as well)
> Code to reproduce the issue
>
> {code:java}
> scala> spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
> scala> val DF = Seq(("test1", 123)).toDF("name", "num")
> scala> DF.write.option("path",
> "gs://test_bucket/table").mode("overwrite").partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1")
> {code}
>
> The above code succeeds and creates external Hive table, but {*}there is no
> SUCCESS file generated{*}.
> Adding the content of the bucket after table creation
> !image-2023-08-25-13-01-42-137.png|width=500,height=130!
> The same code when running with spark 2.4.0 (with or without external path),
> generates the SUCCESS file.
> {code:java}
> scala>
> DF.write.mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1"){code}
> !image-2023-08-20-18-46-53-342.png|width=465,height=166!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-44884) Spark doesn't create SUCCESS file in Spark 3.3.0+
[
https://issues.apache.org/jira/browse/SPARK-44884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dipayan Dev updated SPARK-44884:
Description:
The issue is not happening in Spark 2.x (I am using 2.4.0), but only in 3.3.0
(tested with 3.4.1 as well)
Code to reproduce the issue
{code:java}
scala> spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
scala> val DF = Seq(("test1", 123)).toDF("name", "num")
scala> DF.write.option("path",
"gs://test_bucket/table").mode("overwrite").partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1")
{code}
The above code succeeds and creates external Hive table, but {*}there is no
SUCCESS file generated{*}.
Adding the content of the bucket after table creation
!image-2023-08-25-13-01-42-137.png|width=500,height=130!
The same code when running with spark 2.4.0 (with or without external path),
generates the SUCCESS file.
{code:java}
scala>
DF.write.mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1"){code}
!image-2023-08-20-18-46-53-342.png|width=465,height=166!
was:
The issue is not happening in Spark 2.x (I am using 2.4.0), but only in 3.3.0
(tested with 3.4.1 as well)
Code to reproduce the issue
{code:java}
scala> spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
scala> val DF = Seq(("test1", 123)).toDF("name", "num")
scala> DF.write.option("path",
"gs://test_bucket/table").mode("overwrite").partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1")
{code}
The above code succeeds and creates external Hive table, but {*}there is no
SUCCESS file generated{*}.
Adding the content of the bucket after table creation
!image-2023-08-25-13-01-42-137.png|width=500,height=130!
The same code when running with spark 2.4.0 (with or without external path),
generates the SUCCESS file.
{code:java}
scala>
DF.write.mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1"){code}
!image-2023-08-20-18-46-53-342.png|width=465,height=166!
> Spark doesn't create SUCCESS file in Spark 3.3.0+
> -
>
> Key: SPARK-44884
> URL: https://issues.apache.org/jira/browse/SPARK-44884
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.3.0
>Reporter: Dipayan Dev
>Priority: Critical
> Attachments: image-2023-08-20-18-46-53-342.png,
> image-2023-08-25-13-01-42-137.png
>
>
> The issue is not happening in Spark 2.x (I am using 2.4.0), but only in 3.3.0
> (tested with 3.4.1 as well)
> Code to reproduce the issue
>
> {code:java}
> scala> spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
> scala> val DF = Seq(("test1", 123)).toDF("name", "num")
> scala> DF.write.option("path",
> "gs://test_bucket/table").mode("overwrite").partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1")
> {code}
>
> The above code succeeds and creates external Hive table, but {*}there is no
> SUCCESS file generated{*}.
> Adding the content of the bucket after table creation
> !image-2023-08-25-13-01-42-137.png|width=500,height=130!
> The same code when running with spark 2.4.0 (with or without external path),
> generates the SUCCESS file.
> {code:java}
> scala>
> DF.write.mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1"){code}
> !image-2023-08-20-18-46-53-342.png|width=465,height=166!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-44884) Spark doesn't create SUCCESS file in Spark 3.3.0+
[
https://issues.apache.org/jira/browse/SPARK-44884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dipayan Dev updated SPARK-44884:
Description:
The issue is not happening in Spark 2.x (I am using 2.4.0), but only in 3.3.0
(tested with 3.4.1 as well)
Code to reproduce the issue
{code:java}
scala> spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
scala> val DF = Seq(("test1", 123)).toDF("name", "num")
scala> DF.write.option("path",
"gs://test_bucket/table").mode("overwrite").partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1")
{code}
The above code succeeds and creates external Hive table, but {*}there is no
SUCCESS file generated{*}.
Adding the content of the bucket after table creation
!image-2023-08-25-13-01-42-137.png|width=500,height=130!
The same code when running with spark 2.4.0 (with or without external path),
generates the SUCCESS file.
{code:java}
scala>
DF.write.mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1"){code}
!image-2023-08-20-18-46-53-342.png|width=465,height=166!
was:
The issue is not happening in Spark 2.x (I am using 2.4.0), but only in 3.3.0
Code to reproduce the issue
{code:java}
scala> spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
scala> val DF = Seq(("test1", 123)).toDF("name", "num")
scala> DF.write.option("path",
"gs://test_bucket/table").mode("overwrite").partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1")
{code}
The above code succeeds and creates external Hive table, but {*}there is no
SUCCESS file generated{*}.
Adding the content of the bucket after table creation
!image-2023-08-25-13-01-42-137.png|width=500,height=130!
The same code when running with spark 2.4.0 (with or without external path),
generates the SUCCESS file.
{code:java}
scala>
DF.write.mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1"){code}
!image-2023-08-20-18-46-53-342.png|width=465,height=166!
> Spark doesn't create SUCCESS file in Spark 3.3.0+
> -
>
> Key: SPARK-44884
> URL: https://issues.apache.org/jira/browse/SPARK-44884
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.3.0
>Reporter: Dipayan Dev
>Priority: Critical
> Attachments: image-2023-08-20-18-46-53-342.png,
> image-2023-08-25-13-01-42-137.png
>
>
> The issue is not happening in Spark 2.x (I am using 2.4.0), but only in 3.3.0
> (tested with 3.4.1 as well)
> Code to reproduce the issue
>
> {code:java}
> scala> spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
> scala> val DF = Seq(("test1", 123)).toDF("name", "num")
> scala> DF.write.option("path",
> "gs://test_bucket/table").mode("overwrite").partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1")
> {code}
>
> The above code succeeds and creates external Hive table, but {*}there is no
> SUCCESS file generated{*}.
> Adding the content of the bucket after table creation
> !image-2023-08-25-13-01-42-137.png|width=500,height=130!
> The same code when running with spark 2.4.0 (with or without external path),
> generates the SUCCESS file.
> {code:java}
> scala>
> DF.write.mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1"){code}
> !image-2023-08-20-18-46-53-342.png|width=465,height=166!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-44884) Spark doesn't create SUCCESS file in Spark 3.3.0+
[
https://issues.apache.org/jira/browse/SPARK-44884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dipayan Dev updated SPARK-44884:
Description:
The issue is not happening in Spark 2.x (I am using 2.4.0), but only in 3.3.0
Code to reproduce the issue
{code:java}
scala> spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
scala> val DF = Seq(("test1", 123)).toDF("name", "num")
scala> DF.write.option("path",
"gs://test_bucket/table").mode("overwrite").partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1")
{code}
The above code succeeds and creates external Hive table, but {*}there is no
SUCCESS file generated{*}.
Adding the content of the bucket after table creation
!image-2023-08-25-13-01-42-137.png|width=500,height=130!
The same code when running with spark 2.4.0 (with or without external path),
generates the SUCCESS file.
{code:java}
scala>
DF.write.mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1"){code}
!image-2023-08-20-18-46-53-342.png|width=465,height=166!
was:
The issue is not happening in Spark 2.x (I am using 2.4.0), but only in 3.3.0
Code to reproduce the issue.
{code:java}
scala> spark.conf.set("spark.sql.orc.char.enabled", true)
scala> val DF = Seq(("test1", 123)).toDF("name", "num")
scala> DF.write.option("path",
"gs://test_dd123/").mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("test_schema.table_name")
23/08/20 12:31:43 WARN SessionState: METASTORE_FILTER_HOOK will be ignored,
since hive.security.authorization.manager is set to instance of
HiveAuthorizerFactory. {code}
The above code succeeds and creates the External Hive table, but {*}there is no
SUCCESS file generated{*}. The same code when running spark 2.4.0, generating a
SUCCESS file.
Adding the content of the bucket after table creation
!image-2023-08-20-18-08-38-531.png|width=453,height=162!
But when I don’t pass the external path as following, the SUCCESS file is
generated
{code:java}
scala>
DF.write.mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("us_wm_supply_chain_rcv_pre_prod.test_tb1")
{code}
!image-2023-08-20-18-46-53-342.png|width=465,height=166!
> Spark doesn't create SUCCESS file in Spark 3.3.0+
> -
>
> Key: SPARK-44884
> URL: https://issues.apache.org/jira/browse/SPARK-44884
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.3.0
>Reporter: Dipayan Dev
>Priority: Critical
> Attachments: image-2023-08-20-18-46-53-342.png,
> image-2023-08-25-13-01-42-137.png
>
>
> The issue is not happening in Spark 2.x (I am using 2.4.0), but only in 3.3.0
> Code to reproduce the issue
>
> {code:java}
> scala> spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
> scala> val DF = Seq(("test1", 123)).toDF("name", "num")
> scala> DF.write.option("path",
> "gs://test_bucket/table").mode("overwrite").partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1")
> {code}
>
> The above code succeeds and creates external Hive table, but {*}there is no
> SUCCESS file generated{*}.
> Adding the content of the bucket after table creation
> !image-2023-08-25-13-01-42-137.png|width=500,height=130!
> The same code when running with spark 2.4.0 (with or without external path),
> generates the SUCCESS file.
> {code:java}
> scala>
> DF.write.mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1"){code}
> !image-2023-08-20-18-46-53-342.png|width=465,height=166!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-44884) Spark doesn't create SUCCESS file in Spark 3.3.0+
[
https://issues.apache.org/jira/browse/SPARK-44884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dipayan Dev updated SPARK-44884:
Attachment: (was: image-2023-08-20-18-08-38-531.png)
> Spark doesn't create SUCCESS file in Spark 3.3.0+
> -
>
> Key: SPARK-44884
> URL: https://issues.apache.org/jira/browse/SPARK-44884
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.3.0
>Reporter: Dipayan Dev
>Priority: Critical
> Attachments: image-2023-08-20-18-46-53-342.png,
> image-2023-08-25-13-01-42-137.png
>
>
> The issue is not happening in Spark 2.x (I am using 2.4.0), but only in 3.3.0
> Code to reproduce the issue
>
> {code:java}
> scala> spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
> scala> val DF = Seq(("test1", 123)).toDF("name", "num")
> scala> DF.write.option("path",
> "gs://test_bucket/table").mode("overwrite").partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1")
> {code}
>
> The above code succeeds and creates external Hive table, but {*}there is no
> SUCCESS file generated{*}.
> Adding the content of the bucket after table creation
> !image-2023-08-25-13-01-42-137.png|width=500,height=130!
> The same code when running with spark 2.4.0 (with or without external path),
> generates the SUCCESS file.
> {code:java}
> scala>
> DF.write.mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("test_schema.test_tb1"){code}
> !image-2023-08-20-18-46-53-342.png|width=465,height=166!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-44884) Spark doesn't create SUCCESS file in Spark 3.3.0+
[
https://issues.apache.org/jira/browse/SPARK-44884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dipayan Dev updated SPARK-44884:
Attachment: image-2023-08-25-13-01-42-137.png
> Spark doesn't create SUCCESS file in Spark 3.3.0+
> -
>
> Key: SPARK-44884
> URL: https://issues.apache.org/jira/browse/SPARK-44884
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.3.0
>Reporter: Dipayan Dev
>Priority: Critical
> Attachments: image-2023-08-20-18-08-38-531.png,
> image-2023-08-20-18-46-53-342.png, image-2023-08-25-13-01-42-137.png
>
>
> The issue is not happening in Spark 2.x (I am using 2.4.0), but only in 3.3.0
> Code to reproduce the issue.
>
> {code:java}
> scala> spark.conf.set("spark.sql.orc.char.enabled", true)
> scala> val DF = Seq(("test1", 123)).toDF("name", "num")
> scala> DF.write.option("path",
> "gs://test_dd123/").mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("test_schema.table_name")
> 23/08/20 12:31:43 WARN SessionState: METASTORE_FILTER_HOOK will be ignored,
> since hive.security.authorization.manager is set to instance of
> HiveAuthorizerFactory. {code}
> The above code succeeds and creates the External Hive table, but {*}there is
> no SUCCESS file generated{*}. The same code when running spark 2.4.0,
> generating a SUCCESS file.
> Adding the content of the bucket after table creation
>
> !image-2023-08-20-18-08-38-531.png|width=453,height=162!
>
> But when I don’t pass the external path as following, the SUCCESS file is
> generated
> {code:java}
> scala>
> DF.write.mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("us_wm_supply_chain_rcv_pre_prod.test_tb1")
> {code}
> !image-2023-08-20-18-46-53-342.png|width=465,height=166!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-44884) Spark doesn't create SUCCESS file in Spark 3.3.0+
[
https://issues.apache.org/jira/browse/SPARK-44884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dipayan Dev updated SPARK-44884:
Summary: Spark doesn't create SUCCESS file in Spark 3.3.0+ (was: Spark
doesn't create SUCCESS file when external path is passed)
> Spark doesn't create SUCCESS file in Spark 3.3.0+
> -
>
> Key: SPARK-44884
> URL: https://issues.apache.org/jira/browse/SPARK-44884
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.3.0
>Reporter: Dipayan Dev
>Priority: Critical
> Attachments: image-2023-08-20-18-08-38-531.png,
> image-2023-08-20-18-46-53-342.png
>
>
> The issue is not happening in Spark 2.x (I am using 2.4.0), but only in 3.3.0
> Code to reproduce the issue.
>
> {code:java}
> scala> spark.conf.set("spark.sql.orc.char.enabled", true)
> scala> val DF = Seq(("test1", 123)).toDF("name", "num")
> scala> DF.write.option("path",
> "gs://test_dd123/").mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("test_schema.table_name")
> 23/08/20 12:31:43 WARN SessionState: METASTORE_FILTER_HOOK will be ignored,
> since hive.security.authorization.manager is set to instance of
> HiveAuthorizerFactory. {code}
> The above code succeeds and creates the External Hive table, but {*}there is
> no SUCCESS file generated{*}. The same code when running spark 2.4.0,
> generating a SUCCESS file.
> Adding the content of the bucket after table creation
>
> !image-2023-08-20-18-08-38-531.png|width=453,height=162!
>
> But when I don’t pass the external path as following, the SUCCESS file is
> generated
> {code:java}
> scala>
> DF.write.mode(SaveMode.Overwrite).partitionBy("num").format("orc").saveAsTable("us_wm_supply_chain_rcv_pre_prod.test_tb1")
> {code}
> !image-2023-08-20-18-46-53-342.png|width=465,height=166!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
