from:"kevinshin \(Jira\)"

[jira] [Commented] (SPARK-42804) when target table format is textfile using `insert into select` will got error

2023-03-16 Thread kevinshin (Jira)



[ 
https://issues.apache.org/jira/browse/SPARK-42804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17701010#comment-17701010
 ] 

kevinshin commented on SPARK-42804:
---

@[~yumwang]  below is my step by step reproduce  this issue : 
 
hive version is HDP 3.1.0.3.1.4.0-315
 
[bigtop@hdpdev243 spark3]$ {color:#4c9aff}cat conf/spark-defaults.conf{color}
# Generated by Apache Ambari. Tue Apr 27 11:19:24 2021
 
spark.sql.hive.convertMetastoreOrc true
spark.sql.orc.filterPushdown true
spark.sql.orc.impl native
spark.sql.legacy.createHiveTableByDefault false
 
[bigtop@hdpdev243 spark3]$ {color:#4c9aff}bin/spark-sql{color}
23/03/16 15:03:29 WARN NativeCodeLoader: Unable to load native-hadoop library 
for your platform... using builtin-java classes where applicable
23/03/16 15:03:29 WARN HiveConf: HiveConf of name 
hive.materializedview.rewriting.incremental does not exist
23/03/16 15:03:29 WARN HiveConf: HiveConf of name 
hive.metastore.event.db.notification.api.auth does not exist
23/03/16 15:03:29 WARN HiveConf: HiveConf of name 
hive.server2.webui.cors.allowed.headers does not exist
23/03/16 15:03:29 WARN HiveConf: HiveConf of name 
hive.hook.proto.base-directory does not exist
23/03/16 15:03:29 WARN HiveConf: HiveConf of name hive.load.data.owner does not 
exist
23/03/16 15:03:29 WARN HiveConf: HiveConf of name 
hive.service.metrics.codahale.reporter.classes does not exist
23/03/16 15:03:29 WARN HiveConf: HiveConf of name hive.strict.managed.tables 
does not exist
23/03/16 15:03:29 WARN HiveConf: HiveConf of name hive.create.as.insert.only 
does not exist
23/03/16 15:03:29 WARN HiveConf: HiveConf of name hive.metastore.db.type does 
not exist
23/03/16 15:03:29 WARN HiveConf: HiveConf of name 
hive.tez.cartesian-product.enabled does not exist
23/03/16 15:03:29 WARN HiveConf: HiveConf of name 
hive.metastore.warehouse.external.dir does not exist
23/03/16 15:03:29 WARN HiveConf: HiveConf of name hive.heapsize does not exist
23/03/16 15:03:29 WARN HiveConf: HiveConf of name 
hive.server2.webui.enable.cors does not exist
23/03/16 15:03:29 WARN HiveClientImpl: Detected HiveConf hive.execution.engine 
is 'tez' and will be reset to 'mr' to disable useless hive logic
23/03/16 15:03:30 WARN DomainSocketFactory: The short-circuit local reads 
feature cannot be used because libhadoop cannot be loaded.
Spark master: local[*], Application Id: local-1678950211606
spark-sql> select version();
3.2.3 b53c341e0fefbb33d115ab630369a18765b7763d
Time taken: 3.956 seconds, Fetched 1 row(s)
spark-sql> {color:#4c9aff}create table test.tex_t1(name string, address string) 
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE;{color}
23/03/16 15:03:51 WARN SessionState: METASTORE_FILTER_HOOK will be ignored, 
since hive.security.authorization.manager is set to instance of 
HiveAuthorizerFactory.
Time taken: 0.753 seconds
spark-sql> {color:#4c9aff}create table test.tex_t2(name string, address 
string);{color}
Time taken: 0.326 seconds
spark-sql> {color:#4c9aff}insert into test.tex_t2 select 'a', 'b';{color}
Time taken: 2.011 seconds
spark-sql> {color:#4c9aff}insert into test.tex_t1 select 'a', 'b';{color}
23/03/16 15:04:13 WARN HdfsUtils: Unable to inherit permissions for file 
hdfs://nsdev/warehouse/tablespace/managed/hive/test.db/tex_t1/part-0-57c15f7a-7462-4101-af5d-9f4a22cf69df-c000
 from file hdfs://nsdev/warehouse/tablespace/man
aged/hive/test.db/tex_t1
23/03/16 15:04:13 WARN RetryingMetaStoreClient: MetaStoreClient lost 
connection. Attempting to reconnect (1 of 24) after 5s. fireListenerEvent
org.apache.thrift.transport.TTransportException
at 
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:425)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:321)
at 
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:225)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_fire_listener_event(ThriftHiveMetastore.java:4977)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.fire_listener_event(ThriftHiveMetastore.java:4964)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.fireListenerEvent(HiveMetaStoreClient.java:2296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)
at com.sun.proxy.$Proxy21.fireListenerEvent(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at

[jira] [Comment Edited] (SPARK-42804) when target table format is textfile using `insert into select` will got error

2023-03-16 Thread kevinshin (Jira)



[ 
https://issues.apache.org/jira/browse/SPARK-42804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17700926#comment-17700926
 ] 

kevinshin edited comment on SPARK-42804 at 3/16/23 6:47 AM:


orc and parquet table won't have this problem.

directly use hive beeline connect to hive also have no problem.


was (Author: JIRAUSER281772):
orc and parquet table won't have this problem.

> when target table format is textfile using `insert into select` will got error
> --
>
> Key: SPARK-42804
> URL: https://issues.apache.org/jira/browse/SPARK-42804
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
>Reporter: kevinshin
>Priority: Major
>
> *create* *table* test.tex_t1(name string, address string) *ROW* FORMAT 
> DELIMITED FIELDS TERMINATED *BY* ',' STORED *AS* TEXTFILE;
> *insert* *into* test.tex_t1 *select* 'a', 'b';
> will got alot of message about :
> WARN RetryingMetaStoreClient: MetaStoreClient lost connection. Attempting to 
> reconnect (24 of 24) after 5s. fireListenerEvent
> org.apache.thrift.transport.TTransportException
>  
> But the data was actual write to table.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-42804) when target table format is textfile using `insert into select` will got error

2023-03-15 Thread kevinshin (Jira)



[ 
https://issues.apache.org/jira/browse/SPARK-42804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17700926#comment-17700926
 ] 

kevinshin commented on SPARK-42804:
---

orc and parquet table won't have this problem.

> when target table format is textfile using `insert into select` will got error
> --
>
> Key: SPARK-42804
> URL: https://issues.apache.org/jira/browse/SPARK-42804
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
>Reporter: kevinshin
>Priority: Major
>
> *create* *table* test.tex_t1(name string, address string) *ROW* FORMAT 
> DELIMITED FIELDS TERMINATED *BY* ',' STORED *AS* TEXTFILE;
> *insert* *into* test.tex_t1 *select* 'a', 'b';
> will got alot of message about :
> WARN RetryingMetaStoreClient: MetaStoreClient lost connection. Attempting to 
> reconnect (24 of 24) after 5s. fireListenerEvent
> org.apache.thrift.transport.TTransportException
>  
> But the data was actual write to table.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-42804) when target table format is textfile using `insert into select` will got error

2023-03-15 Thread kevinshin (Jira)

kevinshin created SPARK-42804:
-

 Summary: when target table format is textfile using `insert into 
select` will got error
 Key: SPARK-42804
 URL: https://issues.apache.org/jira/browse/SPARK-42804
 Project: Spark
  Issue Type: Bug
  Components: SQL
Affects Versions: 3.2.3
Reporter: kevinshin


*create* *table* test.tex_t1(name string, address string) *ROW* FORMAT 
DELIMITED FIELDS TERMINATED *BY* ',' STORED *AS* TEXTFILE;


*insert* *into* test.tex_t1 *select* 'a', 'b';

will got alot of message about :
WARN RetryingMetaStoreClient: MetaStoreClient lost connection. Attempting to 
reconnect (24 of 24) after 5s. fireListenerEvent
org.apache.thrift.transport.TTransportException
 
But the data was actual write to table.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-42728) when enable AQE update or drop a small iceberg table should not generate as much task as `spark.sql.adaptive.coalescePartitions.initialPartitionNum` config

2023-03-09 Thread kevinshin (Jira)



 [ 
https://issues.apache.org/jira/browse/SPARK-42728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevinshin updated SPARK-42728:
--
Description: 
*spark config :* 
spark.sql.adaptive.enabled true
spark.sql.adaptive.advisoryPartitionSizeInBytes 256m
spark.sql.adaptive.coalescePartitions.enabled true
spark.sql.adaptive.coalescePartitions.minPartitionNum 1
spark.sql.adaptive.coalescePartitions.initialPartitionNum 2000
spark.sql.extensions 
org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog
spark.sql.catalog.spark_catalog.type hive
 
*Sql script:* 
 
{color:#4c9aff}*create* *table* spark_catalog.test.iceberg5(id {*}bigint{*}, 
*data* string) *USING* iceberg;{color}

{color:#4c9aff}*INSERT* *INTO* spark_catalog.test.iceberg5 *VALUES* (1, 'a'), 
(2, 'b'), (3, 'c');{color}

{color:#4c9aff}*update* test.iceberg5 *set* `data`{*}={*}'cc' *where* 
id{*}={*}3;{color}

{color:#4c9aff}*drop* *table* spark_catalog.test.iceberg5 PURGE;{color}

*spark web :*  

!image-2023-03-09-17-34-26-428.png!

  was:
spark config : 
spark.sql.adaptive.enabled true
spark.sql.adaptive.advisoryPartitionSizeInBytes 256m
spark.sql.adaptive.coalescePartitions.enabled true
spark.sql.adaptive.coalescePartitions.minPartitionNum 1
spark.sql.adaptive.coalescePartitions.initialPartitionNum 2000
spark.sql.extensions 
org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog
spark.sql.catalog.spark_catalog.type hive
 
Sql script: 
 
*create* *table* spark_catalog.test.iceberg5(id {*}bigint{*}, *data* string) 
*USING* iceberg;

*INSERT* *INTO* spark_catalog.test.iceberg5 *VALUES* (1, 'a'), (2, 'b'), (3, 
'c');

*update* test.iceberg5 *set* `data`{*}={*}'cc' *where* id{*}={*}3;

*drop* *table* spark_catalog.test.iceberg5 PURGE;

spark web :  

!image-2023-03-09-17-34-26-428.png!


> when enable AQE update or drop a small iceberg table should not generate as 
> much task as `spark.sql.adaptive.coalescePartitions.initialPartitionNum` 
> config
> ---
>
> Key: SPARK-42728
> URL: https://issues.apache.org/jira/browse/SPARK-42728
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 3.2.1, 3.2.3
> Environment: spark3.2.1 or 3.2.3; iceberg 0.14.1 or 1.1.0
>  
>Reporter: kevinshin
>Priority: Minor
> Attachments: image-2023-03-09-17-34-26-428.png
>
>
> *spark config :* 
> spark.sql.adaptive.enabled true
> spark.sql.adaptive.advisoryPartitionSizeInBytes 256m
> spark.sql.adaptive.coalescePartitions.enabled true
> spark.sql.adaptive.coalescePartitions.minPartitionNum 1
> spark.sql.adaptive.coalescePartitions.initialPartitionNum 2000
> spark.sql.extensions 
> org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
> spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog
> spark.sql.catalog.spark_catalog.type hive
>  
> *Sql script:* 
>  
> {color:#4c9aff}*create* *table* spark_catalog.test.iceberg5(id {*}bigint{*}, 
> *data* string) *USING* iceberg;{color}
> {color:#4c9aff}*INSERT* *INTO* spark_catalog.test.iceberg5 *VALUES* (1, 'a'), 
> (2, 'b'), (3, 'c');{color}
> {color:#4c9aff}*update* test.iceberg5 *set* `data`{*}={*}'cc' *where* 
> id{*}={*}3;{color}
> {color:#4c9aff}*drop* *table* spark_catalog.test.iceberg5 PURGE;{color}
> *spark web :*  
> !image-2023-03-09-17-34-26-428.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-42728) when enable AQE update or drop a small iceberg table should not generate as much task as `spark.sql.adaptive.coalescePartitions.initialPartitionNum` config

2023-03-09 Thread kevinshin (Jira)



 [ 
https://issues.apache.org/jira/browse/SPARK-42728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevinshin updated SPARK-42728:
--
Description: 
spark config : 
spark.sql.adaptive.enabled true
spark.sql.adaptive.advisoryPartitionSizeInBytes 256m
spark.sql.adaptive.coalescePartitions.enabled true
spark.sql.adaptive.coalescePartitions.minPartitionNum 1
spark.sql.adaptive.coalescePartitions.initialPartitionNum 2000
spark.sql.extensions 
org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog
spark.sql.catalog.spark_catalog.type hive
 
Sql script: 
 
*create* *table* spark_catalog.test.iceberg5(id {*}bigint{*}, *data* string) 
*USING* iceberg;

*INSERT* *INTO* spark_catalog.test.iceberg5 *VALUES* (1, 'a'), (2, 'b'), (3, 
'c');

*update* test.iceberg5 *set* `data`{*}={*}'cc' *where* id{*}={*}3;

*drop* *table* spark_catalog.test.iceberg5 PURGE;

spark web :  

!image-2023-03-09-17-34-26-428.png!

  was:
spark config : 
spark.sql.adaptive.enabled true
spark.sql.adaptive.advisoryPartitionSizeInBytes 256m
spark.sql.adaptive.coalescePartitions.enabled true
spark.sql.adaptive.coalescePartitions.minPartitionNum 1
spark.sql.adaptive.coalescePartitions.initialPartitionNum 2000
spark.sql.extensions 
org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog
spark.sql.catalog.spark_catalog.type hive
 
Sql script: 
 
*create* *table* spark_catalog.test.iceberg5(id *bigint*, *data* string) 
*USING* iceberg;

*INSERT* *INTO* spark_catalog.test.iceberg5 *VALUES* (1, 'a'), (2, 'b'), (3, 
'c');

*update* test.iceberg5 *set* `data`{*}={*}'cc' *where* id{*}={*}3;

*drop* *table* spark_catalog.test.iceberg5 PURGE;

spark web :  

!image-2023-03-09-17-33-14-940.png!


> when enable AQE update or drop a small iceberg table should not generate as 
> much task as `spark.sql.adaptive.coalescePartitions.initialPartitionNum` 
> config
> ---
>
> Key: SPARK-42728
> URL: https://issues.apache.org/jira/browse/SPARK-42728
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 3.2.1, 3.2.3
> Environment: spark3.2.1 or 3.2.3; iceberg 0.14.1 or 1.1.0
>  
>Reporter: kevinshin
>Priority: Minor
> Attachments: image-2023-03-09-17-34-26-428.png
>
>
> spark config : 
> spark.sql.adaptive.enabled true
> spark.sql.adaptive.advisoryPartitionSizeInBytes 256m
> spark.sql.adaptive.coalescePartitions.enabled true
> spark.sql.adaptive.coalescePartitions.minPartitionNum 1
> spark.sql.adaptive.coalescePartitions.initialPartitionNum 2000
> spark.sql.extensions 
> org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
> spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog
> spark.sql.catalog.spark_catalog.type hive
>  
> Sql script: 
>  
> *create* *table* spark_catalog.test.iceberg5(id {*}bigint{*}, *data* string) 
> *USING* iceberg;
> *INSERT* *INTO* spark_catalog.test.iceberg5 *VALUES* (1, 'a'), (2, 'b'), (3, 
> 'c');
> *update* test.iceberg5 *set* `data`{*}={*}'cc' *where* id{*}={*}3;
> *drop* *table* spark_catalog.test.iceberg5 PURGE;
> spark web :  
> !image-2023-03-09-17-34-26-428.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-42728) when enable AQE update or drop a small iceberg table should not generate as much task as `spark.sql.adaptive.coalescePartitions.initialPartitionNum` config

2023-03-09 Thread kevinshin (Jira)



 [ 
https://issues.apache.org/jira/browse/SPARK-42728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevinshin updated SPARK-42728:
--
Attachment: image-2023-03-09-17-34-26-428.png

> when enable AQE update or drop a small iceberg table should not generate as 
> much task as `spark.sql.adaptive.coalescePartitions.initialPartitionNum` 
> config
> ---
>
> Key: SPARK-42728
> URL: https://issues.apache.org/jira/browse/SPARK-42728
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 3.2.1, 3.2.3
> Environment: spark3.2.1 or 3.2.3; iceberg 0.14.1 or 1.1.0
>  
>Reporter: kevinshin
>Priority: Minor
> Attachments: image-2023-03-09-17-34-26-428.png
>
>
> spark config : 
> spark.sql.adaptive.enabled true
> spark.sql.adaptive.advisoryPartitionSizeInBytes 256m
> spark.sql.adaptive.coalescePartitions.enabled true
> spark.sql.adaptive.coalescePartitions.minPartitionNum 1
> spark.sql.adaptive.coalescePartitions.initialPartitionNum 2000
> spark.sql.extensions 
> org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
> spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog
> spark.sql.catalog.spark_catalog.type hive
>  
> Sql script: 
>  
> *create* *table* spark_catalog.test.iceberg5(id *bigint*, *data* string) 
> *USING* iceberg;
> *INSERT* *INTO* spark_catalog.test.iceberg5 *VALUES* (1, 'a'), (2, 'b'), (3, 
> 'c');
> *update* test.iceberg5 *set* `data`{*}={*}'cc' *where* id{*}={*}3;
> *drop* *table* spark_catalog.test.iceberg5 PURGE;
> spark web :  
> !image-2023-03-09-17-33-14-940.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-42728) when enable AQE update or drop a small iceberg table should not generate as much task as `spark.sql.adaptive.coalescePartitions.initialPartitionNum` config

2023-03-09 Thread kevinshin (Jira)

kevinshin created SPARK-42728:
-

 Summary: when enable AQE update or drop a small iceberg table 
should not generate as much task as 
`spark.sql.adaptive.coalescePartitions.initialPartitionNum` config
 Key: SPARK-42728
 URL: https://issues.apache.org/jira/browse/SPARK-42728
 Project: Spark
  Issue Type: Improvement
  Components: SQL
Affects Versions: 3.2.3, 3.2.1
 Environment: spark3.2.1 or 3.2.3; iceberg 0.14.1 or 1.1.0

 
Reporter: kevinshin


spark config : 
spark.sql.adaptive.enabled true
spark.sql.adaptive.advisoryPartitionSizeInBytes 256m
spark.sql.adaptive.coalescePartitions.enabled true
spark.sql.adaptive.coalescePartitions.minPartitionNum 1
spark.sql.adaptive.coalescePartitions.initialPartitionNum 2000
spark.sql.extensions 
org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog
spark.sql.catalog.spark_catalog.type hive
 
Sql script: 
 
*create* *table* spark_catalog.test.iceberg5(id *bigint*, *data* string) 
*USING* iceberg;

*INSERT* *INTO* spark_catalog.test.iceberg5 *VALUES* (1, 'a'), (2, 'b'), (3, 
'c');

*update* test.iceberg5 *set* `data`{*}={*}'cc' *where* id{*}={*}3;

*drop* *table* spark_catalog.test.iceberg5 PURGE;

spark web :  

!image-2023-03-09-17-33-14-940.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-42650) insert overwrite table will casue table location lost if java.lang.ArithmeticException is thrown

2023-03-07 Thread kevinshin (Jira)



 [ 
https://issues.apache.org/jira/browse/SPARK-42650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevinshin updated SPARK-42650:
--
Summary: insert overwrite table will casue table location lost if 
java.lang.ArithmeticException is thrown  (was: link issue SPARK-42550)

> insert overwrite table will casue table location lost if 
> java.lang.ArithmeticException is thrown
> 
>
> Key: SPARK-42650
> URL: https://issues.apache.org/jira/browse/SPARK-42650
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
>Reporter: kevinshin
>Priority: Major
>
> When use 
> [KyuubiSparkSQLExtension|https://kyuubi.readthedocs.io/en/v1.6.1-incubating/extensions/engines/spark/]
>  and when a `insert overwrite` statment meet exception ，a no partion table's 
> home directory will lost ,partion table will lost partion directory.
>  
> my spark-defaults.conf config : 
> spark.sql.extensions org.apache.kyuubi.sql.KyuubiSparkSQLExtension
>  
> because I can't reopen SPARK-42550 , for detail and reproduce please 
> reference: 
> https://issues.apache.org/jira/browse/SPARK-42550
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-42650) link issue SPARK-42550

2023-03-06 Thread kevinshin (Jira)



[ 
https://issues.apache.org/jira/browse/SPARK-42650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17696810#comment-17696810
 ] 

kevinshin commented on SPARK-42650:
---

Thank you @[~yumwang] 

> link issue SPARK-42550
> --
>
> Key: SPARK-42650
> URL: https://issues.apache.org/jira/browse/SPARK-42650
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
>Reporter: kevinshin
>Priority: Major
>
> When use 
> [KyuubiSparkSQLExtension|https://kyuubi.readthedocs.io/en/v1.6.1-incubating/extensions/engines/spark/]
>  and when a `insert overwrite` statment meet exception ，a no partion table's 
> home directory will lost ,partion table will lost partion directory.
>  
> my spark-defaults.conf config : 
> spark.sql.extensions org.apache.kyuubi.sql.KyuubiSparkSQLExtension
>  
> because I can't reopen SPARK-42550 , for detail and reproduce please 
> reference: 
> https://issues.apache.org/jira/browse/SPARK-42550
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-42650) link issue SPARK-42550

2023-03-05 Thread kevinshin (Jira)



[ 
https://issues.apache.org/jira/browse/SPARK-42650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17696645#comment-17696645
 ] 

kevinshin commented on SPARK-42650:
---

Thanks [~ulysses] , I can reproduced it without any spark's extension.

> link issue SPARK-42550
> --
>
> Key: SPARK-42650
> URL: https://issues.apache.org/jira/browse/SPARK-42650
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
>Reporter: kevinshin
>Priority: Major
>
> When use 
> [KyuubiSparkSQLExtension|https://kyuubi.readthedocs.io/en/v1.6.1-incubating/extensions/engines/spark/]
>  and when a `insert overwrite` statment meet exception ，a no partion table's 
> home directory will lost ,partion table will lost partion directory.
>  
> my spark-defaults.conf config : 
> spark.sql.extensions org.apache.kyuubi.sql.KyuubiSparkSQLExtension
>  
> because I can't reopen SPARK-42550 , for detail and reproduce please 
> reference: 
> https://issues.apache.org/jira/browse/SPARK-42550
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-42650) link issue SPARK-42550

2023-03-02 Thread kevinshin (Jira)



[ 
https://issues.apache.org/jira/browse/SPARK-42650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17696037#comment-17696037
 ] 

kevinshin commented on SPARK-42650:
---

Spark and Kyuubi are both belong to apache.

May Apache community help to figure out the detail of this issue? Will this 
issue keep exist the next releases.

> link issue SPARK-42550
> --
>
> Key: SPARK-42650
> URL: https://issues.apache.org/jira/browse/SPARK-42650
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
>Reporter: kevinshin
>Priority: Major
>
> When use 
> [KyuubiSparkSQLExtension|https://kyuubi.readthedocs.io/en/v1.6.1-incubating/extensions/engines/spark/]
>  and when a `insert overwrite` statment meet exception ，a no partion table's 
> home directory will lost ,partion table will lost partion directory.
>  
> my spark-defaults.conf config : 
> spark.sql.extensions org.apache.kyuubi.sql.KyuubiSparkSQLExtension
>  
> because I can't reopen SPARK-42550 , for detail and reproduce please 
> reference: 
> https://issues.apache.org/jira/browse/SPARK-42550
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-42650) link issue SPARK-42550

2023-03-02 Thread kevinshin (Jira)



 [ 
https://issues.apache.org/jira/browse/SPARK-42650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevinshin updated SPARK-42650:
--
Description: 
When use 
[KyuubiSparkSQLExtension|https://kyuubi.readthedocs.io/en/v1.6.1-incubating/extensions/engines/spark/]
 and when a `insert overwrite` statment meet exception ，a no partion table's 
home directory will lost ,partion table will lost partion directory.
 
spark-defaults.conf: 
spark.sql.extensions org.apache.kyuubi.sql.KyuubiSparkSQLExtension
 

because I can't reopen SPARK-42550 , for detail and reproduce please reference: 

https://issues.apache.org/jira/browse/SPARK-42550

 

  was:
When use 
[KyuubiSparkSQLExtension|https://kyuubi.readthedocs.io/en/v1.6.1-incubating/extensions/engines/spark/]
 and when a{{ insert overwrite}} statment meet exception ，a no partion table's 
home directory will lost ,partion table will lost partion directory.
 
spark-defaults.conf: 
spark.sql.extensions org.apache.kyuubi.sql.KyuubiSparkSQLExtension
 

because I can't reopen SPARK-42550 , for detail and reproduce please reference: 

https://issues.apache.org/jira/browse/SPARK-42550

 


> link issue SPARK-42550
> --
>
> Key: SPARK-42650
> URL: https://issues.apache.org/jira/browse/SPARK-42650
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
>Reporter: kevinshin
>Priority: Major
>
> When use 
> [KyuubiSparkSQLExtension|https://kyuubi.readthedocs.io/en/v1.6.1-incubating/extensions/engines/spark/]
>  and when a `insert overwrite` statment meet exception ，a no partion table's 
> home directory will lost ,partion table will lost partion directory.
>  
> spark-defaults.conf: 
> spark.sql.extensions org.apache.kyuubi.sql.KyuubiSparkSQLExtension
>  
> because I can't reopen SPARK-42550 , for detail and reproduce please 
> reference: 
> https://issues.apache.org/jira/browse/SPARK-42550
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-42650) link issue SPARK-42550

2023-03-02 Thread kevinshin (Jira)



 [ 
https://issues.apache.org/jira/browse/SPARK-42650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevinshin updated SPARK-42650:
--
Description: 
When use 
[KyuubiSparkSQLExtension|https://kyuubi.readthedocs.io/en/v1.6.1-incubating/extensions/engines/spark/]
 and when a `insert overwrite` statment meet exception ，a no partion table's 
home directory will lost ,partion table will lost partion directory.
 
my spark-defaults.conf config : 
spark.sql.extensions org.apache.kyuubi.sql.KyuubiSparkSQLExtension
 

because I can't reopen SPARK-42550 , for detail and reproduce please reference: 

https://issues.apache.org/jira/browse/SPARK-42550

 

  was:
When use 
[KyuubiSparkSQLExtension|https://kyuubi.readthedocs.io/en/v1.6.1-incubating/extensions/engines/spark/]
 and when a `insert overwrite` statment meet exception ，a no partion table's 
home directory will lost ,partion table will lost partion directory.
 
spark-defaults.conf: 
spark.sql.extensions org.apache.kyuubi.sql.KyuubiSparkSQLExtension
 

because I can't reopen SPARK-42550 , for detail and reproduce please reference: 

https://issues.apache.org/jira/browse/SPARK-42550

 


> link issue SPARK-42550
> --
>
> Key: SPARK-42650
> URL: https://issues.apache.org/jira/browse/SPARK-42650
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
>Reporter: kevinshin
>Priority: Major
>
> When use 
> [KyuubiSparkSQLExtension|https://kyuubi.readthedocs.io/en/v1.6.1-incubating/extensions/engines/spark/]
>  and when a `insert overwrite` statment meet exception ，a no partion table's 
> home directory will lost ,partion table will lost partion directory.
>  
> my spark-defaults.conf config : 
> spark.sql.extensions org.apache.kyuubi.sql.KyuubiSparkSQLExtension
>  
> because I can't reopen SPARK-42550 , for detail and reproduce please 
> reference: 
> https://issues.apache.org/jira/browse/SPARK-42550
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-42650) link issue SPARK-42550

2023-03-02 Thread kevinshin (Jira)



 [ 
https://issues.apache.org/jira/browse/SPARK-42650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevinshin updated SPARK-42650:
--
Description: 
When use 
[KyuubiSparkSQLExtension|https://kyuubi.readthedocs.io/en/v1.6.1-incubating/extensions/engines/spark/]
 and when a{{ insert overwrite}} statment meet exception ，a no partion table's 
home directory will lost ,partion table will lost partion directory.
 
spark-defaults.conf: 
spark.sql.extensions org.apache.kyuubi.sql.KyuubiSparkSQLExtension
 

because I can't reopen SPARK-42550 , for detail and reproduce please reference: 

https://issues.apache.org/jira/browse/SPARK-42550

 

  was:https://issues.apache.org/jira/browse/SPARK-42550


> link issue SPARK-42550
> --
>
> Key: SPARK-42650
> URL: https://issues.apache.org/jira/browse/SPARK-42650
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
>Reporter: kevinshin
>Priority: Major
>
> When use 
> [KyuubiSparkSQLExtension|https://kyuubi.readthedocs.io/en/v1.6.1-incubating/extensions/engines/spark/]
>  and when a{{ insert overwrite}} statment meet exception ，a no partion 
> table's home directory will lost ,partion table will lost partion directory.
>  
> spark-defaults.conf: 
> spark.sql.extensions org.apache.kyuubi.sql.KyuubiSparkSQLExtension
>  
> because I can't reopen SPARK-42550 , for detail and reproduce please 
> reference: 
> https://issues.apache.org/jira/browse/SPARK-42550
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-42650) link issue SPARK-42550

2023-03-02 Thread kevinshin (Jira)

kevinshin created SPARK-42650:
-

 Summary: link issue SPARK-42550
 Key: SPARK-42650
 URL: https://issues.apache.org/jira/browse/SPARK-42650
 Project: Spark
  Issue Type: Bug
  Components: SQL
Affects Versions: 3.2.3
Reporter: kevinshin


https://issues.apache.org/jira/browse/SPARK-42550



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Closed] (SPARK-42550) table directory will lost on hdfs when `INSERT OVERWRITE` faild

2023-02-27 Thread kevinshin (Jira)



 [ 
https://issues.apache.org/jira/browse/SPARK-42550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevinshin closed SPARK-42550.
-

> table directory will lost on hdfs when `INSERT OVERWRITE` faild
> ---
>
> Key: SPARK-42550
> URL: https://issues.apache.org/jira/browse/SPARK-42550
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
> Environment: spark 3.2.3 / HDP 3.1.4
>Reporter: kevinshin
>Priority: Critical
> Attachments: image-2023-02-24-15-21-55-273.png, 
> image-2023-02-24-15-23-32-977.png, image-2023-02-24-15-25-57-770.png
>
>
> {color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during 
> execution, the table's directory will be deleted.  this is not happen in 
> spark 3.2.1.{color}
> {color:#4c9aff}for example: {color}
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 644164;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* 
> test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow 
> exception*{color}
> {color:#4c9aff}and then :{color}
> *select* * *from* test.spark32_overwrite; 
> {color:#4c9aff}will got error:{color}
> {color:#172b4d}java.io.FileNotFoundException{color}
> {color:#172b4d}!image-2023-02-24-15-21-55-273.png!{color}
> {color:#172b4d}the table's directory is losted. use `hdfs dfs -ls` cmd to 
> check:{color}
> !image-2023-02-24-15-23-32-977.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-42550) table directory will lost on hdfs when `INSERT OVERWRITE` faild

2023-02-27 Thread kevinshin (Jira)



[ 
https://issues.apache.org/jira/browse/SPARK-42550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17694270#comment-17694270
 ] 

kevinshin commented on SPARK-42550:
---

this is not spark's issue

> table directory will lost on hdfs when `INSERT OVERWRITE` faild
> ---
>
> Key: SPARK-42550
> URL: https://issues.apache.org/jira/browse/SPARK-42550
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
> Environment: spark 3.2.3 / HDP 3.1.4
>Reporter: kevinshin
>Priority: Critical
> Attachments: image-2023-02-24-15-21-55-273.png, 
> image-2023-02-24-15-23-32-977.png, image-2023-02-24-15-25-57-770.png
>
>
> {color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during 
> execution, the table's directory will be deleted.  this is not happen in 
> spark 3.2.1.{color}
> {color:#4c9aff}for example: {color}
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 644164;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* 
> test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow 
> exception*{color}
> {color:#4c9aff}and then :{color}
> *select* * *from* test.spark32_overwrite; 
> {color:#4c9aff}will got error:{color}
> {color:#172b4d}java.io.FileNotFoundException{color}
> {color:#172b4d}!image-2023-02-24-15-21-55-273.png!{color}
> {color:#172b4d}the table's directory is losted. use `hdfs dfs -ls` cmd to 
> check:{color}
> !image-2023-02-24-15-23-32-977.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-42550) table directory will lost on hdfs when `INSERT OVERWRITE` faild

2023-02-25 Thread kevinshin (Jira)



[ 
https://issues.apache.org/jira/browse/SPARK-42550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17693467#comment-17693467
 ] 

kevinshin commented on SPARK-42550:
---

https://github.com/apache/kyuubi/issues/4416

> table directory will lost on hdfs when `INSERT OVERWRITE` faild
> ---
>
> Key: SPARK-42550
> URL: https://issues.apache.org/jira/browse/SPARK-42550
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
> Environment: spark 3.2.3 / HDP 3.1.4
>Reporter: kevinshin
>Priority: Critical
> Attachments: image-2023-02-24-15-21-55-273.png, 
> image-2023-02-24-15-23-32-977.png, image-2023-02-24-15-25-57-770.png
>
>
> {color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during 
> execution, the table's directory will be deleted.  this is not happen in 
> spark 3.2.1.{color}
> {color:#4c9aff}for example: {color}
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 644164;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* 
> test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow 
> exception*{color}
> {color:#4c9aff}and then :{color}
> *select* * *from* test.spark32_overwrite; 
> {color:#4c9aff}will got error:{color}
> {color:#172b4d}java.io.FileNotFoundException{color}
> {color:#172b4d}!image-2023-02-24-15-21-55-273.png!{color}
> {color:#172b4d}the table's directory is losted. use `hdfs dfs -ls` cmd to 
> check:{color}
> !image-2023-02-24-15-23-32-977.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-42550) table directory will lost on hdfs when `INSERT OVERWRITE` faild

2023-02-25 Thread kevinshin (Jira)



[ 
https://issues.apache.org/jira/browse/SPARK-42550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17693457#comment-17693457
 ] 

kevinshin commented on SPARK-42550:
---

It seems like my environment problem, I can't reproduce this issue too after 
change to a new environment, than you @ [~yumwang] 

> table directory will lost on hdfs when `INSERT OVERWRITE` faild
> ---
>
> Key: SPARK-42550
> URL: https://issues.apache.org/jira/browse/SPARK-42550
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
> Environment: spark 3.2.3 / HDP 3.1.4
>Reporter: kevinshin
>Priority: Critical
> Attachments: image-2023-02-24-15-21-55-273.png, 
> image-2023-02-24-15-23-32-977.png, image-2023-02-24-15-25-57-770.png
>
>
> {color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during 
> execution, the table's directory will be deleted.  this is not happen in 
> spark 3.2.1.{color}
> {color:#4c9aff}for example: {color}
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 644164;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* 
> test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow 
> exception*{color}
> {color:#4c9aff}and then :{color}
> *select* * *from* test.spark32_overwrite; 
> {color:#4c9aff}will got error:{color}
> {color:#172b4d}java.io.FileNotFoundException{color}
> {color:#172b4d}!image-2023-02-24-15-21-55-273.png!{color}
> {color:#172b4d}the table's directory is losted. use `hdfs dfs -ls` cmd to 
> check:{color}
> !image-2023-02-24-15-23-32-977.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-42550) table directory will lost on hdfs when `INSERT OVERWRITE` faild

2023-02-23 Thread kevinshin (Jira)



[ 
https://issues.apache.org/jira/browse/SPARK-42550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17693044#comment-17693044
 ] 

kevinshin edited comment on SPARK-42550 at 2/24/23 7:44 AM:


when try to 
refresh table test.spark32_overwrite
!image-2023-02-24-15-25-57-770.png!


was (Author: JIRAUSER281772):
when try to 
refresh table test.spark32_overwrite
!image-2023-02-24-15-26-01-846.png!

> table directory will lost on hdfs when `INSERT OVERWRITE` faild
> ---
>
> Key: SPARK-42550
> URL: https://issues.apache.org/jira/browse/SPARK-42550
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
> Environment: spark 3.2.3 / HDP 3.1.4
>Reporter: kevinshin
>Priority: Critical
> Attachments: image-2023-02-24-15-21-55-273.png, 
> image-2023-02-24-15-23-32-977.png, image-2023-02-24-15-25-57-770.png
>
>
> {color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during 
> execution, the table's directory will be deleted.  this is not happen in 
> spark 3.2.1.{color}
> {color:#4c9aff}for example: {color}
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 644164;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* 
> test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow 
> exception*{color}
> {color:#4c9aff}and then :{color}
> *select* * *from* test.spark32_overwrite; 
> {color:#4c9aff}will got error:{color}
> {color:#172b4d}java.io.FileNotFoundException{color}
> {color:#172b4d}!image-2023-02-24-15-21-55-273.png!{color}
> {color:#172b4d}the table's directory is losted. use `hdfs dfs -ls` cmd to 
> check:{color}
> !image-2023-02-24-15-23-32-977.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-42550) table directory will lost on hdfs when `INSERT OVERWRITE` faild

2023-02-23 Thread kevinshin (Jira)



 [ 
https://issues.apache.org/jira/browse/SPARK-42550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevinshin updated SPARK-42550:
--
Attachment: (was: image-2023-02-24-15-26-01-846.png)

> table directory will lost on hdfs when `INSERT OVERWRITE` faild
> ---
>
> Key: SPARK-42550
> URL: https://issues.apache.org/jira/browse/SPARK-42550
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
> Environment: spark 3.2.3 / HDP 3.1.4
>Reporter: kevinshin
>Priority: Critical
> Attachments: image-2023-02-24-15-21-55-273.png, 
> image-2023-02-24-15-23-32-977.png, image-2023-02-24-15-25-57-770.png
>
>
> {color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during 
> execution, the table's directory will be deleted.  this is not happen in 
> spark 3.2.1.{color}
> {color:#4c9aff}for example: {color}
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 644164;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* 
> test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow 
> exception*{color}
> {color:#4c9aff}and then :{color}
> *select* * *from* test.spark32_overwrite; 
> {color:#4c9aff}will got error:{color}
> {color:#172b4d}java.io.FileNotFoundException{color}
> {color:#172b4d}!image-2023-02-24-15-21-55-273.png!{color}
> {color:#172b4d}the table's directory is losted. use `hdfs dfs -ls` cmd to 
> check:{color}
> !image-2023-02-24-15-23-32-977.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-42550) table directory will lost on hdfs when `INSERT OVERWRITE` faild

2023-02-23 Thread kevinshin (Jira)



 [ 
https://issues.apache.org/jira/browse/SPARK-42550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevinshin updated SPARK-42550:
--
Attachment: (was: image-2023-02-24-15-22-45-876.png)

> table directory will lost on hdfs when `INSERT OVERWRITE` faild
> ---
>
> Key: SPARK-42550
> URL: https://issues.apache.org/jira/browse/SPARK-42550
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
> Environment: spark 3.2.3 / HDP 3.1.4
>Reporter: kevinshin
>Priority: Critical
> Attachments: image-2023-02-24-15-21-55-273.png, 
> image-2023-02-24-15-23-32-977.png, image-2023-02-24-15-25-57-770.png
>
>
> {color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during 
> execution, the table's directory will be deleted.  this is not happen in 
> spark 3.2.1.{color}
> {color:#4c9aff}for example: {color}
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 644164;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* 
> test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow 
> exception*{color}
> {color:#4c9aff}and then :{color}
> *select* * *from* test.spark32_overwrite; 
> {color:#4c9aff}will got error:{color}
> {color:#172b4d}java.io.FileNotFoundException{color}
> {color:#172b4d}!image-2023-02-24-15-21-55-273.png!{color}
> {color:#172b4d}the table's directory is losted. use `hdfs dfs -ls` cmd to 
> check:{color}
> !image-2023-02-24-15-23-32-977.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-42550) table directory will lost on hdfs when `INSERT OVERWRITE` faild

2023-02-23 Thread kevinshin (Jira)



[ 
https://issues.apache.org/jira/browse/SPARK-42550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17693053#comment-17693053
 ] 

kevinshin commented on SPARK-42550:
---

the diffrent is : 

1、spark3.2.1 will delete table's data files but spark3.2.3 delete the table's 
home directory.

2、spark3.2.1, the FileNotFoundException only happen when table create at the 
first time and '{*}INSERT{*} OVERWRITE' statment have exception in the same 
session. Because the table's home directory still on hdfs , use `refresh table` 
is ok. 

> table directory will lost on hdfs when `INSERT OVERWRITE` faild
> ---
>
> Key: SPARK-42550
> URL: https://issues.apache.org/jira/browse/SPARK-42550
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
> Environment: spark 3.2.3 / HDP 3.1.4
>Reporter: kevinshin
>Priority: Critical
> Attachments: image-2023-02-24-15-21-55-273.png, 
> image-2023-02-24-15-22-45-876.png, image-2023-02-24-15-23-32-977.png, 
> image-2023-02-24-15-25-57-770.png, image-2023-02-24-15-26-01-846.png
>
>
> {color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during 
> execution, the table's directory will be deleted.  this is not happen in 
> spark 3.2.1.{color}
> {color:#4c9aff}for example: {color}
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 644164;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* 
> test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow 
> exception*{color}
> {color:#4c9aff}and then :{color}
> *select* * *from* test.spark32_overwrite; 
> {color:#4c9aff}will got error:{color}
> {color:#172b4d}java.io.FileNotFoundException{color}
> {color:#172b4d}!image-2023-02-24-15-21-55-273.png!{color}
> {color:#172b4d}the table's directory is losted. use `hdfs dfs -ls` cmd to 
> check:{color}
> !image-2023-02-24-15-23-32-977.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-42550) table directory will lost on hdfs when `INSERT OVERWRITE` faild

2023-02-23 Thread kevinshin (Jira)



[ 
https://issues.apache.org/jira/browse/SPARK-42550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17693044#comment-17693044
 ] 

kevinshin commented on SPARK-42550:
---

when try to 
refresh table test.spark32_overwrite
!image-2023-02-24-15-26-01-846.png!

> table directory will lost on hdfs when `INSERT OVERWRITE` faild
> ---
>
> Key: SPARK-42550
> URL: https://issues.apache.org/jira/browse/SPARK-42550
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
> Environment: spark 3.2.3 / HDP 3.1.4
>Reporter: kevinshin
>Priority: Critical
> Attachments: image-2023-02-24-15-21-55-273.png, 
> image-2023-02-24-15-22-45-876.png, image-2023-02-24-15-23-32-977.png, 
> image-2023-02-24-15-25-57-770.png, image-2023-02-24-15-26-01-846.png
>
>
> {color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during 
> execution, the table's directory will be deleted.  this is not happen in 
> spark 3.2.1.{color}
> {color:#4c9aff}for example: {color}
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 644164;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* 
> test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow 
> exception*{color}
> {color:#4c9aff}and then :{color}
> *select* * *from* test.spark32_overwrite; 
> {color:#4c9aff}will got error:{color}
> {color:#172b4d}java.io.FileNotFoundException{color}
> {color:#172b4d}!image-2023-02-24-15-21-55-273.png!{color}
> {color:#172b4d}the table's directory is losted. use `hdfs dfs -ls` cmd to 
> check:{color}
> !image-2023-02-24-15-23-32-977.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-42550) table directory will lost on hdfs when `INSERT OVERWRITE` faild

2023-02-23 Thread kevinshin (Jira)



 [ 
https://issues.apache.org/jira/browse/SPARK-42550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevinshin updated SPARK-42550:
--
Attachment: image-2023-02-24-15-26-01-846.png

> table directory will lost on hdfs when `INSERT OVERWRITE` faild
> ---
>
> Key: SPARK-42550
> URL: https://issues.apache.org/jira/browse/SPARK-42550
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
> Environment: spark 3.2.3 / HDP 3.1.4
>Reporter: kevinshin
>Priority: Critical
> Attachments: image-2023-02-24-15-21-55-273.png, 
> image-2023-02-24-15-22-45-876.png, image-2023-02-24-15-23-32-977.png, 
> image-2023-02-24-15-25-57-770.png, image-2023-02-24-15-26-01-846.png
>
>
> {color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during 
> execution, the table's directory will be deleted.  this is not happen in 
> spark 3.2.1.{color}
> {color:#4c9aff}for example: {color}
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 644164;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* 
> test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow 
> exception*{color}
> {color:#4c9aff}and then :{color}
> *select* * *from* test.spark32_overwrite; 
> {color:#4c9aff}will got error:{color}
> {color:#172b4d}java.io.FileNotFoundException{color}
> {color:#172b4d}!image-2023-02-24-15-21-55-273.png!{color}
> {color:#172b4d}the table's directory is losted. use `hdfs dfs -ls` cmd to 
> check:{color}
> !image-2023-02-24-15-23-32-977.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-42550) table directory will lost on hdfs when `INSERT OVERWRITE` faild

2023-02-23 Thread kevinshin (Jira)



 [ 
https://issues.apache.org/jira/browse/SPARK-42550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevinshin updated SPARK-42550:
--
Attachment: image-2023-02-24-15-25-57-770.png

> table directory will lost on hdfs when `INSERT OVERWRITE` faild
> ---
>
> Key: SPARK-42550
> URL: https://issues.apache.org/jira/browse/SPARK-42550
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
> Environment: spark 3.2.3 / HDP 3.1.4
>Reporter: kevinshin
>Priority: Critical
> Attachments: image-2023-02-24-15-21-55-273.png, 
> image-2023-02-24-15-22-45-876.png, image-2023-02-24-15-23-32-977.png, 
> image-2023-02-24-15-25-57-770.png, image-2023-02-24-15-26-01-846.png
>
>
> {color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during 
> execution, the table's directory will be deleted.  this is not happen in 
> spark 3.2.1.{color}
> {color:#4c9aff}for example: {color}
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 644164;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* 
> test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow 
> exception*{color}
> {color:#4c9aff}and then :{color}
> *select* * *from* test.spark32_overwrite; 
> {color:#4c9aff}will got error:{color}
> {color:#172b4d}java.io.FileNotFoundException{color}
> {color:#172b4d}!image-2023-02-24-15-21-55-273.png!{color}
> {color:#172b4d}the table's directory is losted. use `hdfs dfs -ls` cmd to 
> check:{color}
> !image-2023-02-24-15-23-32-977.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-42550) table directory will lost on hdfs when `INSERT OVERWRITE` faild

2023-02-23 Thread kevinshin (Jira)



 [ 
https://issues.apache.org/jira/browse/SPARK-42550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevinshin updated SPARK-42550:
--
Description: 
{color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during 
execution, the table's directory will be deleted.  this is not happen in spark 
3.2.1.{color}

{color:#4c9aff}for example: {color}

*CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) 
STORED *AS* ORC;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;

*CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) STORED 
*AS* ORC;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 644164;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* 
test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow 
exception*{color}

{color:#4c9aff}and then :{color}

*select* * *from* test.spark32_overwrite; 

{color:#4c9aff}will got error:{color}

{color:#172b4d}java.io.FileNotFoundException{color}

{color:#172b4d}!image-2023-02-24-15-21-55-273.png!{color}

{color:#172b4d}the table's directory is losted. use `hdfs dfs -ls` cmd to 
check:{color}

!image-2023-02-24-15-23-32-977.png!

  was:
{color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during 
execution, the table's directory will be deleted.  this is not happen in spark 
3.2.1.{color}

{color:#4c9aff}for example: {color}

*CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) 
STORED *AS* ORC;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;

*CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) STORED 
*AS* ORC;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 644164;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* 
test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow 
exception*{color}

{color:#4c9aff}and then :{color}

*select* * *from* test.spark32_overwrite; 

{color:#4c9aff}will got error:{color}

{color:#172b4d}java.io.FileNotFoundException{color}

{color:#172b4d}!image-2023-02-24-15-21-55-273.png!{color}

{color:#172b4d}the table's directory is losted.{color}

{color:#172b4d}!image-2023-02-24-15-22-45-876.png!{color}


> table directory will lost on hdfs when `INSERT OVERWRITE` faild
> ---
>
> Key: SPARK-42550
> URL: https://issues.apache.org/jira/browse/SPARK-42550
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
> Environment: spark 3.2.3 / HDP 3.1.4
>Reporter: kevinshin
>Priority: Critical
> Attachments: image-2023-02-24-15-21-55-273.png, 
> image-2023-02-24-15-22-45-876.png, image-2023-02-24-15-23-32-977.png
>
>
> {color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during 
> execution, the table's directory will be deleted.  this is not happen in 
> spark 3.2.1.{color}
> {color:#4c9aff}for example: {color}
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 644164;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* 
> test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow 
> exception*{color}
> {color:#4c9aff}and then :{color}
> *select* * *from* test.spark32_overwrite; 
> {color:#4c9aff}will got error:{color}
> {color:#172b4d}java.io.FileNotFoundException{color}
> {color:#172b4d}!image-2023-02-24-15-21-55-273.png!{color}
> {color:#172b4d}the table's directory is losted. use `hdfs dfs -ls` cmd to 
> check:{color}
> !image-2023-02-24-15-23-32-977.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-42550) table directory will lost on hdfs when `INSERT OVERWRITE` faild

2023-02-23 Thread kevinshin (Jira)



 [ 
https://issues.apache.org/jira/browse/SPARK-42550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevinshin updated SPARK-42550:
--
Attachment: image-2023-02-24-15-23-32-977.png

> table directory will lost on hdfs when `INSERT OVERWRITE` faild
> ---
>
> Key: SPARK-42550
> URL: https://issues.apache.org/jira/browse/SPARK-42550
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
> Environment: spark 3.2.3 / HDP 3.1.4
>Reporter: kevinshin
>Priority: Critical
> Attachments: image-2023-02-24-15-21-55-273.png, 
> image-2023-02-24-15-22-45-876.png, image-2023-02-24-15-23-32-977.png
>
>
> {color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during 
> execution, the table's directory will be deleted.  this is not happen in 
> spark 3.2.1.{color}
> {color:#4c9aff}for example: {color}
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 644164;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* 
> test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow 
> exception*{color}
> {color:#4c9aff}and then :{color}
> *select* * *from* test.spark32_overwrite; 
> {color:#4c9aff}will got error:{color}
> {color:#172b4d}java.io.FileNotFoundException{color}
> {color:#172b4d}!image-2023-02-24-15-21-55-273.png!{color}
> {color:#172b4d}the table's directory is losted.{color}
> {color:#172b4d}!image-2023-02-24-15-22-45-876.png!{color}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-42550) table directory will lost on hdfs when `INSERT OVERWRITE` faild

2023-02-23 Thread kevinshin (Jira)



 [ 
https://issues.apache.org/jira/browse/SPARK-42550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevinshin updated SPARK-42550:
--
Description: 
{color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during 
execution, the table's directory will be deleted.  this is not happen in spark 
3.2.1.{color}

{color:#4c9aff}for example: {color}

*CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) 
STORED *AS* ORC;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;

*CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) STORED 
*AS* ORC;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 644164;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* 
test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow 
exception*{color}

{color:#4c9aff}and then :{color}

*select* * *from* test.spark32_overwrite; 

{color:#4c9aff}will got error:{color}

{color:#172b4d}java.io.FileNotFoundException{color}

{color:#172b4d}!image-2023-02-24-15-21-55-273.png!{color}

{color:#172b4d}the table's directory is losted.{color}

{color:#172b4d}!image-2023-02-24-15-22-45-876.png!{color}

  was:
{color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during 
execution, the table's directory will be deleted.  this is not happen in spark 
3.2.1.{color}

{color:#4c9aff}for example: {color}

*CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) 
STORED *AS* ORC;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;

*CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) STORED 
*AS* ORC;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite2 *select* 644164;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* 
test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow 
exception*{color}

{color:#4c9aff}and then :{color}

*select*  *  *from* test.spark32_overwrite; 

{color:#4c9aff}will got error:{color}

{color:#172b4d}java.io.FileNotFoundException{color}

{color:#172b4d}the table's directory is losted.{color}


> table directory will lost on hdfs when `INSERT OVERWRITE` faild
> ---
>
> Key: SPARK-42550
> URL: https://issues.apache.org/jira/browse/SPARK-42550
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
> Environment: spark 3.2.3 / HDP 3.1.4
>Reporter: kevinshin
>Priority: Critical
> Attachments: image-2023-02-24-15-21-55-273.png, 
> image-2023-02-24-15-22-45-876.png
>
>
> {color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during 
> execution, the table's directory will be deleted.  this is not happen in 
> spark 3.2.1.{color}
> {color:#4c9aff}for example: {color}
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 644164;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* 
> test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow 
> exception*{color}
> {color:#4c9aff}and then :{color}
> *select* * *from* test.spark32_overwrite; 
> {color:#4c9aff}will got error:{color}
> {color:#172b4d}java.io.FileNotFoundException{color}
> {color:#172b4d}!image-2023-02-24-15-21-55-273.png!{color}
> {color:#172b4d}the table's directory is losted.{color}
> {color:#172b4d}!image-2023-02-24-15-22-45-876.png!{color}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-42550) table directory will lost on hdfs when `INSERT OVERWRITE` faild

2023-02-23 Thread kevinshin (Jira)



 [ 
https://issues.apache.org/jira/browse/SPARK-42550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevinshin updated SPARK-42550:
--
Attachment: image-2023-02-24-15-22-45-876.png

> table directory will lost on hdfs when `INSERT OVERWRITE` faild
> ---
>
> Key: SPARK-42550
> URL: https://issues.apache.org/jira/browse/SPARK-42550
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
> Environment: spark 3.2.3 / HDP 3.1.4
>Reporter: kevinshin
>Priority: Critical
> Attachments: image-2023-02-24-15-21-55-273.png, 
> image-2023-02-24-15-22-45-876.png
>
>
> {color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during 
> execution, the table's directory will be deleted.  this is not happen in 
> spark 3.2.1.{color}
> {color:#4c9aff}for example: {color}
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite2 *select* 644164;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* 
> test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow 
> exception*{color}
> {color:#4c9aff}and then :{color}
> *select*  *  *from* test.spark32_overwrite; 
> {color:#4c9aff}will got error:{color}
> {color:#172b4d}java.io.FileNotFoundException{color}
> {color:#172b4d}the table's directory is losted.{color}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-42550) table directory will lost on hdfs when `INSERT OVERWRITE` faild

2023-02-23 Thread kevinshin (Jira)



 [ 
https://issues.apache.org/jira/browse/SPARK-42550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevinshin updated SPARK-42550:
--
Attachment: image-2023-02-24-15-21-55-273.png

> table directory will lost on hdfs when `INSERT OVERWRITE` faild
> ---
>
> Key: SPARK-42550
> URL: https://issues.apache.org/jira/browse/SPARK-42550
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
> Environment: spark 3.2.3 / HDP 3.1.4
>Reporter: kevinshin
>Priority: Critical
> Attachments: image-2023-02-24-15-21-55-273.png
>
>
> {color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during 
> execution, the table's directory will be deleted.  this is not happen in 
> spark 3.2.1.{color}
> {color:#4c9aff}for example: {color}
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite2 *select* 644164;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* 
> test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow 
> exception*{color}
> {color:#4c9aff}and then :{color}
> *select*  *  *from* test.spark32_overwrite; 
> {color:#4c9aff}will got error:{color}
> {color:#172b4d}java.io.FileNotFoundException{color}
> {color:#172b4d}the table's directory is losted.{color}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-42550) table directory will lost on hdfs when `INSERT OVERWRITE` faild

2023-02-23 Thread kevinshin (Jira)



 [ 
https://issues.apache.org/jira/browse/SPARK-42550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevinshin updated SPARK-42550:
--
Description: 
{color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during 
execution, the table's directory will be deleted.  this is not happen in spark 
3.2.1.{color}

{color:#4c9aff}for example: {color}

*CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) 
STORED *AS* ORC;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;

*CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) STORED 
*AS* ORC;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 644164;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* 
test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow 
exception*{color}

{color:#4c9aff}and then :{color}

*select* * *from* test.spark32_overwrite; 

{color:#4c9aff}will got error:{color}

{color:#172b4d}java.io.FileNotFoundException{color}

{color:#172b4d}the table's directory is losted.{color}

  was:
{color:#4c9aff}when a  `*INSERT* OVERWRITE *TABLE`* statment faild during 
execution, the table's directory will be deleted.  this is not happen in spark 
3.2.1.{color}

{color:#4c9aff}for example: {color}

*CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 *int*) STORED 
*AS* ORC;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;

*CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) STORED 
*AS* ORC;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 644164;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* 
test.spark32_overwrite2;    -- {color:#de350b}*this will got Casting overflow 
exception*{color}

{color:#4c9aff}and then :{color}

*select* *** *from* test.spark32_overwrite; 

{color:#4c9aff}will got error:{color}

{color:#172b4d}java.io.FileNotFoundException{color}

{color:#172b4d}the table's directory is losted.{color}


> table directory will lost on hdfs when `INSERT OVERWRITE` faild
> ---
>
> Key: SPARK-42550
> URL: https://issues.apache.org/jira/browse/SPARK-42550
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.2.3
> Environment: spark 3.2.3 / HDP 3.1.4
>Reporter: kevinshin
>Priority: Critical
>
> {color:#4c9aff}when a  `{*}INSERT{*} OVERWRITE *TABLE`* statment faild during 
> execution, the table's directory will be deleted.  this is not happen in 
> spark 3.2.1.{color}
> {color:#4c9aff}for example: {color}
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 {*}int{*}) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;
> *CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) 
> STORED *AS* ORC;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 644164;
> *INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* 
> test.spark32_overwrite2;    – {color:#de350b}*this will got Casting overflow 
> exception*{color}
> {color:#4c9aff}and then :{color}
> *select* * *from* test.spark32_overwrite; 
> {color:#4c9aff}will got error:{color}
> {color:#172b4d}java.io.FileNotFoundException{color}
> {color:#172b4d}the table's directory is losted.{color}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-42550) table directory will lost on hdfs when `INSERT OVERWRITE` faild

2023-02-23 Thread kevinshin (Jira)

kevinshin created SPARK-42550:
-

 Summary: table directory will lost on hdfs when `INSERT OVERWRITE` 
faild
 Key: SPARK-42550
 URL: https://issues.apache.org/jira/browse/SPARK-42550
 Project: Spark
  Issue Type: Bug
  Components: SQL
Affects Versions: 3.2.3
 Environment: spark 3.2.3 / HDP 3.1.4
Reporter: kevinshin


{color:#4c9aff}when a  `*INSERT* OVERWRITE *TABLE`* statment faild during 
execution, the table's directory will be deleted.  this is not happen in spark 
3.2.1.{color}

{color:#4c9aff}for example: {color}

*CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite(amt1 *int*) STORED 
*AS* ORC;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 128;

*CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark32_overwrite2(amt1 long) STORED 
*AS* ORC;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* 644164;

*INSERT* OVERWRITE *TABLE* test.spark32_overwrite *select* amt1 *from* 
test.spark32_overwrite2;    -- {color:#de350b}*this will got Casting overflow 
exception*{color}

{color:#4c9aff}and then :{color}

*select* *** *from* test.spark32_overwrite; 

{color:#4c9aff}will got error:{color}

{color:#172b4d}java.io.FileNotFoundException{color}

{color:#172b4d}the table's directory is losted.{color}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-42473) An explicit cast will be needed when INSERT OVERWRITE SELECT UNION ALL

2023-02-19 Thread kevinshin (Jira)



[ 
https://issues.apache.org/jira/browse/SPARK-42473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17690965#comment-17690965
 ] 

kevinshin edited comment on SPARK-42473 at 2/20/23 1:47 AM:


[~yumwang]  'What is your test.spark33_decimal_orc column type?'

{color:#4c9aff}*CREATE* *TABLE* *IF* *NOT* *EXISTS* 
test.spark33_decimal_orc({color}

{color:#4c9aff}   amt1        {*}decimal{*}(20,8),{color}

{color:#4c9aff}   amt2        {*}decimal{*}(20,8){color}

{color:#4c9aff})STORED *AS* ORC;{color}


was (Author: JIRAUSER281772):
[~yumwang]  'What is your test.spark33_decimal_orc column type?'

*CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark33_decimal_orc(

   amt1        *decimal*(20,8),

   amt2        *decimal*(20,8)

)STORED *AS* ORC;

> An explicit cast will be needed when INSERT OVERWRITE SELECT UNION ALL
> --
>
> Key: SPARK-42473
> URL: https://issues.apache.org/jira/browse/SPARK-42473
> Project: Spark
>  Issue Type: Bug
>  Components: Optimizer
>Affects Versions: 3.3.1
> Environment: spark 3.3.1
>Reporter: kevinshin
>Priority: Major
>
> *when 'union all' and one select statement use* *Literal as column value , 
> the other* *select statement  has computed expression at the same column , 
> then the whole statement will compile failed. A explicit cast will be needed.*
> for example:
> {color:#4c9aff}explain{color}
> {color:#4c9aff}*INSERT* OVERWRITE *TABLE* test.spark33_decimal_orc{color}
> {color:#4c9aff}*select* *null* *as* amt1, {*}cast{*}('256.99' *as* 
> {*}decimal{*}(20,8)) *as* amt2{color}
> {color:#4c9aff}*union* *all*{color}
> {color:#4c9aff}*select* {*}cast{*}('200.99' *as* 
> {*}decimal{*}(20,8)){*}/{*}100 *as* amt1,{*}cast{*}('256.99' *as* 
> {*}decimal{*}(20,8)) *as* amt2;{color}
> *will got error :* 
> org.apache.spark.{*}sql{*}.catalyst.expressions.Literal cannot be *cast* *to* 
> org.apache.spark.{*}sql{*}.catalyst.expressions.AnsiCast
> The SQL will need to change to : 
> {color:#4c9aff}explain{color}
> {color:#4c9aff}*INSERT* OVERWRITE *TABLE* test.spark33_decimal_orc{color}
> {color:#4c9aff}*select* *null* *as* amt1,{*}cast{*}('256.99' *as* 
> {*}decimal{*}(20,8)) *as* amt2{color}
> {color:#4c9aff}*union* *all*{color}
> {color:#4c9aff}*select* {color:#de350b}{*}cast{*}({color}{*}cast{*}('200.99' 
> *as* {*}decimal{*}(20,8)){*}/{*}100 *as* 
> {*}decimal{*}(20,8){color:#de350b}){color} *as* amt1,{*}cast{*}('256.99' *as* 
> {*}decimal{*}(20,8)) *as* amt2;{color}
>  
> *but this is not need in spark3.2.1 , is this a bug for spark3.3.1 ?* 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-42473) An explicit cast will be needed when INSERT OVERWRITE SELECT UNION ALL

2023-02-19 Thread kevinshin (Jira)



[ 
https://issues.apache.org/jira/browse/SPARK-42473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17690965#comment-17690965
 ] 

kevinshin commented on SPARK-42473:
---

[~yumwang]  'What is your test.spark33_decimal_orc column type?'

*CREATE* *TABLE* *IF* *NOT* *EXISTS* test.spark33_decimal_orc(

   amt1        *decimal*(20,8),

   amt2        *decimal*(20,8)

)STORED *AS* ORC;

> An explicit cast will be needed when INSERT OVERWRITE SELECT UNION ALL
> --
>
> Key: SPARK-42473
> URL: https://issues.apache.org/jira/browse/SPARK-42473
> Project: Spark
>  Issue Type: Bug
>  Components: Optimizer
>Affects Versions: 3.3.1
> Environment: spark 3.3.1
>Reporter: kevinshin
>Priority: Major
>
> *when 'union all' and one select statement use* *Literal as column value , 
> the other* *select statement  has computed expression at the same column , 
> then the whole statement will compile failed. A explicit cast will be needed.*
> for example:
> {color:#4c9aff}explain{color}
> {color:#4c9aff}*INSERT* OVERWRITE *TABLE* test.spark33_decimal_orc{color}
> {color:#4c9aff}*select* *null* *as* amt1, {*}cast{*}('256.99' *as* 
> {*}decimal{*}(20,8)) *as* amt2{color}
> {color:#4c9aff}*union* *all*{color}
> {color:#4c9aff}*select* {*}cast{*}('200.99' *as* 
> {*}decimal{*}(20,8)){*}/{*}100 *as* amt1,{*}cast{*}('256.99' *as* 
> {*}decimal{*}(20,8)) *as* amt2;{color}
> *will got error :* 
> org.apache.spark.{*}sql{*}.catalyst.expressions.Literal cannot be *cast* *to* 
> org.apache.spark.{*}sql{*}.catalyst.expressions.AnsiCast
> The SQL will need to change to : 
> {color:#4c9aff}explain{color}
> {color:#4c9aff}*INSERT* OVERWRITE *TABLE* test.spark33_decimal_orc{color}
> {color:#4c9aff}*select* *null* *as* amt1,{*}cast{*}('256.99' *as* 
> {*}decimal{*}(20,8)) *as* amt2{color}
> {color:#4c9aff}*union* *all*{color}
> {color:#4c9aff}*select* {color:#de350b}{*}cast{*}({color}{*}cast{*}('200.99' 
> *as* {*}decimal{*}(20,8)){*}/{*}100 *as* 
> {*}decimal{*}(20,8){color:#de350b}){color} *as* amt1,{*}cast{*}('256.99' *as* 
> {*}decimal{*}(20,8)) *as* amt2;{color}
>  
> *but this is not need in spark3.2.1 , is this a bug for spark3.3.1 ?* 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-42473) An explicit cast will be needed when INSERT OVERWRITE SELECT UNION ALL

2023-02-16 Thread kevinshin (Jira)



 [ 
https://issues.apache.org/jira/browse/SPARK-42473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevinshin updated SPARK-42473:
--
Description: 
*when 'union all' and one select statement use* *Literal as column value , the 
other* *select statement  has computed expression at the same column , then the 
whole statement will compile failed. A explicit cast will be needed.*

for example:

{color:#4c9aff}explain{color}

{color:#4c9aff}*INSERT* OVERWRITE *TABLE* test.spark33_decimal_orc{color}

{color:#4c9aff}*select* *null* *as* amt1, {*}cast{*}('256.99' *as* 
{*}decimal{*}(20,8)) *as* amt2{color}

{color:#4c9aff}*union* *all*{color}

{color:#4c9aff}*select* {*}cast{*}('200.99' *as* {*}decimal{*}(20,8)){*}/{*}100 
*as* amt1,{*}cast{*}('256.99' *as* {*}decimal{*}(20,8)) *as* amt2;{color}

*will got error :* 

org.apache.spark.{*}sql{*}.catalyst.expressions.Literal cannot be *cast* *to* 
org.apache.spark.{*}sql{*}.catalyst.expressions.AnsiCast

The SQL will need to change to : 

{color:#4c9aff}explain{color}

{color:#4c9aff}*INSERT* OVERWRITE *TABLE* test.spark33_decimal_orc{color}

{color:#4c9aff}*select* *null* *as* amt1,{*}cast{*}('256.99' *as* 
{*}decimal{*}(20,8)) *as* amt2{color}

{color:#4c9aff}*union* *all*{color}

{color:#4c9aff}*select* {color:#de350b}{*}cast{*}({color}{*}cast{*}('200.99' 
*as* {*}decimal{*}(20,8)){*}/{*}100 *as* 
{*}decimal{*}(20,8){color:#de350b}){color} *as* amt1,{*}cast{*}('256.99' *as* 
{*}decimal{*}(20,8)) *as* amt2;{color}

 

*but this is not need in spark3.2.1 , is this a bug for spark3.3.1 ?* 

  was:
*when 'union all' and one select statement use* *Literal as column value , the 
other* *select statement  has computed expression at the same column , then the 
whole statement will compile failed. A explicit cast will be needed.*

for example:

explain

*INSERT* OVERWRITE *TABLE* test.spark33_decimal_orc

*select* *null* *as* amt1, *cast*('256.99' *as* *decimal*(20,8)) *as* amt2

*union* *all*

*select* *cast*('200.99' *as* *decimal*(20,8)){*}/{*}100 *as* 
amt1,*cast*('256.99' *as* *decimal*(20,8)) *as* amt2

*will got error :* 

org.apache.spark.*sql*.catalyst.expressions.Literal cannot be *cast* *to* 
org.apache.spark.*sql*.catalyst.expressions.AnsiCast

The SQL will need to change to : 

explain

*INSERT* OVERWRITE *TABLE* test.spark33_decimal_orc

*select* *null* *as* amt1,*cast*('256.99' *as* *decimal*(20,8)) *as* amt2

*union* *all*

*select* {*}cast{*}({*}cast{*}('200.99' *as* {*}decimal{*}(20,8)){*}/{*}100 
*as* {*}decimal{*}(20,8)) *as* amt1,*cast*('256.99' *as* *decimal*(20,8)) *as* 
amt2

 

but this is not need in spark3.2.1 , is this a bug for spark3.3.1 ? 


> An explicit cast will be needed when INSERT OVERWRITE SELECT UNION ALL
> --
>
> Key: SPARK-42473
> URL: https://issues.apache.org/jira/browse/SPARK-42473
> Project: Spark
>  Issue Type: Bug
>  Components: Optimizer
>Affects Versions: 3.3.1
> Environment: spark 3.3.1
>Reporter: kevinshin
>Priority: Major
>
> *when 'union all' and one select statement use* *Literal as column value , 
> the other* *select statement  has computed expression at the same column , 
> then the whole statement will compile failed. A explicit cast will be needed.*
> for example:
> {color:#4c9aff}explain{color}
> {color:#4c9aff}*INSERT* OVERWRITE *TABLE* test.spark33_decimal_orc{color}
> {color:#4c9aff}*select* *null* *as* amt1, {*}cast{*}('256.99' *as* 
> {*}decimal{*}(20,8)) *as* amt2{color}
> {color:#4c9aff}*union* *all*{color}
> {color:#4c9aff}*select* {*}cast{*}('200.99' *as* 
> {*}decimal{*}(20,8)){*}/{*}100 *as* amt1,{*}cast{*}('256.99' *as* 
> {*}decimal{*}(20,8)) *as* amt2;{color}
> *will got error :* 
> org.apache.spark.{*}sql{*}.catalyst.expressions.Literal cannot be *cast* *to* 
> org.apache.spark.{*}sql{*}.catalyst.expressions.AnsiCast
> The SQL will need to change to : 
> {color:#4c9aff}explain{color}
> {color:#4c9aff}*INSERT* OVERWRITE *TABLE* test.spark33_decimal_orc{color}
> {color:#4c9aff}*select* *null* *as* amt1,{*}cast{*}('256.99' *as* 
> {*}decimal{*}(20,8)) *as* amt2{color}
> {color:#4c9aff}*union* *all*{color}
> {color:#4c9aff}*select* {color:#de350b}{*}cast{*}({color}{*}cast{*}('200.99' 
> *as* {*}decimal{*}(20,8)){*}/{*}100 *as* 
> {*}decimal{*}(20,8){color:#de350b}){color} *as* amt1,{*}cast{*}('256.99' *as* 
> {*}decimal{*}(20,8)) *as* amt2;{color}
>  
> *but this is not need in spark3.2.1 , is this a bug for spark3.3.1 ?* 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-42473) An explicit cast will be needed when INSERT OVERWRITE SELECT UNION ALL

2023-02-16 Thread kevinshin (Jira)

kevinshin created SPARK-42473:
-

 Summary: An explicit cast will be needed when INSERT OVERWRITE 
SELECT UNION ALL
 Key: SPARK-42473
 URL: https://issues.apache.org/jira/browse/SPARK-42473
 Project: Spark
  Issue Type: Bug
  Components: Optimizer
Affects Versions: 3.3.1
 Environment: spark 3.3.1
Reporter: kevinshin


*when 'union all' and one select statement use* *Literal as column value , the 
other* *select statement  has computed expression at the same column , then the 
whole statement will compile failed. A explicit cast will be needed.*

for example:

explain

*INSERT* OVERWRITE *TABLE* test.spark33_decimal_orc

*select* *null* *as* amt1, *cast*('256.99' *as* *decimal*(20,8)) *as* amt2

*union* *all*

*select* *cast*('200.99' *as* *decimal*(20,8)){*}/{*}100 *as* 
amt1,*cast*('256.99' *as* *decimal*(20,8)) *as* amt2

*will got error :* 

org.apache.spark.*sql*.catalyst.expressions.Literal cannot be *cast* *to* 
org.apache.spark.*sql*.catalyst.expressions.AnsiCast

The SQL will need to change to : 

explain

*INSERT* OVERWRITE *TABLE* test.spark33_decimal_orc

*select* *null* *as* amt1,*cast*('256.99' *as* *decimal*(20,8)) *as* amt2

*union* *all*

*select* {*}cast{*}({*}cast{*}('200.99' *as* {*}decimal{*}(20,8)){*}/{*}100 
*as* {*}decimal{*}(20,8)) *as* amt1,*cast*('256.99' *as* *decimal*(20,8)) *as* 
amt2

 

but this is not need in spark3.2.1 , is this a bug for spark3.3.1 ? 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-41727) ClassCastException when config spark.sql.hive.metastore* properties under jdk17

2023-02-14 Thread kevinshin (Jira)



[ 
https://issues.apache.org/jira/browse/SPARK-41727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17688377#comment-17688377
 ] 

kevinshin commented on SPARK-41727:
---

https://github.com/apache/hive/commit/93f2274b5ddce0454f5fcaef605823618c5d9c77

> ClassCastException when config spark.sql.hive.metastore* properties under 
> jdk17
> ---
>
> Key: SPARK-41727
> URL: https://issues.apache.org/jira/browse/SPARK-41727
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.3.1
> Environment: Apache spark3.3.1 \ HDP3.1.5 with hive 3.1.0
>Reporter: kevinshin
>Priority: Major
> Attachments: hms-init-error.txt
>
>
> Apache spark3.3.1 \ HDP3.1.5 with hive 3.1.0
> when config properties about spark.sql.hive.metastore* to use 
> hive.metastore.version 3.1.2: 
> *spark.sql.hive.metastore.jars /data/soft/spark3/standalone-metastore/**
> *spark.sql.hive.metastore.version 3.1.2*
> then start spark-shell with master = local[*] under jdk17 
> try to select a hive table, will got error:
> 13:44:52.428 [main] ERROR 
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils - Got exception: 
> java.lang.ClassCastException class [Ljava.lang.Object; cannot be cast to 
> class [Ljava.net.URI; ([Ljava.lang.Object; and [Ljava.net.URI; are in module 
> java.base of loader 'bootstrap')
> java.lang.ClassCastException: class [Ljava.lang.Object; cannot be cast to 
> class [Ljava.net.URI; ([Ljava.lang.Object; and [Ljava.net.URI; are in module 
> java.base of loader 'bootstrap')
>         at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.resolveUris(HiveMetaStoreClient.java:262)
>  ~[hive-standalone-metastore-3.1.2.jar:3.1.2]
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-41727) ClassCastException when config spark.sql.hive.metastore* properties under jdk17

2023-02-14 Thread kevinshin (Jira)



 [ 
https://issues.apache.org/jira/browse/SPARK-41727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevinshin resolved SPARK-41727.
---
Resolution: Not A Bug

it is not a issue about spark, but for hive

> ClassCastException when config spark.sql.hive.metastore* properties under 
> jdk17
> ---
>
> Key: SPARK-41727
> URL: https://issues.apache.org/jira/browse/SPARK-41727
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.3.1
> Environment: Apache spark3.3.1 \ HDP3.1.5 with hive 3.1.0
>Reporter: kevinshin
>Priority: Major
> Attachments: hms-init-error.txt
>
>
> Apache spark3.3.1 \ HDP3.1.5 with hive 3.1.0
> when config properties about spark.sql.hive.metastore* to use 
> hive.metastore.version 3.1.2: 
> *spark.sql.hive.metastore.jars /data/soft/spark3/standalone-metastore/**
> *spark.sql.hive.metastore.version 3.1.2*
> then start spark-shell with master = local[*] under jdk17 
> try to select a hive table, will got error:
> 13:44:52.428 [main] ERROR 
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils - Got exception: 
> java.lang.ClassCastException class [Ljava.lang.Object; cannot be cast to 
> class [Ljava.net.URI; ([Ljava.lang.Object; and [Ljava.net.URI; are in module 
> java.base of loader 'bootstrap')
> java.lang.ClassCastException: class [Ljava.lang.Object; cannot be cast to 
> class [Ljava.net.URI; ([Ljava.lang.Object; and [Ljava.net.URI; are in module 
> java.base of loader 'bootstrap')
>         at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.resolveUris(HiveMetaStoreClient.java:262)
>  ~[hive-standalone-metastore-3.1.2.jar:3.1.2]
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-41727) ClassCastException when config spark.sql.hive.metastore* properties under jdk17

2022-12-26 Thread kevinshin (Jira)



 [ 
https://issues.apache.org/jira/browse/SPARK-41727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevinshin updated SPARK-41727:
--
Attachment: hms-init-error.txt

> ClassCastException when config spark.sql.hive.metastore* properties under 
> jdk17
> ---
>
> Key: SPARK-41727
> URL: https://issues.apache.org/jira/browse/SPARK-41727
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.3.1
> Environment: Apache spark3.3.1 \ HDP3.1.5 with hive 3.1.0
>Reporter: kevinshin
>Priority: Critical
> Attachments: hms-init-error.txt
>
>
> Apache spark3.3.1 \ HDP3.1.5 with hive 3.1.0
> when config properties about spark.sql.hive.metastore* to use 
> hive.metastore.version 3.1.2: 
> *spark.sql.hive.metastore.jars /data/soft/spark3/standalone-metastore/**
> *spark.sql.hive.metastore.version 3.1.2*
> then start spark-shell with master = local[*] under jdk17 
> try to select a hive table, will got error:
> 13:44:52.428 [main] ERROR 
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils - Got exception: 
> java.lang.ClassCastException class [Ljava.lang.Object; cannot be cast to 
> class [Ljava.net.URI; ([Ljava.lang.Object; and [Ljava.net.URI; are in module 
> java.base of loader 'bootstrap')
> java.lang.ClassCastException: class [Ljava.lang.Object; cannot be cast to 
> class [Ljava.net.URI; ([Ljava.lang.Object; and [Ljava.net.URI; are in module 
> java.base of loader 'bootstrap')
>         at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.resolveUris(HiveMetaStoreClient.java:262)
>  ~[hive-standalone-metastore-3.1.2.jar:3.1.2]
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-41727) ClassCastException when config spark.sql.hive.metastore* properties under jdk17

2022-12-26 Thread kevinshin (Jira)



 [ 
https://issues.apache.org/jira/browse/SPARK-41727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kevinshin updated SPARK-41727:
--
Description: 
Apache spark3.3.1 \ HDP3.1.5 with hive 3.1.0

when config properties about spark.sql.hive.metastore* to use 
hive.metastore.version 3.1.2: 
*spark.sql.hive.metastore.jars /data/soft/spark3/standalone-metastore/**
*spark.sql.hive.metastore.version 3.1.2*
then start spark-shell with master = local[*] under jdk17 

try to select a hive table, will got error:

13:44:52.428 [main] ERROR org.apache.hadoop.hive.metastore.utils.MetaStoreUtils 
- Got exception: java.lang.ClassCastException class [Ljava.lang.Object; cannot 
be cast to class [Ljava.net.URI; ([Ljava.lang.Object; and [Ljava.net.URI; are 
in module java.base of loader 'bootstrap')
java.lang.ClassCastException: class [Ljava.lang.Object; cannot be cast to class 
[Ljava.net.URI; ([Ljava.lang.Object; and [Ljava.net.URI; are in module 
java.base of loader 'bootstrap')
        at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.resolveUris(HiveMetaStoreClient.java:262)
 ~[hive-standalone-metastore-3.1.2.jar:3.1.2]

 

  was:
Apache spark3.3.1 \ HDP3.1.5 with hive 3.1.0

when config properties about spark.sql.hive.metastore* to use 
hive.metastore.version 3.1.2

!image-2022-12-27-13-54-18-238.png!

start spark-shell with master = local[*] under jdk17 

try to select a hive table, will got error:

13:44:52.428 [main] ERROR org.apache.hadoop.hive.metastore.utils.MetaStoreUtils 
- Got exception: java.lang.ClassCastException class [Ljava.lang.Object; cannot 
be cast to class [Ljava.net.URI; ([Ljava.lang.Object; and [Ljava.net.URI; are 
in module java.base of loader 'bootstrap')
java.lang.ClassCastException: class [Ljava.lang.Object; cannot be cast to class 
[Ljava.net.URI; ([Ljava.lang.Object; and [Ljava.net.URI; are in module 
java.base of loader 'bootstrap')
        at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.resolveUris(HiveMetaStoreClient.java:262)
 ~[hive-standalone-metastore-3.1.2.jar:3.1.2]

 


> ClassCastException when config spark.sql.hive.metastore* properties under 
> jdk17
> ---
>
> Key: SPARK-41727
> URL: https://issues.apache.org/jira/browse/SPARK-41727
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.3.1
> Environment: Apache spark3.3.1 \ HDP3.1.5 with hive 3.1.0
>Reporter: kevinshin
>Priority: Critical
>
> Apache spark3.3.1 \ HDP3.1.5 with hive 3.1.0
> when config properties about spark.sql.hive.metastore* to use 
> hive.metastore.version 3.1.2: 
> *spark.sql.hive.metastore.jars /data/soft/spark3/standalone-metastore/**
> *spark.sql.hive.metastore.version 3.1.2*
> then start spark-shell with master = local[*] under jdk17 
> try to select a hive table, will got error:
> 13:44:52.428 [main] ERROR 
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils - Got exception: 
> java.lang.ClassCastException class [Ljava.lang.Object; cannot be cast to 
> class [Ljava.net.URI; ([Ljava.lang.Object; and [Ljava.net.URI; are in module 
> java.base of loader 'bootstrap')
> java.lang.ClassCastException: class [Ljava.lang.Object; cannot be cast to 
> class [Ljava.net.URI; ([Ljava.lang.Object; and [Ljava.net.URI; are in module 
> java.base of loader 'bootstrap')
>         at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.resolveUris(HiveMetaStoreClient.java:262)
>  ~[hive-standalone-metastore-3.1.2.jar:3.1.2]
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-41727) ClassCastException when config spark.sql.hive.metastore* properties under jdk17

2022-12-26 Thread kevinshin (Jira)

kevinshin created SPARK-41727:
-

 Summary: ClassCastException when config spark.sql.hive.metastore* 
properties under jdk17
 Key: SPARK-41727
 URL: https://issues.apache.org/jira/browse/SPARK-41727
 Project: Spark
  Issue Type: Bug
  Components: SQL
Affects Versions: 3.3.1
 Environment: Apache spark3.3.1 \ HDP3.1.5 with hive 3.1.0
Reporter: kevinshin


Apache spark3.3.1 \ HDP3.1.5 with hive 3.1.0

when config properties about spark.sql.hive.metastore* to use 
hive.metastore.version 3.1.2

!image-2022-12-27-13-54-18-238.png!

start spark-shell with master = local[*] under jdk17 

try to select a hive table, will got error:

13:44:52.428 [main] ERROR org.apache.hadoop.hive.metastore.utils.MetaStoreUtils 
- Got exception: java.lang.ClassCastException class [Ljava.lang.Object; cannot 
be cast to class [Ljava.net.URI; ([Ljava.lang.Object; and [Ljava.net.URI; are 
in module java.base of loader 'bootstrap')
java.lang.ClassCastException: class [Ljava.lang.Object; cannot be cast to class 
[Ljava.net.URI; ([Ljava.lang.Object; and [Ljava.net.URI; are in module 
java.base of loader 'bootstrap')
        at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.resolveUris(HiveMetaStoreClient.java:262)
 ~[hive-standalone-metastore-3.1.2.jar:3.1.2]

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-32166) Metastore problem on Spark3.0 with Hive3.0

2022-12-26 Thread kevinshin (Jira)



[ 
https://issues.apache.org/jira/browse/SPARK-32166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17651989#comment-17651989
 ] 

kevinshin commented on SPARK-32166:
---

try set 
spark.sql.legacy.createHiveTableByDefault to false

>  Metastore problem on Spark3.0 with Hive3.0
> ---
>
> Key: SPARK-32166
> URL: https://issues.apache.org/jira/browse/SPARK-32166
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.0.0
>Reporter: hzk
>Priority: Major
>
> When i use spark-sql to create table ,the problem appear.
> {code:java}
> create table bigbig as select b.user_id , b.name , b.age , c.address , c.city 
> , a.position , a.object , a.problem , a.complaint_time from ( select user_id 
> , position , object , problem , complaint_time from 
> HIVE_COMBINE_7efde4e2dcb34c218b3fb08872e698d5 ) as a left join 
> HIVE_ODS_17_TEST_DEMO_ODS_USERS_INFO_20200608141945 as b on b.user_id = 
> a.user_id left join HIVE_ODS_17_TEST_ADDRESS_CITY_20200608141942 as c on 
> c.address_id = b.address_id;
> {code}
> It opened a connection to hive metastore.
> my hive version is 3.1.0.
> {code:java}
> org.apache.thrift.TApplicationException: Required field 'filesAdded' is 
> unset! 
> Struct:InsertEventRequestData(filesAdded:null)org.apache.thrift.TApplicationException:
>  Required field 'filesAdded' is unset! 
> Struct:InsertEventRequestData(filesAdded:null) at 
> org.apache.thrift.TApplicationException.read(TApplicationException.java:111) 
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_fire_listener_event(ThriftHiveMetastore.java:4182)
>  at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.fire_listener_event(ThriftHiveMetastore.java:4169)
>  at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.fireListenerEvent(HiveMetaStoreClient.java:1954)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:156)
>  at com.sun.proxy.$Proxy5.fireListenerEvent(Unknown Source) at 
> org.apache.hadoop.hive.ql.metadata.Hive.fireInsertEvent(Hive.java:1947) at 
> org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1673) at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> org.apache.spark.sql.hive.client.Shim_v0_14.loadTable(HiveShim.scala:847) at 
> org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply$mcV$sp(HiveClientImpl.scala:757)
>  at 
> org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply(HiveClientImpl.scala:757)
>  at 
> org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadTable$1.apply(HiveClientImpl.scala:757)
>  at 
> org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:272)
>  at 
> org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:210)
>  at 
> org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:209)
>  at 
> org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:255)
>  at 
> org.apache.spark.sql.hive.client.HiveClientImpl.loadTable(HiveClientImpl.scala:756)
>  at 
> org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadTable$1.apply$mcV$sp(HiveExternalCatalog.scala:829)
>  at 
> org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadTable$1.apply(HiveExternalCatalog.scala:827)
>  at 
> org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadTable$1.apply(HiveExternalCatalog.scala:827)
>  at 
> org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
>  at 
> org.apache.spark.sql.hive.HiveExternalCatalog.loadTable(HiveExternalCatalog.scala:827)
>  at 
> org.apache.spark.sql.catalyst.catalog.SessionCatalog.loadTable(SessionCatalog.scala:416)
>  at 
> org.apache.spark.sql.execution.command.LoadDataCommand.run(tables.scala:403) 
> at 
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
>  at 
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
>  at 
> org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
>  at

44 matches

Mail list logo