[jira] [Updated] (SPARK-39904) Rename inferDate to preferDate and fix an issue when inferring schema

2022-07-27 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-39904: - Description: Follow-up for https://issues.apache.org/jira/browse/SPARK-39469. > Rename

[jira] [Created] (SPARK-39904) Rename inferDate to preferDate and fix an issue when inferring schema

2022-07-27 Thread Ivan Sadikov (Jira)
Ivan Sadikov created SPARK-39904: Summary: Rename inferDate to preferDate and fix an issue when inferring schema Key: SPARK-39904 URL: https://issues.apache.org/jira/browse/SPARK-39904 Project: Spark

[jira] [Commented] (SPARK-39833) Filtered parquet data frame count() and show() produce inconsistent results when spark.sql.parquet.filterPushdown is true

2022-07-27 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571768#comment-17571768 ] Ivan Sadikov commented on SPARK-39833: -- Interesting, I will take a look. > Filtered parquet data

[jira] [Updated] (SPARK-39802) Support recursive references in Avro schemas in Spark

2022-07-17 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-39802: - Description: This is a follow-up for https://issues.apache.org/jira/browse/SPARK-25718.  It

[jira] [Updated] (SPARK-39802) Support recursive references in Avro schemas in Spark

2022-07-17 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-39802: - Summary: Support recursive references in Avro schemas in Spark (was: Support recursive Avro

[jira] [Updated] (SPARK-39802) Support recursive Avro schemas in Spark

2022-07-17 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-39802: - Summary: Support recursive Avro schemas in Spark (was: Support Avro recursive schemas in

[jira] [Commented] (SPARK-39802) Support Avro recursive schemas in Spark

2022-07-17 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17567759#comment-17567759 ] Ivan Sadikov commented on SPARK-39802: -- [~Gengliang.Wang] Would you be able to comment on this

[jira] [Updated] (SPARK-39802) Support Avro recursive schemas in Spark

2022-07-17 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-39802: - Description: This is a follow-up for https://issues.apache.org/jira/browse/SPARK-25718.  It

[jira] [Updated] (SPARK-39802) Support Avro recursive schemas in Spark

2022-07-17 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-39802: - Description: This is a follow-up for https://issues.apache.org/jira/browse/SPARK-25718.  It

[jira] [Updated] (SPARK-39802) Support Avro recursive schemas in Spark

2022-07-17 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-39802: - Description: This is a follow-up for https://issues.apache.org/jira/browse/SPARK-25718.  It

[jira] [Updated] (SPARK-39802) Support Avro recursive schemas in Spark

2022-07-17 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-39802: - Description: This is a follow-up for https://issues.apache.org/jira/browse/SPARK-25718.  It

[jira] [Created] (SPARK-39802) Support Avro recursive schemas in Spark

2022-07-17 Thread Ivan Sadikov (Jira)
Ivan Sadikov created SPARK-39802: Summary: Support Avro recursive schemas in Spark Key: SPARK-39802 URL: https://issues.apache.org/jira/browse/SPARK-39802 Project: Spark Issue Type:

[jira] [Updated] (SPARK-39731) Correctness issue when parsing dates with yyyyMMdd format in CSV and JSON

2022-07-14 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-39731: - Description: In Spark 3.x, when reading CSV data like this: {code:java} name,mydate 1,2020011

[jira] [Updated] (SPARK-39731) Correctness issue when parsing dates with yyyyMMdd format in CSV and JSON

2022-07-14 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-39731: - Summary: Correctness issue when parsing dates with MMdd format in CSV and JSON (was:

[jira] [Updated] (SPARK-39731) Correctness issue when parsing dates with yyyyMMdd format in CSV

2022-07-10 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-39731: - Description: In Spark 3.x, when reading CSV data like this: {code:java} name,mydate 1,2020011

[jira] [Updated] (SPARK-39731) Correctness issue when parsing dates with yyyyMMdd format in CSV

2022-07-10 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-39731: - Description: In Spark 3.x, when reading CSV data like this: {code:java} name,mydate 1,2020011

[jira] [Updated] (SPARK-39731) Correctness issue when parsing dates with yyyyMMdd format in CSV

2022-07-10 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-39731: - Description: In Spark 3.x, when reading CSV data like this: {code:java} name,mydate 1,2020011

[jira] [Created] (SPARK-39731) Correctness issue when parsing dates with yyyyMMdd format in CSV

2022-07-10 Thread Ivan Sadikov (Jira)
Ivan Sadikov created SPARK-39731: Summary: Correctness issue when parsing dates with MMdd format in CSV Key: SPARK-39731 URL: https://issues.apache.org/jira/browse/SPARK-39731 Project: Spark

[jira] [Created] (SPARK-39339) Support TimestampNTZ in JDBC data source

2022-05-30 Thread Ivan Sadikov (Jira)
Ivan Sadikov created SPARK-39339: Summary: Support TimestampNTZ in JDBC data source Key: SPARK-39339 URL: https://issues.apache.org/jira/browse/SPARK-39339 Project: Spark Issue Type:

[jira] [Updated] (SPARK-39086) Support UDT in Spark Parquet vectorized reader

2022-05-02 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-39086: - Summary: Support UDT in Spark Parquet vectorized reader (was: Support UDT in Parquet OSS

[jira] [Created] (SPARK-39086) Support UDT in Parquet OSS vectorised reader

2022-05-02 Thread Ivan Sadikov (Jira)
Ivan Sadikov created SPARK-39086: Summary: Support UDT in Parquet OSS vectorised reader Key: SPARK-39086 URL: https://issues.apache.org/jira/browse/SPARK-39086 Project: Spark Issue Type:

[jira] [Updated] (SPARK-39084) df.rdd.isEmpty() results in unexpected executor failure and JVM crash

2022-05-01 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-39084: - Description: It was discovered that a particular data distribution in a DataFrame with groupBy

[jira] [Commented] (SPARK-39084) df.rdd.isEmpty() results in unexpected executor failure and JVM crash

2022-05-01 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530600#comment-17530600 ] Ivan Sadikov commented on SPARK-39084: -- I am going to open a PR to fix this shortly. >

[jira] [Created] (SPARK-39084) df.rdd.isEmpty() results in unexpected executor failure and JVM crash

2022-05-01 Thread Ivan Sadikov (Jira)
Ivan Sadikov created SPARK-39084: Summary: df.rdd.isEmpty() results in unexpected executor failure and JVM crash Key: SPARK-39084 URL: https://issues.apache.org/jira/browse/SPARK-39084 Project: Spark

[jira] [Commented] (SPARK-38829) New configuration for controlling timestamp inference of Parquet

2022-04-11 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17520870#comment-17520870 ] Ivan Sadikov commented on SPARK-38829: -- [~Gengliang.Wang] Do you still want to merge the PR for 3.3

[jira] [Commented] (SPARK-38829) New configuration for controlling timestamp inference of Parquet

2022-04-10 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17520296#comment-17520296 ] Ivan Sadikov commented on SPARK-38829: -- I opened [https://github.com/apache/spark/pull/36137] to

[jira] [Commented] (SPARK-37771) Race condition in withHiveState and limited logic in IsolatedClientLoader result in ClassNotFoundException

2022-02-02 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17486107#comment-17486107 ] Ivan Sadikov commented on SPARK-37771: -- I could not manage to work around the issue with Hadoop

[jira] [Updated] (SPARK-37771) Race condition in withHiveState and limited logic in IsolatedClientLoader result in ClassNotFoundException

2021-12-28 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-37771: - Description: There is a race condition between creating a Hive client and loading classes that

[jira] [Updated] (SPARK-37771) Race condition in withHiveState and limited logic in IsolatedClientLoader result in ClassNotFoundException

2021-12-28 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-37771: - Description: There is a race condition between creating a Hive client and loading classes that

[jira] [Updated] (SPARK-37771) Race condition in withHiveState and limited logic in IsolatedClientLoader result in ClassNotFoundException

2021-12-28 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-37771: - Issue Type: Bug (was: Improvement) > Race condition in withHiveState and limited logic in

[jira] [Created] (SPARK-37771) Race condition in withHiveState and limited logic in IsolatedClientLoader result in ClassNotFoundException

2021-12-28 Thread Ivan Sadikov (Jira)
Ivan Sadikov created SPARK-37771: Summary: Race condition in withHiveState and limited logic in IsolatedClientLoader result in ClassNotFoundException Key: SPARK-37771 URL:

[jira] [Updated] (SPARK-37722) Escape dot character in partition names

2021-12-22 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-37722: - Summary: Escape dot character in partition names (was: Escape dots in partition names) >

[jira] [Updated] (SPARK-37722) Escape dots in partition names

2021-12-22 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-37722: - Description: Some file systems (for example, ABFS) do not support file names/paths ending with

[jira] [Updated] (SPARK-37722) Escape dots in partition names

2021-12-22 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-37722: - Description: Some file systems (for example, ABFS) do not support file names/paths ending with

[jira] [Updated] (SPARK-37722) Escape dots in partition names

2021-12-22 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-37722: - Description: Some file systems (for example, ABFS) do not support file names/paths ending with

[jira] [Created] (SPARK-37722) Escape dots in partition names

2021-12-22 Thread Ivan Sadikov (Jira)
Ivan Sadikov created SPARK-37722: Summary: Escape dots in partition names Key: SPARK-37722 URL: https://issues.apache.org/jira/browse/SPARK-37722 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-37385) Add tests for TimestampNTZ and TimestampLTZ for Parquet data source

2021-11-18 Thread Ivan Sadikov (Jira)
Ivan Sadikov created SPARK-37385: Summary: Add tests for TimestampNTZ and TimestampLTZ for Parquet data source Key: SPARK-37385 URL: https://issues.apache.org/jira/browse/SPARK-37385 Project: Spark

[jira] [Created] (SPARK-37360) Support TimestampNTZ in JSON data source

2021-11-17 Thread Ivan Sadikov (Jira)
Ivan Sadikov created SPARK-37360: Summary: Support TimestampNTZ in JSON data source Key: SPARK-37360 URL: https://issues.apache.org/jira/browse/SPARK-37360 Project: Spark Issue Type:

[jira] [Created] (SPARK-37326) Support TimestampNTZ in CSV data source

2021-11-14 Thread Ivan Sadikov (Jira)
Ivan Sadikov created SPARK-37326: Summary: Support TimestampNTZ in CSV data source Key: SPARK-37326 URL: https://issues.apache.org/jira/browse/SPARK-37326 Project: Spark Issue Type: Sub-task

<    1   2