[jira] [Resolved] (SPARK-36651) NPE in HiveShim$.toCatalystDecimal(HiveShim.scala:110)

2021-09-06 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36651. -- Resolution: Incomplete > NPE in HiveShim$.toCatalystDecimal(HiveShim.scala:110) >

[jira] [Resolved] (SPARK-36672) Using value from map in grouping sets result org.apache.spark.sql.AnalysisException

2021-09-06 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36672. -- Resolution: Duplicate > Using value from map in grouping sets result >

[jira] [Resolved] (SPARK-36668) py4j.protocol.Py4JJavaError: java.util.NoSuchElementException: None.get

2021-09-06 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36668. -- Resolution: Invalid > py4j.protocol.Py4JJavaError: java.util.NoSuchElementException: None.get

[jira] [Commented] (SPARK-36668) py4j.protocol.Py4JJavaError: java.util.NoSuchElementException: None.get

2021-09-06 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410919#comment-17410919 ] Hyukjin Kwon commented on SPARK-36668: -- Seems like the error is from accessing Py4J instances which

[jira] [Updated] (SPARK-36680) Supports Dynamic Table Options for Spark SQL

2021-09-06 Thread wangzhun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangzhun updated SPARK-36680: - Issue Type: Wish (was: New Feature) > Supports Dynamic Table Options for Spark SQL >

[jira] [Updated] (SPARK-36680) Supports Dynamic Table Options for Spark SQL

2021-09-06 Thread wangzhun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangzhun updated SPARK-36680: - Description: Now a DataFrame API user can implement dynamic options through the

[jira] [Created] (SPARK-36680) Supports Dynamic Table Options for Spark SQL

2021-09-06 Thread wangzhun (Jira)
wangzhun created SPARK-36680: Summary: Supports Dynamic Table Options for Spark SQL Key: SPARK-36680 URL: https://issues.apache.org/jira/browse/SPARK-36680 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-36153) Add SQL doc about transform for current behavior

2021-09-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410890#comment-17410890 ] Apache Spark commented on SPARK-36153: -- User 'sarutak' has created a pull request for this issue:

[jira] [Commented] (SPARK-36678) Migrate SHOW TABLES to use V2 command by default

2021-09-06 Thread Terry Kim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410872#comment-17410872 ] Terry Kim commented on SPARK-36678: --- [~cloud_fan], this command may require some catalog API changes

[jira] [Resolved] (SPARK-36667) Close resources properly in StateStoreSuite/RocksDBStateStoreSuite

2021-09-06 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-36667. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33916

[jira] [Assigned] (SPARK-36667) Close resources properly in StateStoreSuite/RocksDBStateStoreSuite

2021-09-06 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-36667: --- Assignee: Jungtaek Lim > Close resources properly in

[jira] [Assigned] (SPARK-35803) Spark SQL does not support creating views using DataSource v2 based data sources

2021-09-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35803: Assignee: (was: Apache Spark) > Spark SQL does not support creating views using

[jira] [Assigned] (SPARK-35803) Spark SQL does not support creating views using DataSource v2 based data sources

2021-09-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35803: Assignee: Apache Spark > Spark SQL does not support creating views using DataSource v2

[jira] [Commented] (SPARK-35803) Spark SQL does not support creating views using DataSource v2 based data sources

2021-09-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410844#comment-17410844 ] Apache Spark commented on SPARK-35803: -- User 'planga82' has created a pull request for this issue:

[jira] [Commented] (SPARK-35181) Use zstd for spark.io.compression.codec by default

2021-09-06 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410842#comment-17410842 ] Dongjoon Hyun commented on SPARK-35181: --- According to your log, it seems that you are hitting ZSTD

[jira] [Commented] (SPARK-35181) Use zstd for spark.io.compression.codec by default

2021-09-06 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410841#comment-17410841 ] Dongjoon Hyun commented on SPARK-35181: --- No. We are using it by default for

[jira] [Created] (SPARK-36679) Remove lz4 hadoop wrapper classes after Hadoop 3.3.2

2021-09-06 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-36679: --- Summary: Remove lz4 hadoop wrapper classes after Hadoop 3.3.2 Key: SPARK-36679 URL: https://issues.apache.org/jira/browse/SPARK-36679 Project: Spark Issue

[jira] [Created] (SPARK-36678) Migrate SHOW TABLES to use V2 command by default

2021-09-06 Thread Terry Kim (Jira)
Terry Kim created SPARK-36678: - Summary: Migrate SHOW TABLES to use V2 command by default Key: SPARK-36678 URL: https://issues.apache.org/jira/browse/SPARK-36678 Project: Spark Issue Type:

[jira] [Updated] (SPARK-36639) SQL sequence function with interval returns unexpected error in latest versions

2021-09-06 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36639: -- Fix Version/s: 3.0.4 > SQL sequence function with interval returns unexpected error in latest

[jira] [Resolved] (SPARK-36675) Support ScriptTransformation for timestamp_ntz

2021-09-06 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-36675. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 33920

[jira] [Commented] (SPARK-36677) NestedColumnAliasing pushes down aggregate functions into projections

2021-09-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410760#comment-17410760 ] Apache Spark commented on SPARK-36677: -- User 'vicennial' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36677) NestedColumnAliasing pushes down aggregate functions into projections

2021-09-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36677: Assignee: Apache Spark > NestedColumnAliasing pushes down aggregate functions into

[jira] [Assigned] (SPARK-36677) NestedColumnAliasing pushes down aggregate functions into projections

2021-09-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36677: Assignee: (was: Apache Spark) > NestedColumnAliasing pushes down aggregate functions

[jira] [Commented] (SPARK-36677) NestedColumnAliasing pushes down aggregate functions into projections

2021-09-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410759#comment-17410759 ] Apache Spark commented on SPARK-36677: -- User 'vicennial' has created a pull request for this issue:

[jira] [Commented] (SPARK-36677) NestedColumnAliasing pushes down aggregate functions into projections

2021-09-06 Thread Venkata Sai Akhil Gudesa (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410734#comment-17410734 ] Venkata Sai Akhil Gudesa commented on SPARK-36677: -- I have a fix for this and have a PR

[jira] [Created] (SPARK-36677) NestedColumnAliasing pushes down aggregate functions into projections

2021-09-06 Thread Venkata Sai Akhil Gudesa (Jira)
Venkata Sai Akhil Gudesa created SPARK-36677: Summary: NestedColumnAliasing pushes down aggregate functions into projections Key: SPARK-36677 URL: https://issues.apache.org/jira/browse/SPARK-36677

[jira] [Commented] (SPARK-36676) Create shaded Hive module and upgrade to higher version of Guava

2021-09-06 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410726#comment-17410726 ] Chao Sun commented on SPARK-36676: -- Will post a PR soon > Create shaded Hive module and upgrade to

[jira] [Created] (SPARK-36676) Create shaded Hive module and upgrade to higher version of Guava

2021-09-06 Thread Chao Sun (Jira)
Chao Sun created SPARK-36676: Summary: Create shaded Hive module and upgrade to higher version of Guava Key: SPARK-36676 URL: https://issues.apache.org/jira/browse/SPARK-36676 Project: Spark

[jira] [Assigned] (SPARK-36675) Support ScriptTransformation for timestamp_ntz

2021-09-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36675: Assignee: Kousuke Saruta (was: Apache Spark) > Support ScriptTransformation for

[jira] [Assigned] (SPARK-36675) Support ScriptTransformation for timestamp_ntz

2021-09-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36675: Assignee: Apache Spark (was: Kousuke Saruta) > Support ScriptTransformation for

[jira] [Commented] (SPARK-36675) Support ScriptTransformation for timestamp_ntz

2021-09-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410598#comment-17410598 ] Apache Spark commented on SPARK-36675: -- User 'sarutak' has created a pull request for this issue:

[jira] [Created] (SPARK-36675) Support ScriptTransformation for timestamp_ntz

2021-09-06 Thread Kousuke Saruta (Jira)
Kousuke Saruta created SPARK-36675: -- Summary: Support ScriptTransformation for timestamp_ntz Key: SPARK-36675 URL: https://issues.apache.org/jira/browse/SPARK-36675 Project: Spark Issue

[jira] [Commented] (SPARK-36674) Support ILIKE - case insensitive Like

2021-09-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410581#comment-17410581 ] Apache Spark commented on SPARK-36674: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36674) Support ILIKE - case insensitive Like

2021-09-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36674: Assignee: Max Gekk (was: Apache Spark) > Support ILIKE - case insensitive Like >

[jira] [Commented] (SPARK-36674) Support ILIKE - case insensitive Like

2021-09-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410580#comment-17410580 ] Apache Spark commented on SPARK-36674: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-36674) Support ILIKE - case insensitive Like

2021-09-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36674: Assignee: Apache Spark (was: Max Gekk) > Support ILIKE - case insensitive Like >

[jira] [Created] (SPARK-36674) Support ILIKE - case insensitive Like

2021-09-06 Thread Max Gekk (Jira)
Max Gekk created SPARK-36674: Summary: Support ILIKE - case insensitive Like Key: SPARK-36674 URL: https://issues.apache.org/jira/browse/SPARK-36674 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-36622) spark.history.kerberos.principal doesn't take value _HOST

2021-09-06 Thread pralabhkumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410578#comment-17410578 ] pralabhkumar commented on SPARK-36622: -- [~angerszhuuu] [~tgraves] [~hyukjin.kwon]   Have

[jira] [Commented] (SPARK-35181) Use zstd for spark.io.compression.codec by default

2021-09-06 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410492#comment-17410492 ] angerszhu commented on SPARK-35181: --- {code} 2021-08-28 22:38:19 INFO CodecPool:184 - Got brand-new

[jira] [Commented] (SPARK-36673) Incorrect Unions of struct with mismatched field name case

2021-09-06 Thread Marco Gaido (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410488#comment-17410488 ] Marco Gaido commented on SPARK-36673: - AFAIK, in SQL the names in the struct are case sensitive,

[jira] [Commented] (SPARK-36673) Incorrect Unions of struct with mismatched field name case

2021-09-06 Thread Shardul Mahadik (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410477#comment-17410477 ] Shardul Mahadik commented on SPARK-36673: - [~mgaido] [~cloud_fan] Since you guys were involved

[jira] [Updated] (SPARK-36673) Incorrect Unions of struct with mismatched field name case

2021-09-06 Thread Shardul Mahadik (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik updated SPARK-36673: Description: If a nested field has different casing on two sides of the union, the

[jira] [Updated] (SPARK-36673) Incorrect Unions of struct with mismatched field name case

2021-09-06 Thread Shardul Mahadik (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shardul Mahadik updated SPARK-36673: Description: If a nested field has different casing on two sides of the union, the

[jira] [Created] (SPARK-36673) Incorrect Unions of struct with mismatched field name case

2021-09-06 Thread Shardul Mahadik (Jira)
Shardul Mahadik created SPARK-36673: --- Summary: Incorrect Unions of struct with mismatched field name case Key: SPARK-36673 URL: https://issues.apache.org/jira/browse/SPARK-36673 Project: Spark

[jira] [Commented] (SPARK-36669) Fail to load Lz4 codec

2021-09-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410472#comment-17410472 ] Apache Spark commented on SPARK-36669: -- User 'viirya' has created a pull request for this issue:

[jira] [Commented] (SPARK-36626) Support TimestampNTZ in createDataFrame/toPandas and Python UDFs

2021-09-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410463#comment-17410463 ] Apache Spark commented on SPARK-36626: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-36651) NPE in HiveShim$.toCatalystDecimal(HiveShim.scala:110)

2021-09-06 Thread Serge Shikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410443#comment-17410443 ] Serge Shikov commented on SPARK-36651: -- I can't at the moment. As I can see, this issue exists

[jira] [Commented] (SPARK-36672) Using value from map in grouping sets result org.apache.spark.sql.AnalysisException

2021-09-06 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410441#comment-17410441 ] Kousuke Saruta commented on SPARK-36672: [~jhu] After SPARK-33308 was resolved, this issue seems

[jira] [Commented] (SPARK-36649) Support Trigger.AvailableNow on Kafka data source

2021-09-06 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410442#comment-17410442 ] Jungtaek Lim commented on SPARK-36649: -- [~noslowerdna] Would like to see whether you are interested

[jira] [Updated] (SPARK-36622) spark.history.kerberos.principal doesn't take value _HOST

2021-09-06 Thread pralabhkumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pralabhkumar updated SPARK-36622: - Affects Version/s: 3.2.0 > spark.history.kerberos.principal doesn't take value _HOST >

[jira] [Assigned] (SPARK-36622) spark.history.kerberos.principal doesn't take value _HOST

2021-09-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36622: Assignee: (was: Apache Spark) > spark.history.kerberos.principal doesn't take value

[jira] [Commented] (SPARK-36622) spark.history.kerberos.principal doesn't take value _HOST

2021-09-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410439#comment-17410439 ] Apache Spark commented on SPARK-36622: -- User 'pralabhkumar' has created a pull request for this

[jira] [Assigned] (SPARK-36622) spark.history.kerberos.principal doesn't take value _HOST

2021-09-06 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36622: Assignee: Apache Spark > spark.history.kerberos.principal doesn't take value _HOST >