[jira] [Commented] (SPARK-36034) Incorrect datetime filter when reading Parquet files written in legacy mode

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381856#comment-17381856 ] Apache Spark commented on SPARK-36034: -- User 'MaxGekk' has created a pull request f

[jira] [Commented] (SPARK-36034) Incorrect datetime filter when reading Parquet files written in legacy mode

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381857#comment-17381857 ] Apache Spark commented on SPARK-36034: -- User 'MaxGekk' has created a pull request f

[jira] [Assigned] (SPARK-35972) When replace ExtractValue in NestedColumnAliasing we should use semanticEquals

2021-07-16 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-35972: --- Assignee: angerszhu > When replace ExtractValue in NestedColumnAliasing we should use seman

[jira] [Updated] (SPARK-35972) When replace ExtractValue in NestedColumnAliasing we should use semanticEquals

2021-07-16 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-35972: Fix Version/s: 3.1.3 > When replace ExtractValue in NestedColumnAliasing we should use semanticEqu

[jira] [Commented] (SPARK-36099) Group exception messages in core/util

2021-07-16 Thread Shockang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381874#comment-17381874 ] Shockang commented on SPARK-36099: -- [~allisonwang-db] Could I working on this issue? >

[jira] [Resolved] (SPARK-35710) Support DPP + AQE when no reused broadcast exchange

2021-07-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-35710. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32861 [https://gith

[jira] [Assigned] (SPARK-35710) Support DPP + AQE when no reused broadcast exchange

2021-07-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-35710: --- Assignee: Ke Jia > Support DPP + AQE when no reused broadcast exchange > --

[jira] [Created] (SPARK-36176) expost tableExists in pyspark.sql.catalog

2021-07-16 Thread Dominik Gehl (Jira)
Dominik Gehl created SPARK-36176: Summary: expost tableExists in pyspark.sql.catalog Key: SPARK-36176 URL: https://issues.apache.org/jira/browse/SPARK-36176 Project: Spark Issue Type: Improve

[jira] [Updated] (SPARK-36176) expose tableExists in pyspark.sql.catalog

2021-07-16 Thread Dominik Gehl (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dominik Gehl updated SPARK-36176: - Summary: expose tableExists in pyspark.sql.catalog (was: expost tableExists in pyspark.sql.cata

[jira] [Commented] (SPARK-36176) expose tableExists in pyspark.sql.catalog

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381902#comment-17381902 ] Apache Spark commented on SPARK-36176: -- User 'dominikgehl' has created a pull reque

[jira] [Assigned] (SPARK-36176) expose tableExists in pyspark.sql.catalog

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36176: Assignee: (was: Apache Spark) > expose tableExists in pyspark.sql.catalog > -

[jira] [Assigned] (SPARK-36176) expose tableExists in pyspark.sql.catalog

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36176: Assignee: Apache Spark > expose tableExists in pyspark.sql.catalog >

[jira] [Commented] (SPARK-36176) expose tableExists in pyspark.sql.catalog

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381904#comment-17381904 ] Apache Spark commented on SPARK-36176: -- User 'dominikgehl' has created a pull reque

[jira] [Resolved] (SPARK-36048) Wrong HealthTrackerSuite.allExecutorAndHostIds

2021-07-16 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi resolved SPARK-36048. -- Fix Version/s: 3.3.0 Assignee: wuyi Resolution: Fixed Issue resolved by [https://github.com/ap

[jira] [Commented] (SPARK-32530) SPIP: Kotlin support for Apache Spark

2021-07-16 Thread canwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381919#comment-17381919 ] canwang commented on SPARK-32530: - I've been helping with the Jetbrains' [Kotlin Spark

[jira] [Comment Edited] (SPARK-32530) SPIP: Kotlin support for Apache Spark

2021-07-16 Thread canwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381919#comment-17381919 ] canwang edited comment on SPARK-32530 at 7/16/21, 9:17 AM: --- I'

[jira] [Updated] (SPARK-36176) Expose tableExists in pyspark.sql.catalog

2021-07-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36176: - Summary: Expose tableExists in pyspark.sql.catalog (was: expose tableExists in pyspark.sql.cata

[jira] [Created] (SPARK-36177) Disable CRAN in branches lower than the latest version uploaded

2021-07-16 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-36177: Summary: Disable CRAN in branches lower than the latest version uploaded Key: SPARK-36177 URL: https://issues.apache.org/jira/browse/SPARK-36177 Project: Spark

[jira] [Assigned] (SPARK-36177) Disable CRAN in branches lower than the latest version uploaded

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36177: Assignee: (was: Apache Spark) > Disable CRAN in branches lower than the latest versio

[jira] [Assigned] (SPARK-36177) Disable CRAN in branches lower than the latest version uploaded

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36177: Assignee: Apache Spark > Disable CRAN in branches lower than the latest version uploaded

[jira] [Commented] (SPARK-36177) Disable CRAN in branches lower than the latest version uploaded

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381977#comment-17381977 ] Apache Spark commented on SPARK-36177: -- User 'HyukjinKwon' has created a pull reque

[jira] [Commented] (SPARK-36177) Disable CRAN in branches lower than the latest version uploaded

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381979#comment-17381979 ] Apache Spark commented on SPARK-36177: -- User 'HyukjinKwon' has created a pull reque

[jira] [Updated] (SPARK-36177) Disable CRAN in branches lower than the latest version uploaded

2021-07-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36177: - Description: CRAN check in branch-3.0 fails as below: {code} Insufficient package version (subm

[jira] [Commented] (SPARK-36122) Spark does not passon needClientAuth to Jetty SSLContextFactory. Does not allow to configure mTLS authentication.

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381996#comment-17381996 ] Apache Spark commented on SPARK-36122: -- User 'skhandrikagmail' has created a pull r

[jira] [Assigned] (SPARK-36122) Spark does not passon needClientAuth to Jetty SSLContextFactory. Does not allow to configure mTLS authentication.

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36122: Assignee: (was: Apache Spark) > Spark does not passon needClientAuth to Jetty SSLCont

[jira] [Assigned] (SPARK-36122) Spark does not passon needClientAuth to Jetty SSLContextFactory. Does not allow to configure mTLS authentication.

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36122: Assignee: Apache Spark > Spark does not passon needClientAuth to Jetty SSLContextFactory.

[jira] [Assigned] (SPARK-36122) Spark does not passon needClientAuth to Jetty SSLContextFactory. Does not allow to configure mTLS authentication.

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36122: Assignee: Apache Spark > Spark does not passon needClientAuth to Jetty SSLContextFactory.

[jira] [Resolved] (SPARK-34893) Support native session window

2021-07-16 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-34893. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33081 [https://gi

[jira] [Assigned] (SPARK-34893) Support native session window

2021-07-16 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-34893: Assignee: Jungtaek Lim > Support native session window > - >

[jira] [Created] (SPARK-36178) document PySpark Catalog APIs in docs/source/reference/pyspark.sql.rst

2021-07-16 Thread Dominik Gehl (Jira)
Dominik Gehl created SPARK-36178: Summary: document PySpark Catalog APIs in docs/source/reference/pyspark.sql.rst Key: SPARK-36178 URL: https://issues.apache.org/jira/browse/SPARK-36178 Project: Spark

[jira] [Updated] (SPARK-36178) Document PySpark Catalog APIs in docs/source/reference/pyspark.sql.rst

2021-07-16 Thread Dominik Gehl (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dominik Gehl updated SPARK-36178: - Summary: Document PySpark Catalog APIs in docs/source/reference/pyspark.sql.rst (was: document

[jira] [Created] (SPARK-36179) Support TimestampNTZType in SparkGetColumnsOperation

2021-07-16 Thread Kent Yao (Jira)
Kent Yao created SPARK-36179: Summary: Support TimestampNTZType in SparkGetColumnsOperation Key: SPARK-36179 URL: https://issues.apache.org/jira/browse/SPARK-36179 Project: Spark Issue Type: Sub-

[jira] [Created] (SPARK-36180) HMS can not recognize timestamp_ntz

2021-07-16 Thread Kent Yao (Jira)
Kent Yao created SPARK-36180: Summary: HMS can not recognize timestamp_ntz Key: SPARK-36180 URL: https://issues.apache.org/jira/browse/SPARK-36180 Project: Spark Issue Type: Sub-task Co

[jira] [Commented] (SPARK-36178) Document PySpark Catalog APIs in docs/source/reference/pyspark.sql.rst

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382017#comment-17382017 ] Apache Spark commented on SPARK-36178: -- User 'dominikgehl' has created a pull reque

[jira] [Assigned] (SPARK-36178) Document PySpark Catalog APIs in docs/source/reference/pyspark.sql.rst

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36178: Assignee: Apache Spark > Document PySpark Catalog APIs in docs/source/reference/pyspark.s

[jira] [Assigned] (SPARK-36178) Document PySpark Catalog APIs in docs/source/reference/pyspark.sql.rst

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36178: Assignee: (was: Apache Spark) > Document PySpark Catalog APIs in docs/source/referenc

[jira] [Assigned] (SPARK-36179) Support TimestampNTZType in SparkGetColumnsOperation

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36179: Assignee: (was: Apache Spark) > Support TimestampNTZType in SparkGetColumnsOperation

[jira] [Assigned] (SPARK-36179) Support TimestampNTZType in SparkGetColumnsOperation

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36179: Assignee: Apache Spark > Support TimestampNTZType in SparkGetColumnsOperation > -

[jira] [Commented] (SPARK-36179) Support TimestampNTZType in SparkGetColumnsOperation

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382035#comment-17382035 ] Apache Spark commented on SPARK-36179: -- User 'yaooqinn' has created a pull request

[jira] [Commented] (SPARK-36179) Support TimestampNTZType in SparkGetColumnsOperation

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382036#comment-17382036 ] Apache Spark commented on SPARK-36179: -- User 'yaooqinn' has created a pull request

[jira] [Updated] (SPARK-36034) Incorrect datetime filter when reading Parquet files written in legacy mode

2021-07-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36034: - Fix Version/s: 3.0.4 > Incorrect datetime filter when reading Parquet files written in legacy mo

[jira] [Assigned] (SPARK-36160) pyspark sql/column documentation doesn't always match scala documentation

2021-07-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36160: Assignee: Dominik Gehl > pyspark sql/column documentation doesn't always match scala docu

[jira] [Resolved] (SPARK-36160) pyspark sql/column documentation doesn't always match scala documentation

2021-07-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36160. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 33369 [https://gi

[jira] [Assigned] (SPARK-36158) pyspark sql/functions documentation for months_between isn't as precise as scala version

2021-07-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36158: Assignee: Dominik Gehl > pyspark sql/functions documentation for months_between isn't as

[jira] [Assigned] (SPARK-36154) pyspark documentation doesn't mention week and quarter as valid format arguments to trunc

2021-07-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36154: Assignee: Dominik Gehl > pyspark documentation doesn't mention week and quarter as valid

[jira] [Updated] (SPARK-36158) pyspark sql/functions documentation for months_between isn't as precise as scala version

2021-07-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36158: - Fix Version/s: 3.1.3 3.2.0 > pyspark sql/functions documentation for months_b

[jira] [Assigned] (SPARK-36177) Disable CRAN in branches lower than the latest version uploaded

2021-07-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36177: Assignee: Hyukjin Kwon > Disable CRAN in branches lower than the latest version uploaded

[jira] [Updated] (SPARK-36177) Disable CRAN in branches lower than the latest version uploaded

2021-07-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36177: - Fix Version/s: 3.1.3 > Disable CRAN in branches lower than the latest version uploaded > ---

[jira] [Resolved] (SPARK-36177) Disable CRAN in branches lower than the latest version uploaded

2021-07-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36177. -- Fix Version/s: 3.0.4 Resolution: Fixed Issue resolved by pull request 33390 [https://gi

[jira] [Created] (SPARK-36181) Update pyspark sql readwriter documentation to Scala level

2021-07-16 Thread Dominik Gehl (Jira)
Dominik Gehl created SPARK-36181: Summary: Update pyspark sql readwriter documentation to Scala level Key: SPARK-36181 URL: https://issues.apache.org/jira/browse/SPARK-36181 Project: Spark Is

[jira] [Assigned] (SPARK-36181) Update pyspark sql readwriter documentation to Scala level

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36181: Assignee: Apache Spark > Update pyspark sql readwriter documentation to Scala level > ---

[jira] [Commented] (SPARK-36181) Update pyspark sql readwriter documentation to Scala level

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382121#comment-17382121 ] Apache Spark commented on SPARK-36181: -- User 'dominikgehl' has created a pull reque

[jira] [Assigned] (SPARK-36181) Update pyspark sql readwriter documentation to Scala level

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36181: Assignee: (was: Apache Spark) > Update pyspark sql readwriter documentation to Scala

[jira] [Commented] (SPARK-36181) Update pyspark sql readwriter documentation to Scala level

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382123#comment-17382123 ] Apache Spark commented on SPARK-36181: -- User 'dominikgehl' has created a pull reque

[jira] [Created] (SPARK-36182) Support TimestampNTZ type in Parquet file source

2021-07-16 Thread Gengliang Wang (Jira)
Gengliang Wang created SPARK-36182: -- Summary: Support TimestampNTZ type in Parquet file source Key: SPARK-36182 URL: https://issues.apache.org/jira/browse/SPARK-36182 Project: Spark Issue Ty

[jira] [Resolved] (SPARK-36152) Add Scala 2.13 daily build and test GitHub Action job

2021-07-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-36152. --- Fix Version/s: 3.2.0 3.3.0 Resolution: Fixed Issue resolved by pul

[jira] [Assigned] (SPARK-36182) Support TimestampNTZ type in Parquet file source

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36182: Assignee: Gengliang Wang (was: Apache Spark) > Support TimestampNTZ type in Parquet file

[jira] [Commented] (SPARK-36182) Support TimestampNTZ type in Parquet file source

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382126#comment-17382126 ] Apache Spark commented on SPARK-36182: -- User 'gengliangwang' has created a pull req

[jira] [Assigned] (SPARK-36182) Support TimestampNTZ type in Parquet file source

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36182: Assignee: Apache Spark (was: Gengliang Wang) > Support TimestampNTZ type in Parquet file

[jira] [Commented] (SPARK-36134) jackson-databind RCE vulnerability

2021-07-16 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382128#comment-17382128 ] Erik Krogen commented on SPARK-36134: - Whoops, must have missed the 3.1.2 release :)

[jira] [Updated] (SPARK-32210) Failed to serialize large MapStatuses

2021-07-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32210: -- Affects Version/s: 3.1.2 > Failed to serialize large MapStatuses > ---

[jira] [Commented] (SPARK-32210) Failed to serialize large MapStatuses

2021-07-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382131#comment-17382131 ] Dongjoon Hyun commented on SPARK-32210: --- There is a new observation of this situat

[jira] [Commented] (SPARK-32210) Failed to serialize large MapStatuses

2021-07-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382132#comment-17382132 ] Dongjoon Hyun commented on SPARK-32210: --- Please feel free to work on this, [~kazuy

[jira] [Comment Edited] (SPARK-32210) Failed to serialize large MapStatuses

2021-07-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382132#comment-17382132 ] Dongjoon Hyun edited comment on SPARK-32210 at 7/16/21, 3:13 PM: -

[jira] [Updated] (SPARK-36152) Add Scala 2.13 daily build and test GitHub Action job

2021-07-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36152: -- Description: https://github.com/apache/spark/actions/workflows/build_and_test_scala213_daily.y

[jira] [Created] (SPARK-36183) Push down limit 1 through Aggregate

2021-07-16 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-36183: --- Summary: Push down limit 1 through Aggregate Key: SPARK-36183 URL: https://issues.apache.org/jira/browse/SPARK-36183 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-36184) Use ValidateRequirements instead of EnsureRequirements to skip AQE rules that adds extra shuffles

2021-07-16 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-36184: --- Summary: Use ValidateRequirements instead of EnsureRequirements to skip AQE rules that adds extra shuffles Key: SPARK-36184 URL: https://issues.apache.org/jira/browse/SPARK-36184

[jira] [Assigned] (SPARK-36184) Use ValidateRequirements instead of EnsureRequirements to skip AQE rules that adds extra shuffles

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36184: Assignee: Apache Spark > Use ValidateRequirements instead of EnsureRequirements to skip A

[jira] [Commented] (SPARK-36184) Use ValidateRequirements instead of EnsureRequirements to skip AQE rules that adds extra shuffles

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382171#comment-17382171 ] Apache Spark commented on SPARK-36184: -- User 'cloud-fan' has created a pull request

[jira] [Assigned] (SPARK-36184) Use ValidateRequirements instead of EnsureRequirements to skip AQE rules that adds extra shuffles

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36184: Assignee: (was: Apache Spark) > Use ValidateRequirements instead of EnsureRequirement

[jira] [Assigned] (SPARK-36183) Push down limit 1 through Aggregate

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36183: Assignee: (was: Apache Spark) > Push down limit 1 through Aggregate > ---

[jira] [Commented] (SPARK-36183) Push down limit 1 through Aggregate

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382173#comment-17382173 ] Apache Spark commented on SPARK-36183: -- User 'wangyum' has created a pull request f

[jira] [Assigned] (SPARK-36183) Push down limit 1 through Aggregate

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36183: Assignee: Apache Spark > Push down limit 1 through Aggregate > --

[jira] [Commented] (SPARK-36183) Push down limit 1 through Aggregate

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382174#comment-17382174 ] Apache Spark commented on SPARK-36183: -- User 'wangyum' has created a pull request f

[jira] [Commented] (SPARK-36167) Revisit more InternalField managements.

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382241#comment-17382241 ] Apache Spark commented on SPARK-36167: -- User 'ueshin' has created a pull request fo

[jira] [Resolved] (SPARK-36128) CatalogFileIndex.filterPartitions should respect spark.sql.hive.metastorePartitionPruning

2021-07-16 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-36128. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33348 [https://gith

[jira] [Assigned] (SPARK-36128) CatalogFileIndex.filterPartitions should respect spark.sql.hive.metastorePartitionPruning

2021-07-16 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-36128: --- Assignee: Chao Sun > CatalogFileIndex.filterPartitions should respect > spark.sql.hive.met

[jira] [Commented] (SPARK-36099) Group exception messages in core/util

2021-07-16 Thread Allison Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382330#comment-17382330 ] Allison Wang commented on SPARK-36099: -- [~Shockang] Yes of course! > Group excepti

[jira] [Created] (SPARK-36185) Implement functions in CategoricalAccessor/CategoricalIndex

2021-07-16 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-36185: - Summary: Implement functions in CategoricalAccessor/CategoricalIndex Key: SPARK-36185 URL: https://issues.apache.org/jira/browse/SPARK-36185 Project: Spark

[jira] [Created] (SPARK-36186) Add as_ordered/as_unordered to CategoricalAccessor and CategoricalIndex.

2021-07-16 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-36186: - Summary: Add as_ordered/as_unordered to CategoricalAccessor and CategoricalIndex. Key: SPARK-36186 URL: https://issues.apache.org/jira/browse/SPARK-36186 Project: S

[jira] [Assigned] (SPARK-36186) Add as_ordered/as_unordered to CategoricalAccessor and CategoricalIndex.

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36186: Assignee: Apache Spark > Add as_ordered/as_unordered to CategoricalAccessor and Categoric

[jira] [Commented] (SPARK-36186) Add as_ordered/as_unordered to CategoricalAccessor and CategoricalIndex.

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382352#comment-17382352 ] Apache Spark commented on SPARK-36186: -- User 'ueshin' has created a pull request fo

[jira] [Assigned] (SPARK-36186) Add as_ordered/as_unordered to CategoricalAccessor and CategoricalIndex.

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36186: Assignee: (was: Apache Spark) > Add as_ordered/as_unordered to CategoricalAccessor an

[jira] [Commented] (SPARK-35785) Cleanup support for RocksDB instance

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382361#comment-17382361 ] Apache Spark commented on SPARK-35785: -- User 'viirya' has created a pull request fo

[jira] [Created] (SPARK-36187) Commit collision avoidance in dynamicPartitionOverwrite for non-Parquet formats

2021-07-16 Thread Tony Zhang (Jira)
Tony Zhang created SPARK-36187: -- Summary: Commit collision avoidance in dynamicPartitionOverwrite for non-Parquet formats Key: SPARK-36187 URL: https://issues.apache.org/jira/browse/SPARK-36187 Project:

[jira] [Updated] (SPARK-36187) Commit collision avoidance in dynamicPartitionOverwrite for non-Parquet formats

2021-07-16 Thread Tony Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tony Zhang updated SPARK-36187: --- Shepherd: Wenchen Fan Description: Hi, my question here is specifically about [PR #29000|htt

[jira] [Created] (SPARK-36188) Add categories setter to CategoricalAccessor and CategoricalIndex.

2021-07-16 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-36188: - Summary: Add categories setter to CategoricalAccessor and CategoricalIndex. Key: SPARK-36188 URL: https://issues.apache.org/jira/browse/SPARK-36188 Project: Spark

[jira] [Created] (SPARK-36189) Improve bool, string, numeric DataTypeOps tests by avoiding joins

2021-07-16 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36189: Summary: Improve bool, string, numeric DataTypeOps tests by avoiding joins Key: SPARK-36189 URL: https://issues.apache.org/jira/browse/SPARK-36189 Project: Spark

[jira] [Created] (SPARK-36190) Improve the rest of DataTypeOps tests by avoiding joins

2021-07-16 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36190: Summary: Improve the rest of DataTypeOps tests by avoiding joins Key: SPARK-36190 URL: https://issues.apache.org/jira/browse/SPARK-36190 Project: Spark Issue

[jira] [Commented] (SPARK-36188) Add categories setter to CategoricalAccessor and CategoricalIndex.

2021-07-16 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382378#comment-17382378 ] Takuya Ueshin commented on SPARK-36188: --- I'm working on this. > Add categories se

[jira] [Commented] (SPARK-36189) Improve bool, string, numeric DataTypeOps tests by avoiding joins

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17382379#comment-17382379 ] Apache Spark commented on SPARK-36189: -- User 'xinrong-databricks' has created a pul

[jira] [Assigned] (SPARK-36189) Improve bool, string, numeric DataTypeOps tests by avoiding joins

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36189: Assignee: Apache Spark > Improve bool, string, numeric DataTypeOps tests by avoiding join

[jira] [Assigned] (SPARK-36189) Improve bool, string, numeric DataTypeOps tests by avoiding joins

2021-07-16 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36189: Assignee: (was: Apache Spark) > Improve bool, string, numeric DataTypeOps tests by av

[jira] [Updated] (SPARK-36187) Commit collision avoidance in dynamicPartitionOverwrite for non-Parquet formats

2021-07-16 Thread Tony Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tony Zhang updated SPARK-36187: --- Description: Hi, my question here is specifically about [PR #29000|https://github.com/apache/spark/

[jira] [Created] (SPARK-36191) Support ORDER BY and LIMIT to be on the correlation path

2021-07-16 Thread Allison Wang (Jira)
Allison Wang created SPARK-36191: Summary: Support ORDER BY and LIMIT to be on the correlation path Key: SPARK-36191 URL: https://issues.apache.org/jira/browse/SPARK-36191 Project: Spark Issu

[jira] [Created] (SPARK-36192) Better error messages when comparing against list

2021-07-16 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-36192: Summary: Better error messages when comparing against list Key: SPARK-36192 URL: https://issues.apache.org/jira/browse/SPARK-36192 Project: Spark Issue Type

[jira] [Resolved] (SPARK-35276) Write checksum files for shuffle

2021-07-16 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-35276. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 324

[jira] [Assigned] (SPARK-35276) Write checksum files for shuffle

2021-07-16 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35276: --- Assignee: wuyi > Write checksum files for shuffle > --