[jira] [Updated] (SPARK-37082) Implement histogram_numeric aggregate function in spark

2021-10-20 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-37082: -- Summary: Implement histogram_numeric aggregate function in spark (was: Implement histogram_numeric fu

[jira] [Commented] (SPARK-33564) Prometheus metrics for Master and Worker isn't working

2021-10-20 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432225#comment-17432225 ] Dongjoon Hyun commented on SPARK-33564: --- AFAIK, no plan for other resource manager

[jira] [Created] (SPARK-37082) Implement histogram_numeric function in spark

2021-10-20 Thread angerszhu (Jira)
angerszhu created SPARK-37082: - Summary: Implement histogram_numeric function in spark Key: SPARK-37082 URL: https://issues.apache.org/jira/browse/SPARK-37082 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-37069) HiveClientImpl throws NoSuchMethodError: org.apache.hadoop.hive.ql.metadata.Hive.getWithoutRegisterFns

2021-10-20 Thread Zhou Yifan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432206#comment-17432206 ] Zhou Yifan commented on SPARK-37069: Gently ping [~csun] > HiveClientImpl throws No

[jira] [Resolved] (SPARK-37075) move UDAF expression building from sql/catalyst to sql/core

2021-10-20 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37075. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34340 [https://gith

[jira] [Assigned] (SPARK-37075) move UDAF expression building from sql/catalyst to sql/core

2021-10-20 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37075: --- Assignee: Wenchen Fan > move UDAF expression building from sql/catalyst to sql/core > -

[jira] [Resolved] (SPARK-37076) Implement StructType.toString explicitly for Scala 2.13

2021-10-20 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37076. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34341 [https://gi

[jira] [Assigned] (SPARK-37081) Upgrade the version of RDBMS and corresponding JDBC drivers used by docker-integration-tests

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37081: Assignee: Kousuke Saruta (was: Apache Spark) > Upgrade the version of RDBMS and correspo

[jira] [Assigned] (SPARK-37081) Upgrade the version of RDBMS and corresponding JDBC drivers used by docker-integration-tests

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37081: Assignee: Apache Spark (was: Kousuke Saruta) > Upgrade the version of RDBMS and correspo

[jira] [Commented] (SPARK-37081) Upgrade the version of RDBMS and corresponding JDBC drivers used by docker-integration-tests

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432197#comment-17432197 ] Apache Spark commented on SPARK-37081: -- User 'sarutak' has created a pull request f

[jira] [Created] (SPARK-37081) Upgrade the version of RDBMS and corresponding JDBC drivers used by docker-integration-tests

2021-10-20 Thread Kousuke Saruta (Jira)
Kousuke Saruta created SPARK-37081: -- Summary: Upgrade the version of RDBMS and corresponding JDBC drivers used by docker-integration-tests Key: SPARK-37081 URL: https://issues.apache.org/jira/browse/SPARK-37081

[jira] [Updated] (SPARK-36394) Increase pandas API coverage in PySpark

2021-10-20 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36394: - Shepherd: Takuya Ueshin > Increase pandas API coverage in PySpark >

[jira] [Assigned] (SPARK-36394) Increase pandas API coverage in PySpark

2021-10-20 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36394: Assignee: Xinrong Meng > Increase pandas API coverage in PySpark > --

[jira] [Updated] (SPARK-36845) Inline type hint files

2021-10-20 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36845: - Shepherd: Takuya Ueshin > Inline type hint files > -- > > Ke

[jira] [Assigned] (SPARK-36845) Inline type hint files

2021-10-20 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36845: Assignee: Xinrong Meng > Inline type hint files > -- > >

[jira] [Assigned] (SPARK-37080) Add benchmark tool guide in pull request template

2021-10-20 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-37080: Assignee: XiDuo You > Add benchmark tool guide in pull request template > ---

[jira] [Resolved] (SPARK-37080) Add benchmark tool guide in pull request template

2021-10-20 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37080. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34349 [https://gi

[jira] [Commented] (SPARK-37080) Add benchmark tool guide in pull request template

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432173#comment-17432173 ] Apache Spark commented on SPARK-37080: -- User 'ulysses-you' has created a pull reque

[jira] [Assigned] (SPARK-37080) Add benchmark tool guide in pull request template

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37080: Assignee: (was: Apache Spark) > Add benchmark tool guide in pull request template > -

[jira] [Assigned] (SPARK-37080) Add benchmark tool guide in pull request template

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37080: Assignee: Apache Spark > Add benchmark tool guide in pull request template >

[jira] [Updated] (SPARK-37080) Add benchmark tool guide in pull request template

2021-10-20 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-37080: -- Summary: Add benchmark tool guide in pull request template (was: Add benchmark guide in pull request

[jira] [Created] (SPARK-37080) Add benchmark guide in pull request template

2021-10-20 Thread XiDuo You (Jira)
XiDuo You created SPARK-37080: - Summary: Add benchmark guide in pull request template Key: SPARK-37080 URL: https://issues.apache.org/jira/browse/SPARK-37080 Project: Spark Issue Type: Improvemen

[jira] [Issue Comment Deleted] (SPARK-37016) Publicise UpperCaseCharStream

2021-10-20 Thread dohongdayi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dohongdayi updated SPARK-37016: --- Comment: was deleted (was: I have submitted a PR [https://github.com/apache/spark/pull/34290|https:

[jira] [Updated] (SPARK-37016) Publicise UpperCaseCharStream

2021-10-20 Thread dohongdayi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dohongdayi updated SPARK-37016: --- Fix Version/s: (was: 3.2.1) (was: 3.3.0) (was: 3.0.

[jira] [Assigned] (SPARK-37078) Support old 3-parameter Sink constructors

2021-10-20 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37078: - Assignee: Dongjoon Hyun > Support old 3-parameter Sink constructors > -

[jira] [Resolved] (SPARK-37078) Support old 3-parameter Sink constructors

2021-10-20 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37078. --- Fix Version/s: 3.2.1 3.3.0 Resolution: Fixed Issue resolved by pul

[jira] [Comment Edited] (SPARK-37055) Apply 'compute.eager_check' across all the codebase

2021-10-20 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432159#comment-17432159 ] Hyukjin Kwon edited comment on SPARK-37055 at 10/21/21, 1:39 AM: -

[jira] [Commented] (SPARK-37055) Apply 'compute.eager_check' across all the codebase

2021-10-20 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432159#comment-17432159 ] Hyukjin Kwon commented on SPARK-37055: -- Actually, it's more to prevent running Spar

[jira] [Assigned] (SPARK-37079) Fix DataFrameWriterV2.partitionedBy to send the arguments to JVM properly

2021-10-20 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-37079: Assignee: Takuya Ueshin > Fix DataFrameWriterV2.partitionedBy to send the arguments to JV

[jira] [Resolved] (SPARK-37079) Fix DataFrameWriterV2.partitionedBy to send the arguments to JVM properly

2021-10-20 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37079. -- Fix Version/s: 3.1.3 3.2.1 3.3.0 Resolution: Fixed

[jira] [Resolved] (SPARK-36952) Inline type hints for python/pyspark/resource/information.py and python/pyspark/resource/profile.py

2021-10-20 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36952. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34222 [https://gi

[jira] [Resolved] (SPARK-37033) Inline type hints for python/pyspark/resource/requests.py

2021-10-20 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37033. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34321 [https://gi

[jira] [Assigned] (SPARK-37033) Inline type hints for python/pyspark/resource/requests.py

2021-10-20 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-37033: Assignee: dch nguyen > Inline type hints for python/pyspark/resource/requests.py > --

[jira] [Assigned] (SPARK-37061) Custom V2 Metrics uses wrong classname for lookup

2021-10-20 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37061: - Assignee: Russell Spitzer > Custom V2 Metrics uses wrong classname for lookup > ---

[jira] [Resolved] (SPARK-37061) Custom V2 Metrics uses wrong classname for lookup

2021-10-20 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37061. --- Fix Version/s: 3.2.1 3.3.0 Resolution: Fixed Issue resolved by pul

[jira] [Updated] (SPARK-37078) Support old 3-parameter Sink constructors

2021-10-20 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37078: -- Priority: Critical (was: Major) > Support old 3-parameter Sink constructors > ---

[jira] [Commented] (SPARK-37078) Support old 3-parameter Sink constructors

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432134#comment-17432134 ] Apache Spark commented on SPARK-37078: -- User 'dongjoon-hyun' has created a pull req

[jira] [Assigned] (SPARK-37078) Support old 3-parameter Sink constructors

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37078: Assignee: Apache Spark > Support old 3-parameter Sink constructors >

[jira] [Assigned] (SPARK-37078) Support old 3-parameter Sink constructors

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37078: Assignee: (was: Apache Spark) > Support old 3-parameter Sink constructors > -

[jira] [Assigned] (SPARK-37079) Fix DataFrameWriterV2.partitionedBy to send the arguments to JVM properly

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37079: Assignee: Apache Spark > Fix DataFrameWriterV2.partitionedBy to send the arguments to JVM

[jira] [Assigned] (SPARK-37079) Fix DataFrameWriterV2.partitionedBy to send the arguments to JVM properly

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37079: Assignee: (was: Apache Spark) > Fix DataFrameWriterV2.partitionedBy to send the argum

[jira] [Commented] (SPARK-37079) Fix DataFrameWriterV2.partitionedBy to send the arguments to JVM properly

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432128#comment-17432128 ] Apache Spark commented on SPARK-37079: -- User 'ueshin' has created a pull request fo

[jira] [Commented] (SPARK-36645) Aggregate (Min/Max/Count) push down for Parquet

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432127#comment-17432127 ] Apache Spark commented on SPARK-36645: -- User 'huaxingao' has created a pull request

[jira] [Created] (SPARK-37079) Fix DataFrameWriterV2.partitionedBy to send the arguments to JVM properly

2021-10-20 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-37079: - Summary: Fix DataFrameWriterV2.partitionedBy to send the arguments to JVM properly Key: SPARK-37079 URL: https://issues.apache.org/jira/browse/SPARK-37079 Project:

[jira] [Assigned] (SPARK-37025) Upgrade RoaringBitmap to 0.9.22

2021-10-20 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-37025: Assignee: Igor Dvorzhak (was: Lantao Jin) > Upgrade RoaringBitmap to 0.9.22 > --

[jira] [Updated] (SPARK-37025) Upgrade RoaringBitmap to 0.9.22

2021-10-20 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-37025: - Issue Type: Improvement (was: Bug) Priority: Minor (was: Major) > Upgrade RoaringBitmap

[jira] [Resolved] (SPARK-37025) Upgrade RoaringBitmap to 0.9.22

2021-10-20 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-37025. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34300 [https://gi

[jira] [Resolved] (SPARK-37048) Clean up inlining type hints under SQL module

2021-10-20 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-37048. --- Fix Version/s: 3.3.0 Assignee: Takuya Ueshin Resolution: Fixed Issue resolve

[jira] [Updated] (SPARK-37078) Support old 3-parameter Sink constructors

2021-10-20 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37078: -- Summary: Support old 3-parameter Sink constructors (was: Support old 3-parameter Sink constru

[jira] [Created] (SPARK-37078) Support old 3-parameter Sink constructor

2021-10-20 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-37078: - Summary: Support old 3-parameter Sink constructor Key: SPARK-37078 URL: https://issues.apache.org/jira/browse/SPARK-37078 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-37061) Custom V2 Metrics uses wrong classname for lookup

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432119#comment-17432119 ] Apache Spark commented on SPARK-37061: -- User 'RussellSpitzer' has created a pull re

[jira] [Assigned] (SPARK-37061) Custom V2 Metrics uses wrong classname for lookup

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37061: Assignee: (was: Apache Spark) > Custom V2 Metrics uses wrong classname for lookup > -

[jira] [Assigned] (SPARK-37061) Custom V2 Metrics uses wrong classname for lookup

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37061: Assignee: Apache Spark > Custom V2 Metrics uses wrong classname for lookup >

[jira] [Commented] (SPARK-37061) Custom V2 Metrics uses wrong classname for lookup

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17432117#comment-17432117 ] Apache Spark commented on SPARK-37061: -- User 'RussellSpitzer' has created a pull re

[jira] [Assigned] (SPARK-37077) Annotations for pyspark.sql.context.SQLContext.createDataFrame are broken

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37077: Assignee: Apache Spark > Annotations for pyspark.sql.context.SQLContext.createDataFrame a

[jira] [Commented] (SPARK-37077) Annotations for pyspark.sql.context.SQLContext.createDataFrame are broken

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17431880#comment-17431880 ] Apache Spark commented on SPARK-37077: -- User 'zero323' has created a pull request f

[jira] [Assigned] (SPARK-37077) Annotations for pyspark.sql.context.SQLContext.createDataFrame are broken

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37077: Assignee: (was: Apache Spark) > Annotations for pyspark.sql.context.SQLContext.create

[jira] [Commented] (SPARK-37077) Annotations for pyspark.sql.context.SQLContext.createDataFrame are broken

2021-10-20 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17431446#comment-17431446 ] Maciej Szymkiewicz commented on SPARK-37077: I'll fix this in SPARK-36894, s

[jira] [Created] (SPARK-37077) Annotations for pyspark.sql.context.SQLContext.createDataFrame are broken

2021-10-20 Thread Maciej Szymkiewicz (Jira)
Maciej Szymkiewicz created SPARK-37077: -- Summary: Annotations for pyspark.sql.context.SQLContext.createDataFrame are broken Key: SPARK-37077 URL: https://issues.apache.org/jira/browse/SPARK-37077

[jira] [Commented] (SPARK-37076) Implement StructType.toString explicitly for Scala 2.13

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17431421#comment-17431421 ] Apache Spark commented on SPARK-37076: -- User 'sarutak' has created a pull request f

[jira] [Assigned] (SPARK-37076) Implement StructType.toString explicitly for Scala 2.13

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37076: Assignee: Apache Spark (was: Kousuke Saruta) > Implement StructType.toString explicitly

[jira] [Assigned] (SPARK-37076) Implement StructType.toString explicitly for Scala 2.13

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37076: Assignee: Kousuke Saruta (was: Apache Spark) > Implement StructType.toString explicitly

[jira] [Updated] (SPARK-37076) Implement StructType.toString explicitly for Scala 2.13

2021-10-20 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-37076: --- Summary: Implement StructType.toString explicitly for Scala 2.13 (was: Implements StructTyp

[jira] [Created] (SPARK-37076) Implements StructType.toString explicitly for Scala 2.13

2021-10-20 Thread Kousuke Saruta (Jira)
Kousuke Saruta created SPARK-37076: -- Summary: Implements StructType.toString explicitly for Scala 2.13 Key: SPARK-37076 URL: https://issues.apache.org/jira/browse/SPARK-37076 Project: Spark

[jira] [Assigned] (SPARK-37075) move UDAF expression building from sql/catalyst to sql/core

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37075: Assignee: (was: Apache Spark) > move UDAF expression building from sql/catalyst to sq

[jira] [Commented] (SPARK-37075) move UDAF expression building from sql/catalyst to sql/core

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17431360#comment-17431360 ] Apache Spark commented on SPARK-37075: -- User 'cloud-fan' has created a pull request

[jira] [Assigned] (SPARK-37075) move UDAF expression building from sql/catalyst to sql/core

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37075: Assignee: Apache Spark > move UDAF expression building from sql/catalyst to sql/core > --

[jira] [Created] (SPARK-37075) move UDAF expression building from sql/catalyst to sql/core

2021-10-20 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-37075: --- Summary: move UDAF expression building from sql/catalyst to sql/core Key: SPARK-37075 URL: https://issues.apache.org/jira/browse/SPARK-37075 Project: Spark Is

[jira] [Assigned] (SPARK-37074) Push extra predicates through non-join

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37074: Assignee: (was: Apache Spark) > Push extra predicates through non-join >

[jira] [Commented] (SPARK-37074) Push extra predicates through non-join

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17431355#comment-17431355 ] Apache Spark commented on SPARK-37074: -- User 'tanelk' has created a pull request fo

[jira] [Assigned] (SPARK-37074) Push extra predicates through non-join

2021-10-20 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37074: Assignee: Apache Spark > Push extra predicates through non-join > ---

[jira] [Created] (SPARK-37074) Push extra predicates through non-join

2021-10-20 Thread Tanel Kiis (Jira)
Tanel Kiis created SPARK-37074: -- Summary: Push extra predicates through non-join Key: SPARK-37074 URL: https://issues.apache.org/jira/browse/SPARK-37074 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-37073) Pass all UTs in `external/avro` with Java 17

2021-10-20 Thread Yang Jie (Jira)
Yang Jie created SPARK-37073: Summary: Pass all UTs in `external/avro` with Java 17 Key: SPARK-37073 URL: https://issues.apache.org/jira/browse/SPARK-37073 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-37072) Pass all UTs in `repl` with Java 17

2021-10-20 Thread Yang Jie (Jira)
Yang Jie created SPARK-37072: Summary: Pass all UTs in `repl` with Java 17 Key: SPARK-37072 URL: https://issues.apache.org/jira/browse/SPARK-37072 Project: Spark Issue Type: Sub-task Co

[jira] [Assigned] (SPARK-37058) Add spark-shell command line unit test

2021-10-20 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37058: --- Assignee: angerszhu > Add spark-shell command line unit test >

[jira] [Resolved] (SPARK-37058) Add spark-shell command line unit test

2021-10-20 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37058. - Fix Version/s: 3.2.1 3.3.0 Resolution: Fixed Issue resolved by pull re

[jira] [Updated] (SPARK-37070) Pass all UTs in `mllib-local` and `mllib` with Java 17

2021-10-20 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-37070: - Description: execute "mvn clean install -pl mllib-local"  with Java 17 {code:java} Run completed in 714

[jira] [Updated] (SPARK-37070) Pass all UTs in `mllib-local` and `mllib` with Java 17

2021-10-20 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-37070: - Summary: Pass all UTs in `mllib-local` and `mllib` with Java 17 (was: Pass all UTs in `mllib-local` wit

[jira] [Commented] (SPARK-32285) Add PySpark support for nested timestamps with arrow

2021-10-20 Thread pralabhkumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17431140#comment-17431140 ] pralabhkumar commented on SPARK-32285: -- [~bryanc] . Please review the PR .  > Add

[jira] [Created] (SPARK-37071) OpenHashMap should be serializable without reference tracking

2021-10-20 Thread Emil Ejbyfeldt (Jira)
Emil Ejbyfeldt created SPARK-37071: -- Summary: OpenHashMap should be serializable without reference tracking Key: SPARK-37071 URL: https://issues.apache.org/jira/browse/SPARK-37071 Project: Spark

[jira] [Created] (SPARK-37070) Pass all UTs in `mllib-local` with Java 17

2021-10-20 Thread Yang Jie (Jira)
Yang Jie created SPARK-37070: Summary: Pass all UTs in `mllib-local` with Java 17 Key: SPARK-37070 URL: https://issues.apache.org/jira/browse/SPARK-37070 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-36796) Make sql/core and dependent modules all UTs pass on Java 17

2021-10-20 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-36796: - Summary: Make sql/core and dependent modules all UTs pass on Java 17 (was: Make all unit tests pass on

[jira] [Resolved] (SPARK-36898) Make the shuffle hash join factor configurable

2021-10-20 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36898. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34150 [https://gith

[jira] [Updated] (SPARK-37035) Improve error message when use parquet vectorize reader

2021-10-20 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-37035: -- Summary: Improve error message when use parquet vectorize reader (was: Improve error message when use

[jira] [Assigned] (SPARK-36898) Make the shuffle hash join factor configurable

2021-10-20 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-36898: --- Assignee: Ke Jia > Make the shuffle hash join factor configurable > ---

[jira] [Commented] (SPARK-37055) Apply 'compute.eager_check' across all the codebase

2021-10-20 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17431077#comment-17431077 ] Yikun Jiang commented on SPARK-37055: - Looks like it's a method level results runtim

[jira] [Commented] (SPARK-36877) Calling ds.rdd with AQE enabled leads to jobs being run, eventually causing reruns

2021-10-20 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17431069#comment-17431069 ] Wenchen Fan commented on SPARK-36877: - You were calling `df3.repartition(5).write`,

[jira] [Commented] (SPARK-37069) HiveClientImpl throws NoSuchMethodError: org.apache.hadoop.hive.ql.metadata.Hive.getWithoutRegisterFns

2021-10-20 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-37069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17431063#comment-17431063 ] 周一帆 commented on SPARK-37069: - IsolatedClientLoader#hiveVersion resolves version string '2.3

[jira] [Assigned] (SPARK-37046) Alter view does not preserve column case

2021-10-20 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37046: --- Assignee: Abhishek Somani > Alter view does not preserve column case >

[jira] [Resolved] (SPARK-37046) Alter view does not preserve column case

2021-10-20 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37046. - Fix Version/s: 3.2.1 3.3.0 Resolution: Fixed Issue resolved by pull re

[jira] [Updated] (SPARK-37069) HiveClientImpl throws NoSuchMethodError: org.apache.hadoop.hive.ql.metadata.Hive.getWithoutRegisterFns

2021-10-20 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-37069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 周一帆 updated SPARK-37069: Description: If we run Spark SQL with external Hive 2.3.x (before 2.3.9) jars, following error will be thrown: {c

[jira] [Created] (SPARK-37069) HiveClientImpl throws NoSuchMethodError: org.apache.hadoop.hive.ql.metadata.Hive.getWithoutRegisterFns

2021-10-20 Thread Jira
周一帆 created SPARK-37069: --- Summary: HiveClientImpl throws NoSuchMethodError: org.apache.hadoop.hive.ql.metadata.Hive.getWithoutRegisterFns Key: SPARK-37069 URL: https://issues.apache.org/jira/browse/SPARK-37069

[jira] [Resolved] (SPARK-37064) Fix outer join return the wrong max rows if other side is empty

2021-10-20 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37064. - Fix Version/s: 3.2.1 3.3.0 Resolution: Fixed Issue resolved by pull re