[jira] [Updated] (SPARK-37802) composite field name like `field name` doesn't work with Aggregate push down

[ https://issues.apache.org/jira/browse/SPARK-37802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxin Gao updated SPARK-37802: --- Fix Version/s: 3.2.0 > composite field name like `field name` doesn't work with Aggregate push down

[jira] [Updated] (SPARK-37640) rolled event log still need be clean after compact

[ https://issues.apache.org/jira/browse/SPARK-37640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] muhong updated SPARK-37640: --- Description: when we set "{{{}spark.eventLog.rolling.enabled{}}} =true", the eventlog will be roll and

[jira] [Updated] (SPARK-37640) rolled event log still need be clean after compact

[ https://issues.apache.org/jira/browse/SPARK-37640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] muhong updated SPARK-37640: --- Description: when we set "{{{}spark.eventLog.rolling.enabled{}}} =true", the eventlog will be roll and

[jira] [Updated] (SPARK-37640) rolled event log still need be clean after compact

[ https://issues.apache.org/jira/browse/SPARK-37640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] muhong updated SPARK-37640: --- Description: when we set "{{{}spark.eventLog.rolling.enabled{}}} =true", the eventlog will be roll and

[jira] [Updated] (SPARK-37640) rolled event log still need be clean after compact

[ https://issues.apache.org/jira/browse/SPARK-37640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] muhong updated SPARK-37640: --- Description: when > rolled event log still need be clean after compact >

[jira] [Commented] (SPARK-37640) rolled event log still need be clean after compact

[ https://issues.apache.org/jira/browse/SPARK-37640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471684#comment-17471684 ] muhong commented on SPARK-37640: ok，i will add a more detail description > rolled event log still need

[jira] [Commented] (SPARK-37821) spark thrift server RDD ID overflow lead sql execute failed

[ https://issues.apache.org/jira/browse/SPARK-37821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471681#comment-17471681 ] muhong commented on SPARK-37821: yes，just the modification is much complicated； > spark thrift server

[jira] [Commented] (SPARK-37852) Enable flake's E741 rule in PySpark

[ https://issues.apache.org/jira/browse/SPARK-37852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471656#comment-17471656 ] Apache Spark commented on SPARK-37852: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-37852) Enable flake's E741 rule in PySpark

[ https://issues.apache.org/jira/browse/SPARK-37852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37852: Assignee: Apache Spark > Enable flake's E741 rule in PySpark >

[jira] [Assigned] (SPARK-37852) Enable flake's E741 rule in PySpark

[ https://issues.apache.org/jira/browse/SPARK-37852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37852: Assignee: (was: Apache Spark) > Enable flake's E741 rule in PySpark >

[jira] [Commented] (SPARK-37852) Enable flake's E741 rule in PySpark

[ https://issues.apache.org/jira/browse/SPARK-37852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471655#comment-17471655 ] Apache Spark commented on SPARK-37852: -- User 'HyukjinKwon' has created a pull request for this

[jira] (SPARK-37850) Enable flake's E731 rule in PySpark

[ https://issues.apache.org/jira/browse/SPARK-37850 ] Hyukjin Kwon deleted comment on SPARK-37850: -- was (Author: apachespark): User 'HyukjinKwon' has created a pull request for this issue: https://github.com/apache/spark/pull/35152 > Enable

[jira] [Created] (SPARK-37852) Enable flake's E741 rule in PySpark

Hyukjin Kwon created SPARK-37852: Summary: Enable flake's E741 rule in PySpark Key: SPARK-37852 URL: https://issues.apache.org/jira/browse/SPARK-37852 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-37850) Enable flake's E731 rule in PySpark

[ https://issues.apache.org/jira/browse/SPARK-37850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471653#comment-17471653 ] Apache Spark commented on SPARK-37850: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Updated] (SPARK-37851) Mark TPCDS*Suite, SQLQuerySuite and org.apache.spark.sql.hive.execution as slow tests

[ https://issues.apache.org/jira/browse/SPARK-37851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37851: - Summary: Mark TPCDS*Suite, SQLQuerySuite and org.apache.spark.sql.hive.execution as slow tests

[jira] [Updated] (SPARK-37851) Mark TPCDS*Suite and SQLQuerySuite as ExtendedSQLTest

[ https://issues.apache.org/jira/browse/SPARK-37851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37851: - Description: Related to SPARK-33171 and SPARK-32884. We should rebalance both: "sql - slow

[jira] [Updated] (SPARK-37851) Mark TPCDS*Suite and SQLQuerySuite as ExtendedSQLTest

[ https://issues.apache.org/jira/browse/SPARK-37851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37851: - Description: Related to SPARK-33171 and SPARK-32884. We should rebalance both: "sql - slow

[jira] [Assigned] (SPARK-37851) Mark TPCDS*Suite and SQLQuerySuite as ExtendedSQLTest

[ https://issues.apache.org/jira/browse/SPARK-37851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37851: Assignee: Apache Spark > Mark TPCDS*Suite and SQLQuerySuite as ExtendedSQLTest >

[jira] [Assigned] (SPARK-37851) Mark TPCDS*Suite and SQLQuerySuite as ExtendedSQLTest

[ https://issues.apache.org/jira/browse/SPARK-37851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37851: Assignee: (was: Apache Spark) > Mark TPCDS*Suite and SQLQuerySuite as

[jira] [Commented] (SPARK-37851) Mark TPCDS*Suite and SQLQuerySuite as ExtendedSQLTest

[ https://issues.apache.org/jira/browse/SPARK-37851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471651#comment-17471651 ] Apache Spark commented on SPARK-37851: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Created] (SPARK-37851) Mark TPCDS*Suite and SQLQuerySuite as ExtendedSQLTest

Hyukjin Kwon created SPARK-37851: Summary: Mark TPCDS*Suite and SQLQuerySuite as ExtendedSQLTest Key: SPARK-37851 URL: https://issues.apache.org/jira/browse/SPARK-37851 Project: Spark Issue

[jira] [Assigned] (SPARK-37850) Enable flake's E731 rule in PySpark

[ https://issues.apache.org/jira/browse/SPARK-37850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37850: Assignee: Apache Spark > Enable flake's E731 rule in PySpark >

[jira] [Assigned] (SPARK-37850) Enable flake's E731 rule in PySpark

[ https://issues.apache.org/jira/browse/SPARK-37850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37850: Assignee: (was: Apache Spark) > Enable flake's E731 rule in PySpark >

[jira] [Commented] (SPARK-37850) Enable flake's E731 rule in PySpark

[ https://issues.apache.org/jira/browse/SPARK-37850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471649#comment-17471649 ] Apache Spark commented on SPARK-37850: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Updated] (SPARK-37850) Enable flake's E731 rule in PySpark

[ https://issues.apache.org/jira/browse/SPARK-37850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37850: - Description: To comply PEP 8, we should enable this rule (see also

[jira] [Created] (SPARK-37850) Enable flake's E731 rule in PySpark

Hyukjin Kwon created SPARK-37850: Summary: Enable flake's E731 rule in PySpark Key: SPARK-37850 URL: https://issues.apache.org/jira/browse/SPARK-37850 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-35442) Support propagate empty relation through aggregate

[ https://issues.apache.org/jira/browse/SPARK-35442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471640#comment-17471640 ] Apache Spark commented on SPARK-35442: -- User 'ulysses-you' has created a pull request for this

[jira] [Commented] (SPARK-35442) Support propagate empty relation through aggregate

[ https://issues.apache.org/jira/browse/SPARK-35442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471641#comment-17471641 ] Apache Spark commented on SPARK-35442: -- User 'ulysses-you' has created a pull request for this

[jira] [Commented] (SPARK-35442) Support propagate empty relation through aggregate

[ https://issues.apache.org/jira/browse/SPARK-35442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471636#comment-17471636 ] Apache Spark commented on SPARK-35442: -- User 'ulysses-you' has created a pull request for this

[jira] [Commented] (SPARK-35442) Support propagate empty relation through aggregate

[ https://issues.apache.org/jira/browse/SPARK-35442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471634#comment-17471634 ] Apache Spark commented on SPARK-35442: -- User 'ulysses-you' has created a pull request for this

[jira] [Commented] (SPARK-37849) Supports limiting the maximum amount of column data

[ https://issues.apache.org/jira/browse/SPARK-37849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471633#comment-17471633 ] melin commented on SPARK-37849: --- Restrictions can be added in HiveOutputWriter {code:java} override def

[jira] [Commented] (SPARK-37768) Schema pruning for the metadata struct

[ https://issues.apache.org/jira/browse/SPARK-37768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471632#comment-17471632 ] Apache Spark commented on SPARK-37768: -- User 'Yaohua628' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37768) Schema pruning for the metadata struct

[ https://issues.apache.org/jira/browse/SPARK-37768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37768: Assignee: (was: Apache Spark) > Schema pruning for the metadata struct >

[jira] [Commented] (SPARK-37768) Schema pruning for the metadata struct

[ https://issues.apache.org/jira/browse/SPARK-37768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471631#comment-17471631 ] Apache Spark commented on SPARK-37768: -- User 'Yaohua628' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37768) Schema pruning for the metadata struct

[ https://issues.apache.org/jira/browse/SPARK-37768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37768: Assignee: Apache Spark > Schema pruning for the metadata struct >

[jira] [Updated] (SPARK-37848) Supports counting the number of records written to each partition when inserting partitioned tables

[ https://issues.apache.org/jira/browse/SPARK-37848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] melin updated SPARK-37848: -- Description: InputMetrics and OutputMetrics have no table name information. Write information can be counted

[jira] [Updated] (SPARK-37848) Supports counting the number of records written to each partition when inserting partitioned tables

[ https://issues.apache.org/jira/browse/SPARK-37848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] melin updated SPARK-37848: -- Attachment: image-2022-01-10-09-10-47-483.png > Supports counting the number of records written to each

[jira] [Commented] (SPARK-37840) Dynamically update the loaded Hive UDF JAR

[ https://issues.apache.org/jira/browse/SPARK-37840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471629#comment-17471629 ] Hyukjin Kwon commented on SPARK-37840: -- I think we can already overwrite the jars, no? >

[jira] [Resolved] (SPARK-37849) Supports limiting the maximum amount of column data

[ https://issues.apache.org/jira/browse/SPARK-37849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37849. -- Resolution: Invalid > Supports limiting the maximum amount of column data >

[jira] [Commented] (SPARK-37848) Supports counting the number of records written to each partition when inserting partitioned tables

[ https://issues.apache.org/jira/browse/SPARK-37848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471628#comment-17471628 ] Hyukjin Kwon commented on SPARK-37848: -- Image is broken. can you double check please [~melin]? >

[jira] [Commented] (SPARK-37849) Supports limiting the maximum amount of column data

[ https://issues.apache.org/jira/browse/SPARK-37849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471627#comment-17471627 ] Hyukjin Kwon commented on SPARK-37849: -- You can simply leverage assert_true(size(...)) expression

[jira] [Resolved] (SPARK-37828) Push down filters through RebalancePartitions

[ https://issues.apache.org/jira/browse/SPARK-37828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-37828. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35118

[jira] [Assigned] (SPARK-37828) Push down filters through RebalancePartitions

[ https://issues.apache.org/jira/browse/SPARK-37828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-37828: --- Assignee: Yuming Wang > Push down filters through RebalancePartitions >

[jira] [Assigned] (SPARK-37838) Upgrade scalatestplus artifacts to 3.3.0-SNAP3

[ https://issues.apache.org/jira/browse/SPARK-37838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-37838: --- Assignee: Yuming Wang > Upgrade scalatestplus artifacts to 3.3.0-SNAP3 >

[jira] [Resolved] (SPARK-37838) Upgrade scalatestplus artifacts to 3.3.0-SNAP3

[ https://issues.apache.org/jira/browse/SPARK-37838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-37838. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35128

[jira] [Resolved] (SPARK-37835) Fix the comments on SQLQueryTestSuite.scala/ThriftServerQueryTestSuite.scala to more explicit

[ https://issues.apache.org/jira/browse/SPARK-37835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37835. -- Fix Version/s: 3.3.0 Assignee: Haejoon Lee Resolution: Fixed Fixed in

[jira] [Updated] (SPARK-37835) Fix the comments on SQLQueryTestSuite.scala/ThriftServerQueryTestSuite.scala to more explicit

[ https://issues.apache.org/jira/browse/SPARK-37835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37835: - Priority: Minor (was: Major) > Fix the comments on

[jira] [Updated] (SPARK-37835) Fix the comments on SQLQueryTestSuite.scala/ThriftServerQueryTestSuite.scala to more explicit

[ https://issues.apache.org/jira/browse/SPARK-37835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37835: - Priority: Trivial (was: Minor) > Fix the comments on

[jira] [Updated] (SPARK-37835) Fix the comments on SQLQueryTestSuite.scala/ThriftServerQueryTestSuite.scala to more explicit

[ https://issues.apache.org/jira/browse/SPARK-37835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37835: - Component/s: Documentation > Fix the comments on

[jira] [Created] (SPARK-37849) Supports limiting the maximum amount of column data

melin created SPARK-37849: - Summary: Supports limiting the maximum amount of column data Key: SPARK-37849 URL: https://issues.apache.org/jira/browse/SPARK-37849 Project: Spark Issue Type:

[jira] [Created] (SPARK-37848) Supports counting the number of records written to each partition when inserting partitioned tables

melin created SPARK-37848: - Summary: Supports counting the number of records written to each partition when inserting partitioned tables Key: SPARK-37848 URL: https://issues.apache.org/jira/browse/SPARK-37848

[jira] [Assigned] (SPARK-37847) PushBlockStreamCallback should check isTooLate first to avoid NPE

[ https://issues.apache.org/jira/browse/SPARK-37847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37847: Assignee: Apache Spark > PushBlockStreamCallback should check isTooLate first to avoid

[jira] [Assigned] (SPARK-37847) PushBlockStreamCallback should check isTooLate first to avoid NPE

[ https://issues.apache.org/jira/browse/SPARK-37847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37847: Assignee: (was: Apache Spark) > PushBlockStreamCallback should check isTooLate first

[jira] [Commented] (SPARK-37847) PushBlockStreamCallback should check isTooLate first to avoid NPE

[ https://issues.apache.org/jira/browse/SPARK-37847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471371#comment-17471371 ] Apache Spark commented on SPARK-37847: -- User 'pan3793' has created a pull request for this issue:

[jira] [Created] (SPARK-37847) PushBlockStreamCallback should check isTooLate first to avoid NPE

Cheng Pan created SPARK-37847: - Summary: PushBlockStreamCallback should check isTooLate first to avoid NPE Key: SPARK-37847 URL: https://issues.apache.org/jira/browse/SPARK-37847 Project: Spark

[jira] [Commented] (SPARK-37491) Fix Series.asof when values of the series is not sorted

[ https://issues.apache.org/jira/browse/SPARK-37491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471314#comment-17471314 ] pralabhkumar commented on SPARK-37491: -- I am working on it . > Fix Series.asof when values of the

[jira] [Commented] (SPARK-37527) Translate more standard aggregate functions for pushdown

[ https://issues.apache.org/jira/browse/SPARK-37527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471307#comment-17471307 ] Apache Spark commented on SPARK-37527: -- User 'beliefer' has created a pull request for this issue: