[jira] [Commented] (SPARK-40421) Make `spearman` correlation in `DataFrame.corr` support missing values and `min_periods`

[ https://issues.apache.org/jira/browse/SPARK-40421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603892#comment-17603892 ] Apache Spark commented on SPARK-40421: -- User 'zhengruifeng' has created a pull request for this

[jira] [Assigned] (SPARK-40421) Make `spearman` correlation in `DataFrame.corr` support missing values and `min_periods`

[ https://issues.apache.org/jira/browse/SPARK-40421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40421: Assignee: Apache Spark > Make `spearman` correlation in `DataFrame.corr` support missing

[jira] [Assigned] (SPARK-40421) Make `spearman` correlation in `DataFrame.corr` support missing values and `min_periods`

[ https://issues.apache.org/jira/browse/SPARK-40421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40421: Assignee: (was: Apache Spark) > Make `spearman` correlation in `DataFrame.corr`

[jira] [Commented] (SPARK-40421) Make `spearman` correlation in `DataFrame.corr` support missing values and `min_periods`

[ https://issues.apache.org/jira/browse/SPARK-40421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603891#comment-17603891 ] Apache Spark commented on SPARK-40421: -- User 'zhengruifeng' has created a pull request for this

[jira] [Created] (SPARK-40421) Make `spearman` correlation in `DataFrame.corr` support missing values and `min_periods`

Ruifeng Zheng created SPARK-40421: - Summary: Make `spearman` correlation in `DataFrame.corr` support missing values and `min_periods` Key: SPARK-40421 URL: https://issues.apache.org/jira/browse/SPARK-40421

[jira] [Created] (SPARK-40420) Sort message parameters in the JSON formats

Max Gekk created SPARK-40420: Summary: Sort message parameters in the JSON formats Key: SPARK-40420 URL: https://issues.apache.org/jira/browse/SPARK-40420 Project: Spark Issue Type: Sub-task

[jira] [Comment Edited] (SPARK-40303) The performance will be worse after codegen

[ https://issues.apache.org/jira/browse/SPARK-40303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603883#comment-17603883 ] Yang Jie edited comment on SPARK-40303 at 9/14/22 5:24 AM: --- I did a simple

[jira] [Commented] (SPARK-40303) The performance will be worse after codegen

[ https://issues.apache.org/jira/browse/SPARK-40303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603883#comment-17603883 ] Yang Jie commented on SPARK-40303: -- I did a simple experiment to compare the following scenarios： # A

[jira] [Updated] (SPARK-40303) The performance will be worse after codegen

[ https://issues.apache.org/jira/browse/SPARK-40303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-40303: - Attachment: TestApiBenchmark.scala TestApis.java TestParameters.java >

[jira] [Resolved] (SPARK-40414) Fix PythonArrowInput and PythonArrowOutput to be more generic to handle complicated type/data

[ https://issues.apache.org/jira/browse/SPARK-40414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-40414. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37864

[jira] [Assigned] (SPARK-40414) Fix PythonArrowInput and PythonArrowOutput to be more generic to handle complicated type/data

[ https://issues.apache.org/jira/browse/SPARK-40414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-40414: Assignee: Jungtaek Lim > Fix PythonArrowInput and PythonArrowOutput to be more generic

[jira] [Commented] (SPARK-40419) Integrate Grouped Aggregate Pandas UDFs into *.sql test cases

[ https://issues.apache.org/jira/browse/SPARK-40419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603880#comment-17603880 ] Apache Spark commented on SPARK-40419: -- User 'itholic' has created a pull request for this issue:

[jira] [Assigned] (SPARK-40419) Integrate Grouped Aggregate Pandas UDFs into *.sql test cases

[ https://issues.apache.org/jira/browse/SPARK-40419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40419: Assignee: (was: Apache Spark) > Integrate Grouped Aggregate Pandas UDFs into *.sql

[jira] [Assigned] (SPARK-40419) Integrate Grouped Aggregate Pandas UDFs into *.sql test cases

[ https://issues.apache.org/jira/browse/SPARK-40419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40419: Assignee: Apache Spark > Integrate Grouped Aggregate Pandas UDFs into *.sql test cases >

[jira] [Updated] (SPARK-40419) Integrate Grouped Aggregate Pandas UDFs into *.sql test cases

[ https://issues.apache.org/jira/browse/SPARK-40419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-40419: Summary: Integrate Grouped Aggregate Pandas UDFs into *.sql test cases (was: Integrate aggregate

[jira] [Created] (SPARK-40419) Integrate aggregate pandas UDFs into *.sql test cases

Haejoon Lee created SPARK-40419: --- Summary: Integrate aggregate pandas UDFs into *.sql test cases Key: SPARK-40419 URL: https://issues.apache.org/jira/browse/SPARK-40419 Project: Spark Issue

[jira] [Commented] (SPARK-40390) Spark Master UI - SSL implementation

[ https://issues.apache.org/jira/browse/SPARK-40390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603867#comment-17603867 ] Rhajvijay Manoharan commented on SPARK-40390: - [~hyukjin.kwon] We will try using the 3.X

[jira] [Commented] (SPARK-40390) Spark Master UI - SSL implementation

[ https://issues.apache.org/jira/browse/SPARK-40390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603862#comment-17603862 ] Hyukjin Kwon commented on SPARK-40390: -- Is this still an issue in Spark 3.1+? Spark 2.x is EOL >

[jira] [Commented] (SPARK-40405) sparksql throws exception while reading by jdbc

[ https://issues.apache.org/jira/browse/SPARK-40405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603861#comment-17603861 ] Hyukjin Kwon commented on SPARK-40405: -- [~ghsea]seems like it's a classpath problem. how do you

[jira] [Updated] (SPARK-40418) Increase default initialNumPartitions to 10

[ https://issues.apache.org/jira/browse/SPARK-40418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-40418: - Component/s: SQL > Increase default initialNumPartitions to 10 >

[jira] [Updated] (SPARK-40405) sparksql throws exception while reading by jdbc

[ https://issues.apache.org/jira/browse/SPARK-40405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-40405: - Priority: Major (was: Critical) > sparksql throws exception while reading by jdbc >

[jira] [Assigned] (SPARK-40403) Negative size in error message when unsafe array is too big

[ https://issues.apache.org/jira/browse/SPARK-40403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-40403: Assignee: Bruce Robbins > Negative size in error message when unsafe array is too big >

[jira] [Resolved] (SPARK-40403) Negative size in error message when unsafe array is too big

[ https://issues.apache.org/jira/browse/SPARK-40403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40403. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37852

[jira] [Commented] (SPARK-40286) Load Data from S3 deletes data source file

[ https://issues.apache.org/jira/browse/SPARK-40286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603826#comment-17603826 ] Drew commented on SPARK-40286: -- Hi [~ste...@apache.org], Yeah, is there anything significant there that I

[jira] [Commented] (SPARK-40287) Load Data using Spark by a single partition moves entire dataset under same location in S3

[ https://issues.apache.org/jira/browse/SPARK-40287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603819#comment-17603819 ] Drew commented on SPARK-40287: -- Hey [~ste...@apache.org], Yes, I get the same functionality with this

[jira] [Deleted] (SPARK-40309) Introduce sql_conf context manager for pyspark.sql

[ https://issues.apache.org/jira/browse/SPARK-40309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng deleted SPARK-40309: - > Introduce sql_conf context manager for pyspark.sql >

[jira] [Assigned] (SPARK-40384) Do base image real in time build only when infra dockerfile is changed

[ https://issues.apache.org/jira/browse/SPARK-40384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40384: Assignee: Yikun Jiang (was: Apache Spark) > Do base image real in time build only when

[jira] [Assigned] (SPARK-40384) Do base image real in time build only when infra dockerfile is changed

[ https://issues.apache.org/jira/browse/SPARK-40384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40384: Assignee: Apache Spark (was: Yikun Jiang) > Do base image real in time build only when

[jira] [Commented] (SPARK-40384) Do base image real in time build only when infra dockerfile is changed

[ https://issues.apache.org/jira/browse/SPARK-40384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603798#comment-17603798 ] Hyukjin Kwon commented on SPARK-40384: -- Reverted at

[jira] [Reopened] (SPARK-40384) Do base image real in time build only when infra dockerfile is changed

[ https://issues.apache.org/jira/browse/SPARK-40384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-40384: -- > Do base image real in time build only when infra dockerfile is changed >

[jira] [Updated] (SPARK-40384) Do base image real in time build only when infra dockerfile is changed

[ https://issues.apache.org/jira/browse/SPARK-40384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-40384: - Fix Version/s: (was: 3.4.0) > Do base image real in time build only when infra dockerfile

[jira] [Resolved] (SPARK-40417) Use YuniKorn v1.1+

[ https://issues.apache.org/jira/browse/SPARK-40417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-40417. --- Fix Version/s: 3.3.1 3.4.0 Resolution: Fixed Issue resolved by

[jira] [Assigned] (SPARK-40417) Use YuniKorn v1.1+

[ https://issues.apache.org/jira/browse/SPARK-40417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-40417: - Assignee: Dongjoon Hyun > Use YuniKorn v1.1+ > -- > >

[jira] [Assigned] (SPARK-40362) Bug in Canonicalization of expressions like Add & Multiply i.e Commutative Operators

[ https://issues.apache.org/jira/browse/SPARK-40362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-40362: - Assignee: Peter Toth > Bug in Canonicalization of expressions like Add & Multiply i.e

[jira] [Resolved] (SPARK-40362) Bug in Canonicalization of expressions like Add & Multiply i.e Commutative Operators

[ https://issues.apache.org/jira/browse/SPARK-40362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-40362. --- Fix Version/s: 3.3.1 Resolution: Fixed Issue resolved by pull request 37866

[jira] [Created] (SPARK-40418) Increase default initialNumPartitions to 10

Ziqi Liu created SPARK-40418: Summary: Increase default initialNumPartitions to 10 Key: SPARK-40418 URL: https://issues.apache.org/jira/browse/SPARK-40418 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-40417) Use YuniKorn v1.1+

[ https://issues.apache.org/jira/browse/SPARK-40417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-40417: -- Issue Type: Documentation (was: Improvement) > Use YuniKorn v1.1+ > -- > >

[jira] [Assigned] (SPARK-40417) Use YuniKorn v1.1+

[ https://issues.apache.org/jira/browse/SPARK-40417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40417: Assignee: (was: Apache Spark) > Use YuniKorn v1.1+ > -- > >

[jira] [Assigned] (SPARK-40417) Use YuniKorn v1.1+

[ https://issues.apache.org/jira/browse/SPARK-40417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40417: Assignee: Apache Spark > Use YuniKorn v1.1+ > -- > >

[jira] [Commented] (SPARK-40417) Use YuniKorn v1.1+

[ https://issues.apache.org/jira/browse/SPARK-40417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603710#comment-17603710 ] Apache Spark commented on SPARK-40417: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Created] (SPARK-40417) Use YuniKorn v1.1+

Dongjoon Hyun created SPARK-40417: - Summary: Use YuniKorn v1.1+ Key: SPARK-40417 URL: https://issues.apache.org/jira/browse/SPARK-40417 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-40416) Add error classes for subquery expression CheckAnalysis failures

[ https://issues.apache.org/jira/browse/SPARK-40416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40416: Assignee: (was: Apache Spark) > Add error classes for subquery expression

[jira] [Commented] (SPARK-40416) Add error classes for subquery expression CheckAnalysis failures

[ https://issues.apache.org/jira/browse/SPARK-40416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603682#comment-17603682 ] Apache Spark commented on SPARK-40416: -- User 'dtenedor' has created a pull request for this issue:

[jira] [Assigned] (SPARK-40416) Add error classes for subquery expression CheckAnalysis failures

[ https://issues.apache.org/jira/browse/SPARK-40416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40416: Assignee: Apache Spark > Add error classes for subquery expression CheckAnalysis

[jira] [Created] (SPARK-40416) Add error classes for subquery expression CheckAnalysis failures

Daniel created SPARK-40416: -- Summary: Add error classes for subquery expression CheckAnalysis failures Key: SPARK-40416 URL: https://issues.apache.org/jira/browse/SPARK-40416 Project: Spark Issue

[jira] [Updated] (SPARK-40394) Move CheckAnalysis error messages to use the new error framework

[ https://issues.apache.org/jira/browse/SPARK-40394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel updated SPARK-40394: --- Summary: Move CheckAnalysis error messages to use the new error framework (was: Move subquery expression

[jira] [Resolved] (SPARK-40415) Wrong version of okio in spark-deps file

[ https://issues.apache.org/jira/browse/SPARK-40415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-40415. -- Resolution: Won't Fix > Wrong version of okio in spark-deps file >

[jira] [Updated] (SPARK-33152) SPIP: Constraint Propagation code causes OOM issues or increasing compilation time to hours

[ https://issues.apache.org/jira/browse/SPARK-33152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-33152: - Shepherd: Wenchen Fan (was: Arnaud Doucet) Description: h2. Q1. What are you trying to do? Articulate

[jira] [Commented] (SPARK-33152) SPIP: Constraint Propagation code causes OOM issues or increasing compilation time to hours

[ https://issues.apache.org/jira/browse/SPARK-33152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603664#comment-17603664 ] Apache Spark commented on SPARK-33152: -- User 'ahshahid' has created a pull request for this issue:

[jira] [Commented] (SPARK-33152) SPIP: Constraint Propagation code causes OOM issues or increasing compilation time to hours

[ https://issues.apache.org/jira/browse/SPARK-33152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603663#comment-17603663 ] Apache Spark commented on SPARK-33152: -- User 'ahshahid' has created a pull request for this issue:

[jira] [Commented] (SPARK-40397) Migrate selenium-java from 3.1 to 4.2 and upgrade org.scalatestplus:selenium to 3.2.13.0

[ https://issues.apache.org/jira/browse/SPARK-40397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603652#comment-17603652 ] Apache Spark commented on SPARK-40397: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-40397) Migrate selenium-java from 3.1 to 4.2 and upgrade org.scalatestplus:selenium to 3.2.13.0

[ https://issues.apache.org/jira/browse/SPARK-40397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40397: Assignee: (was: Apache Spark) > Migrate selenium-java from 3.1 to 4.2 and upgrade

[jira] [Assigned] (SPARK-40397) Migrate selenium-java from 3.1 to 4.2 and upgrade org.scalatestplus:selenium to 3.2.13.0

[ https://issues.apache.org/jira/browse/SPARK-40397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40397: Assignee: Apache Spark > Migrate selenium-java from 3.1 to 4.2 and upgrade

[jira] [Commented] (SPARK-40397) Migrate selenium-java from 3.1 to 4.2 and upgrade org.scalatestplus:selenium to 3.2.13.0

[ https://issues.apache.org/jira/browse/SPARK-40397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603650#comment-17603650 ] Apache Spark commented on SPARK-40397: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-40415) Wrong version of okio in spark-deps file

[ https://issues.apache.org/jira/browse/SPARK-40415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40415: Assignee: Apache Spark > Wrong version of okio in spark-deps file >

[jira] [Assigned] (SPARK-40415) Wrong version of okio in spark-deps file

[ https://issues.apache.org/jira/browse/SPARK-40415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40415: Assignee: (was: Apache Spark) > Wrong version of okio in spark-deps file >

[jira] [Commented] (SPARK-40415) Wrong version of okio in spark-deps file

[ https://issues.apache.org/jira/browse/SPARK-40415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603631#comment-17603631 ] Apache Spark commented on SPARK-40415: -- User 'LuciferYang' has created a pull request for this

[jira] [Updated] (SPARK-40415) Wrong version of okio in spark-deps file

[ https://issues.apache.org/jira/browse/SPARK-40415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-40415: - Attachment: tree-before.txt > Wrong version of okio in spark-deps file >

[jira] [Created] (SPARK-40415) Wrong version of okio in spark-deps file

Yang Jie created SPARK-40415: Summary: Wrong version of okio in spark-deps file Key: SPARK-40415 URL: https://issues.apache.org/jira/browse/SPARK-40415 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-40362) Bug in Canonicalization of expressions like Add & Multiply i.e Commutative Operators

[ https://issues.apache.org/jira/browse/SPARK-40362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603619#comment-17603619 ] Apache Spark commented on SPARK-40362: -- User 'peter-toth' has created a pull request for this

[jira] [Commented] (SPARK-40334) Implement `GroupBy.prod`.

[ https://issues.apache.org/jira/browse/SPARK-40334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603611#comment-17603611 ] Haejoon Lee commented on SPARK-40334: - [~ayudovin] No worries! Please keep working on your work!

[jira] [Commented] (SPARK-40384) Do base image real in time build only when infra dockerfile is changed

[ https://issues.apache.org/jira/browse/SPARK-40384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603603#comment-17603603 ] Apache Spark commented on SPARK-40384: -- User 'Yikun' has created a pull request for this issue:

[jira] [Resolved] (SPARK-40400) Pass error message parameters to exceptions as a map

[ https://issues.apache.org/jira/browse/SPARK-40400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-40400. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37834

[jira] [Commented] (SPARK-20624) SPIP: Add better handling for node shutdown

[ https://issues.apache.org/jira/browse/SPARK-20624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603577#comment-17603577 ] Juliusz Sompolski commented on SPARK-20624: --- [~holden] Are these new APIs documented? I can't

[jira] [Resolved] (SPARK-40384) Do base image real in time build only when infra dockerfile is changed

[ https://issues.apache.org/jira/browse/SPARK-40384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40384. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37828

[jira] [Assigned] (SPARK-40384) Do base image real in time build only when infra dockerfile is changed

[ https://issues.apache.org/jira/browse/SPARK-40384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-40384: Assignee: Yikun Jiang > Do base image real in time build only when infra dockerfile is

[jira] [Assigned] (SPARK-40414) Fix PythonArrowInput and PythonArrowOutput to be more generic to handle complicated type/data

[ https://issues.apache.org/jira/browse/SPARK-40414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40414: Assignee: Apache Spark > Fix PythonArrowInput and PythonArrowOutput to be more generic

[jira] [Commented] (SPARK-40414) Fix PythonArrowInput and PythonArrowOutput to be more generic to handle complicated type/data

[ https://issues.apache.org/jira/browse/SPARK-40414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603571#comment-17603571 ] Apache Spark commented on SPARK-40414: -- User 'HeartSaVioR' has created a pull request for this

[jira] [Assigned] (SPARK-40414) Fix PythonArrowInput and PythonArrowOutput to be more generic to handle complicated type/data

[ https://issues.apache.org/jira/browse/SPARK-40414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40414: Assignee: (was: Apache Spark) > Fix PythonArrowInput and PythonArrowOutput to be

[jira] [Commented] (SPARK-40414) Fix PythonArrowInput and PythonArrowOutput to be more generic to handle complicated type/data

[ https://issues.apache.org/jira/browse/SPARK-40414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603559#comment-17603559 ] Jungtaek Lim commented on SPARK-40414: -- Will submit a PR sooner. > Fix PythonArrowInput and

[jira] [Created] (SPARK-40414) Fix PythonArrowInput and PythonArrowOutput to be more generic to handle complicated type/data

Jungtaek Lim created SPARK-40414: Summary: Fix PythonArrowInput and PythonArrowOutput to be more generic to handle complicated type/data Key: SPARK-40414 URL: https://issues.apache.org/jira/browse/SPARK-40414

[jira] [Commented] (SPARK-40412) limit(x,y) + 子查询 出现数据丢失和乱序问题

[ https://issues.apache.org/jira/browse/SPARK-40412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603485#comment-17603485 ] FengJia commented on SPARK-40412: - Huawei Cloud's solution is to add Order by > limit(x,y) + 子查询

[jira] [Resolved] (SPARK-40412) limit(x,y) + 子查询 出现数据丢失和乱序问题

[ https://issues.apache.org/jira/browse/SPARK-40412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-40412. - Resolution: Invalid Spark SQL does not support \{{limit n, m}}. Please contact the huawei

[jira] [Commented] (SPARK-40412) limit(x,y) + 子查询 出现数据丢失和乱序问题

[ https://issues.apache.org/jira/browse/SPARK-40412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603473#comment-17603473 ] FengJia commented on SPARK-40412: - The Huawei cloud I use cannot change the Spark version,Did you make a

[jira] [Commented] (SPARK-40334) Implement `GroupBy.prod`.

[ https://issues.apache.org/jira/browse/SPARK-40334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603463#comment-17603463 ] Artsiom Yudovin commented on SPARK-40334: - [~itholic], Hi, I have been started to work on this

[jira] [Commented] (SPARK-40324) Provide a query context of ParseException

[ https://issues.apache.org/jira/browse/SPARK-40324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603462#comment-17603462 ] Apache Spark commented on SPARK-40324: -- User 'gengliangwang' has created a pull request for this

[jira] [Updated] (SPARK-40178) Rebalance/Repartition Hints Not Working in PySpark

[ https://issues.apache.org/jira/browse/SPARK-40178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-40178: Target Version/s: (was: 3.2.0, 3.2.1, 3.3.0, 3.2.2, 3.4.0, 3.3.1) > Rebalance/Repartition Hints

[jira] [Updated] (SPARK-40178) Rebalance/Repartition Hints Not Working in PySpark

[ https://issues.apache.org/jira/browse/SPARK-40178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-40178: Fix Version/s: (was: 3.2.0) (was: 3.3.0) (was:

[jira] [Updated] (SPARK-40177) Simplify join condition of form (a==b) || (a==null&==null) to a<=>b

[ https://issues.apache.org/jira/browse/SPARK-40177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-40177: Target Version/s: (was: 3.3.1) > Simplify join condition of form (a==b) || (a==null&==null) to

[jira] [Updated] (SPARK-40177) Simplify join condition of form (a==b) || (a==null&==null) to a<=>b

[ https://issues.apache.org/jira/browse/SPARK-40177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-40177: Fix Version/s: (was: 3.3.1) > Simplify join condition of form (a==b) || (a==null&==null) to

[jira] [Updated] (SPARK-40413) Column.isin produces non-boolean results

[ https://issues.apache.org/jira/browse/SPARK-40413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andreas Franz updated SPARK-40413: -- Description: I observed an inconsistent behaviour using the Column.isin function. The

[jira] [Resolved] (SPARK-38734) Test the error class: INDEX_OUT_OF_BOUNDS

[ https://issues.apache.org/jira/browse/SPARK-38734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-38734. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37857

[jira] [Assigned] (SPARK-38734) Test the error class: INDEX_OUT_OF_BOUNDS

[ https://issues.apache.org/jira/browse/SPARK-38734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-38734: Assignee: Max Gekk > Test the error class: INDEX_OUT_OF_BOUNDS >

[jira] [Created] (SPARK-40413) Column.isin produces non-boolean results

Andreas Franz created SPARK-40413: - Summary: Column.isin produces non-boolean results Key: SPARK-40413 URL: https://issues.apache.org/jira/browse/SPARK-40413 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-40412) limit(x,y) + 子查询 出现数据丢失和乱序问题

[ https://issues.apache.org/jira/browse/SPARK-40412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-40412: Fix Version/s: (was: 2.4.5) > limit(x,y) + 子查询 出现数据丢失和乱序问题 > > >

[jira] [Commented] (SPARK-40412) limit(x,y) + 子查询 出现数据丢失和乱序问题

[ https://issues.apache.org/jira/browse/SPARK-40412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603436#comment-17603436 ] Yuming Wang commented on SPARK-40412: - Could you test the latest Spark? > limit(x,y) + 子查询

[jira] [Updated] (SPARK-40412) limit(x,y) + 子查询 出现数据丢失和乱序问题

[ https://issues.apache.org/jira/browse/SPARK-40412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-40412: Target Version/s: (was: 2.4.5) > limit(x,y) + 子查询 出现数据丢失和乱序问题 > >

[jira] [Commented] (SPARK-40342) Implement `Rolling.quantile`.

[ https://issues.apache.org/jira/browse/SPARK-40342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603430#comment-17603430 ] Apache Spark commented on SPARK-40342: -- User 'Yikun' has created a pull request for this issue:

[jira] [Assigned] (SPARK-40342) Implement `Rolling.quantile`.

[ https://issues.apache.org/jira/browse/SPARK-40342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40342: Assignee: Apache Spark > Implement `Rolling.quantile`. > - >

[jira] [Commented] (SPARK-40342) Implement `Rolling.quantile`.

[ https://issues.apache.org/jira/browse/SPARK-40342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603429#comment-17603429 ] Apache Spark commented on SPARK-40342: -- User 'Yikun' has created a pull request for this issue:

[jira] [Assigned] (SPARK-40342) Implement `Rolling.quantile`.

[ https://issues.apache.org/jira/browse/SPARK-40342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40342: Assignee: (was: Apache Spark) > Implement `Rolling.quantile`. >

[jira] [Commented] (SPARK-33782) Place spark.files, spark.jars and spark.files under the current working directory on the driver in K8S cluster mode

[ https://issues.apache.org/jira/browse/SPARK-33782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603426#comment-17603426 ] pralabhkumar commented on SPARK-33782: -- [~dongjoon] Please review the PR .  > Place spark.files,

[jira] [Assigned] (SPARK-40347) Implement `RollingGroupby.median`.

[ https://issues.apache.org/jira/browse/SPARK-40347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-40347: - Assignee: Yikun Jiang > Implement `RollingGroupby.median`. >

[jira] [Assigned] (SPARK-40330) Implement `Series.searchsorted`.

[ https://issues.apache.org/jira/browse/SPARK-40330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-40330: - Assignee: Ruifeng Zheng > Implement `Series.searchsorted`. >

[jira] [Assigned] (SPARK-40344) Implement `ExpandingGroupby.median`.

[ https://issues.apache.org/jira/browse/SPARK-40344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-40344: - Assignee: Yikun Jiang > Implement `ExpandingGroupby.median`. >

[jira] [Assigned] (SPARK-40341) Implement `Rolling.median`.

[ https://issues.apache.org/jira/browse/SPARK-40341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-40341: - Assignee: Yikun Jiang > Implement `Rolling.median`. > --- > >

[jira] [Resolved] (SPARK-40399) Make `pearson` correlation in `DataFrame.corr` support missing values and `min_periods`

[ https://issues.apache.org/jira/browse/SPARK-40399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-40399. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37845

[jira] [Assigned] (SPARK-40399) Make `pearson` correlation in `DataFrame.corr` support missing values and `min_periods`

[ https://issues.apache.org/jira/browse/SPARK-40399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-40399: - Assignee: Ruifeng Zheng > Make `pearson` correlation in `DataFrame.corr` support

[jira] [Commented] (SPARK-40348) Implement `RollingGroupby.quantile`.

[ https://issues.apache.org/jira/browse/SPARK-40348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603383#comment-17603383 ] Apache Spark commented on SPARK-40348: -- User 'Yikun' has created a pull request for this issue:

[jira] [Commented] (SPARK-40348) Implement `RollingGroupby.quantile`.

[ https://issues.apache.org/jira/browse/SPARK-40348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603384#comment-17603384 ] Apache Spark commented on SPARK-40348: -- User 'Yikun' has created a pull request for this issue:

  1   2   >