[jira] [Assigned] (SPARK-40486) Implement `spearman` and `kendall` in `DataFrame.corrwith`

2022-09-19 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-40486: - Assignee: Ruifeng Zheng > Implement `spearman` and `kendall` in `DataFrame.corrwith` >

[jira] [Resolved] (SPARK-40486) Implement `spearman` and `kendall` in `DataFrame.corrwith`

2022-09-19 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-40486. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37929 [https://

[jira] [Commented] (SPARK-40495) Add additional tests to StreamingSessionWindowSuite

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606940#comment-17606940 ] Apache Spark commented on SPARK-40495: -- User 'WweiL' has created a pull request for

[jira] [Assigned] (SPARK-40495) Add additional tests to StreamingSessionWindowSuite

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40495: Assignee: (was: Apache Spark) > Add additional tests to StreamingSessionWindowSuite >

[jira] [Assigned] (SPARK-40495) Add additional tests to StreamingSessionWindowSuite

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40495: Assignee: Apache Spark > Add additional tests to StreamingSessionWindowSuite > --

[jira] [Commented] (SPARK-40495) Add additional tests to StreamingSessionWindowSuite

2022-09-19 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606937#comment-17606937 ] Wei Liu commented on SPARK-40495: - Hi there, this is my first time to contribute to OSS

[jira] [Created] (SPARK-40495) Add additional tests to StreamingSessionWindowSuite

2022-09-19 Thread Wei Liu (Jira)
Wei Liu created SPARK-40495: --- Summary: Add additional tests to StreamingSessionWindowSuite Key: SPARK-40495 URL: https://issues.apache.org/jira/browse/SPARK-40495 Project: Spark Issue Type: Test

[jira] [Updated] (SPARK-40493) Revert "[SPARK-33861][SQL] Simplify conditional in predicate"

2022-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-40493: -- Fix Version/s: 3.3.2 (was: 3.3.1) > Revert "[SPARK-33861][SQL] Simplify

[jira] [Assigned] (SPARK-40494) Optimize the performance of `keys.zipWithIndex.toMap` code pattern

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40494: Assignee: (was: Apache Spark) > Optimize the performance of `keys.zipWithIndex.toMap`

[jira] [Assigned] (SPARK-40494) Optimize the performance of `keys.zipWithIndex.toMap` code pattern

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40494: Assignee: Apache Spark > Optimize the performance of `keys.zipWithIndex.toMap` code patte

[jira] [Commented] (SPARK-40494) Optimize the performance of `keys.zipWithIndex.toMap` code pattern

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606915#comment-17606915 ] Apache Spark commented on SPARK-40494: -- User 'LuciferYang' has created a pull reque

[jira] [Updated] (SPARK-40493) Revert "[SPARK-33861][SQL] Simplify conditional in predicate"

2022-09-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-40493: Description: Please see https://github.com/apache/spark/pull/30865#issuecomment-755285940 for more

[jira] [Assigned] (SPARK-40493) Revert "[SPARK-33861][SQL] Simplify conditional in predicate"

2022-09-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-40493: --- Assignee: Yuming Wang > Revert "[SPARK-33861][SQL] Simplify conditional in predicate" > ---

[jira] [Commented] (SPARK-40493) Revert "[SPARK-33861][SQL] Simplify conditional in predicate"

2022-09-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606907#comment-17606907 ] Yuming Wang commented on SPARK-40493: - Issue resolved by pull request 37729 https://

[jira] [Created] (SPARK-40494) Optimize the performance of `keys.zipWithIndex.toMap` code pattern

2022-09-19 Thread Yang Jie (Jira)
Yang Jie created SPARK-40494: Summary: Optimize the performance of `keys.zipWithIndex.toMap` code pattern Key: SPARK-40494 URL: https://issues.apache.org/jira/browse/SPARK-40494 Project: Spark

[jira] [Resolved] (SPARK-40493) Revert "[SPARK-33861][SQL] Simplify conditional in predicate"

2022-09-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-40493. - Resolution: Fixed > Revert "[SPARK-33861][SQL] Simplify conditional in predicate" >

[jira] [Updated] (SPARK-40493) Revert "[SPARK-33861][SQL] Simplify conditional in predicate"

2022-09-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-40493: Fix Version/s: 3.2.3 > Revert "[SPARK-33861][SQL] Simplify conditional in predicate" > ---

[jira] [Updated] (SPARK-40493) Revert "[SPARK-33861][SQL] Simplify conditional in predicate"

2022-09-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-40493: Fix Version/s: 3.3.1 > Revert "[SPARK-33861][SQL] Simplify conditional in predicate" > ---

[jira] [Updated] (SPARK-40493) Revert "[SPARK-33861][SQL] Simplify conditional in predicate"

2022-09-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-40493: Affects Version/s: 3.2.2 3.2.1 3.2.0 > Revert "[SPAR

[jira] [Created] (SPARK-40493) Revert "[SPARK-33861][SQL] Simplify conditional in predicate"

2022-09-19 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-40493: --- Summary: Revert "[SPARK-33861][SQL] Simplify conditional in predicate" Key: SPARK-40493 URL: https://issues.apache.org/jira/browse/SPARK-40493 Project: Spark

[jira] [Updated] (SPARK-38803) Set minio cpu to 250m (0.25) in K8s IT

2022-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38803: -- Fix Version/s: 3.3.2 > Set minio cpu to 250m (0.25) in K8s IT > --

[jira] [Updated] (SPARK-38802) Support spark.kubernetes.test.(driver|executor)RequestCores

2022-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38802: -- Fix Version/s: 3.3.2 > Support spark.kubernetes.test.(driver|executor)RequestCores > -

[jira] [Commented] (SPARK-40492) Perform maintenance of StateStore instances when they become inactive

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606885#comment-17606885 ] Apache Spark commented on SPARK-40492: -- User 'chaoqin-li1123' has created a pull re

[jira] [Commented] (SPARK-40492) Perform maintenance of StateStore instances when they become inactive

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606884#comment-17606884 ] Apache Spark commented on SPARK-40492: -- User 'chaoqin-li1123' has created a pull re

[jira] [Assigned] (SPARK-40492) Perform maintenance of StateStore instances when they become inactive

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40492: Assignee: Apache Spark > Perform maintenance of StateStore instances when they become ina

[jira] [Assigned] (SPARK-40492) Perform maintenance of StateStore instances when they become inactive

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40492: Assignee: (was: Apache Spark) > Perform maintenance of StateStore instances when they

[jira] [Commented] (SPARK-40472) Improve pyspark.sql.function example experience

2022-09-19 Thread deshanxiao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606879#comment-17606879 ] deshanxiao commented on SPARK-40472: [~hyukjin.kwon] OK, thanks~  > Improve pyspark

[jira] [Resolved] (SPARK-40472) Improve pyspark.sql.function example experience

2022-09-19 Thread deshanxiao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] deshanxiao resolved SPARK-40472. Resolution: Fixed > Improve pyspark.sql.function example experience >

[jira] [Created] (SPARK-40492) Perform maintenance of StateStore instances when they become inactive

2022-09-19 Thread Chaoqin Li (Jira)
Chaoqin Li created SPARK-40492: -- Summary: Perform maintenance of StateStore instances when they become inactive Key: SPARK-40492 URL: https://issues.apache.org/jira/browse/SPARK-40492 Project: Spark

[jira] [Updated] (SPARK-37275) Support ANSI intervals in PySpark

2022-09-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-37275: - Fix Version/s: 3.3.0 > Support ANSI intervals in PySpark > - > >

[jira] [Commented] (SPARK-40489) Spark 3.3.0 breaks with SFL4J 2.

2022-09-19 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606875#comment-17606875 ] Jungtaek Lim commented on SPARK-40489: -- It would be nice if you can help Spark to r

[jira] [Updated] (SPARK-40489) Spark 3.3.0 breaks with SFL4J 2.

2022-09-19 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-40489: - Priority: Major (was: Critical) > Spark 3.3.0 breaks with SFL4J 2. > --

[jira] [Commented] (SPARK-40489) Spark 3.3.0 breaks with SFL4J 2.

2022-09-19 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606872#comment-17606872 ] Jungtaek Lim commented on SPARK-40489: -- https://www.slf4j.org/news.html 2022-08-20

[jira] [Updated] (SPARK-40460) Streaming metrics is zero when select _metadata

2022-09-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-40460: - Fix Version/s: 3.3.2 > Streaming metrics is zero when select _metadata > ---

[jira] [Comment Edited] (SPARK-40489) Spark 3.3.0 breaks with SFL4J 2.

2022-09-19 Thread Garret Wilson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606852#comment-17606852 ] Garret Wilson edited comment on SPARK-40489 at 9/20/22 3:15 AM: --

[jira] [Commented] (SPARK-40489) Spark 3.3.0 breaks with SFL4J 2.

2022-09-19 Thread Garret Wilson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606852#comment-17606852 ] Garret Wilson commented on SPARK-40489: --- # Dropping explicit Log4J 1.x support is

[jira] [Commented] (SPARK-40491) Expose a jdbcRDD function in SparkContext

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606850#comment-17606850 ] Apache Spark commented on SPARK-40491: -- User 'beliefer' has created a pull request

[jira] [Commented] (SPARK-40490) `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile` reload after SPARK-17321

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606851#comment-17606851 ] Apache Spark commented on SPARK-40490: -- User 'LuciferYang' has created a pull reque

[jira] [Commented] (SPARK-40491) Expose a jdbcRDD function in SparkContext

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606848#comment-17606848 ] Apache Spark commented on SPARK-40491: -- User 'beliefer' has created a pull request

[jira] [Assigned] (SPARK-40490) `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile` reload after SPARK-17321

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40490: Assignee: Apache Spark > `YarnShuffleIntegrationSuite` no longer verifies `registeredExec

[jira] [Assigned] (SPARK-40491) Expose a jdbcRDD function in SparkContext

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40491: Assignee: Apache Spark > Expose a jdbcRDD function in SparkContext >

[jira] [Assigned] (SPARK-40490) `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile` reload after SPARK-17321

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40490: Assignee: (was: Apache Spark) > `YarnShuffleIntegrationSuite` no longer verifies `reg

[jira] [Commented] (SPARK-40490) `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile` reload after SPARK-17321

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606849#comment-17606849 ] Apache Spark commented on SPARK-40490: -- User 'LuciferYang' has created a pull reque

[jira] [Assigned] (SPARK-40491) Expose a jdbcRDD function in SparkContext

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40491: Assignee: (was: Apache Spark) > Expose a jdbcRDD function in SparkContext > -

[jira] [Updated] (SPARK-40491) Expose a jdbcRDD function in SparkContext

2022-09-19 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-40491: --- Description: According to the legacy document of JdbcRDD, we need to expose a jdbcRDD function in Sp

[jira] [Created] (SPARK-40491) Expose a jdbcRDD function in SparkContext

2022-09-19 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-40491: -- Summary: Expose a jdbcRDD function in SparkContext Key: SPARK-40491 URL: https://issues.apache.org/jira/browse/SPARK-40491 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-40490) `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile` reload after SPARK-17321

2022-09-19 Thread Yang Jie (Jira)
Yang Jie created SPARK-40490: Summary: `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile` reload after SPARK-17321 Key: SPARK-40490 URL: https://issues.apache.org/jira/browse/SPARK-40490

[jira] [Commented] (SPARK-33152) SPIP: Constraint Propagation code causes OOM issues or increasing compilation time to hours

2022-09-19 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606817#comment-17606817 ] Asif commented on SPARK-33152: -- Added a test *CompareNewAndOldConstraintsSuite* in the PR w

[jira] [Reopened] (SPARK-39494) Support `createDataFrame` from a list of scalars when schema is not provided

2022-09-19 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng reopened SPARK-39494: -- > Support `createDataFrame` from a list of scalars when schema is not provided > -

[jira] [Resolved] (SPARK-39494) Support `createDataFrame` from a list of scalars when schema is not provided

2022-09-19 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-39494. -- Resolution: Won't Do > Support `createDataFrame` from a list of scalars when schema is not pro

[jira] [Resolved] (SPARK-40084) Upgrade Py4J from 0.10.9.5 to 0.10.9.7

2022-09-19 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-40084. -- Resolution: Resolved > Upgrade Py4J from 0.10.9.5 to 0.10.9.7 > --

[jira] [Commented] (SPARK-40084) Upgrade Py4J from 0.10.9.5 to 0.10.9.7

2022-09-19 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606813#comment-17606813 ] Xinrong Meng commented on SPARK-40084: -- Resolved by https://github.com/apache/spark

[jira] [Assigned] (SPARK-40084) Upgrade Py4J from 0.10.9.5 to 0.10.9.7

2022-09-19 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng reassigned SPARK-40084: Assignee: BingKun Pan > Upgrade Py4J from 0.10.9.5 to 0.10.9.7 >

[jira] [Resolved] (SPARK-39405) NumPy support in SQL

2022-09-19 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-39405. -- Resolution: Resolved > NumPy support in SQL > > > Key: SP

[jira] [Assigned] (SPARK-39405) NumPy support in SQL

2022-09-19 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng reassigned SPARK-39405: Assignee: Xinrong Meng > NumPy support in SQL > > >

[jira] [Resolved] (SPARK-39745) Accept a list that contains NumPy scalars in `createDataFrame`

2022-09-19 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-39745. -- Resolution: Won't Do > Accept a list that contains NumPy scalars in `createDataFrame` > --

[jira] [Assigned] (SPARK-40466) Improve the error message if the DSv2 source is disabled but DSv1 streaming source is not available

2022-09-19 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-40466: Assignee: Huanli Wang > Improve the error message if the DSv2 source is disabled but DSv1

[jira] [Resolved] (SPARK-40466) Improve the error message if the DSv2 source is disabled but DSv1 streaming source is not available

2022-09-19 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-40466. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37917 [https://gi

[jira] [Reopened] (SPARK-40286) Load Data from S3 deletes data source file

2022-09-19 Thread Drew (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Drew reopened SPARK-40286: -- > Load Data from S3 deletes data source file > -- > > Key:

[jira] [Reopened] (SPARK-40287) Load Data using Spark by a single partition moves entire dataset under same location in S3

2022-09-19 Thread Drew (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Drew reopened SPARK-40287: -- > Load Data using Spark by a single partition moves entire dataset under same > location in S3 >

[jira] [Assigned] (SPARK-39991) AQE should use available column statistics from completed query stages

2022-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-39991: - Assignee: Andy Grove > AQE should use available column statistics from completed query

[jira] [Resolved] (SPARK-39991) AQE should use available column statistics from completed query stages

2022-09-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-39991. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37424 [https://

[jira] [Assigned] (SPARK-40477) Support `NullType` in `ColumnarBatchRow`

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40477: Assignee: (was: Apache Spark) > Support `NullType` in `ColumnarBatchRow` > --

[jira] [Commented] (SPARK-40477) Support `NullType` in `ColumnarBatchRow`

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606744#comment-17606744 ] Apache Spark commented on SPARK-40477: -- User 'kazuyukitanimura' has created a pull

[jira] [Commented] (SPARK-40477) Support `NullType` in `ColumnarBatchRow`

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606742#comment-17606742 ] Apache Spark commented on SPARK-40477: -- User 'kazuyukitanimura' has created a pull

[jira] [Assigned] (SPARK-40477) Support `NullType` in `ColumnarBatchRow`

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40477: Assignee: Apache Spark > Support `NullType` in `ColumnarBatchRow` > -

[jira] [Commented] (SPARK-40474) Infer columns with mixed date and timestamp as String in CSV schema inference

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606684#comment-17606684 ] Apache Spark commented on SPARK-40474: -- User 'xiaonanyang-db' has created a pull re

[jira] [Assigned] (SPARK-40474) Infer columns with mixed date and timestamp as String in CSV schema inference

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40474: Assignee: (was: Apache Spark) > Infer columns with mixed date and timestamp as String

[jira] [Commented] (SPARK-40474) Infer columns with mixed date and timestamp as String in CSV schema inference

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606683#comment-17606683 ] Apache Spark commented on SPARK-40474: -- User 'xiaonanyang-db' has created a pull re

[jira] [Assigned] (SPARK-40474) Infer columns with mixed date and timestamp as String in CSV schema inference

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40474: Assignee: Apache Spark > Infer columns with mixed date and timestamp as String in CSV sch

[jira] [Commented] (SPARK-40460) Streaming metrics is zero when select _metadata

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606679#comment-17606679 ] Apache Spark commented on SPARK-40460: -- User 'Yaohua628' has created a pull request

[jira] [Commented] (SPARK-40460) Streaming metrics is zero when select _metadata

2022-09-19 Thread Yaohua Zhao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606678#comment-17606678 ] Yaohua Zhao commented on SPARK-40460: - [~kabhwan] You are right! Updated > Streamin

[jira] [Updated] (SPARK-40460) Streaming metrics is zero when select _metadata

2022-09-19 Thread Yaohua Zhao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yaohua Zhao updated SPARK-40460: Affects Version/s: 3.4.0 > Streaming metrics is zero when select _metadata > -

[jira] [Updated] (SPARK-40460) Streaming metrics is zero when select _metadata

2022-09-19 Thread Yaohua Zhao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yaohua Zhao updated SPARK-40460: Affects Version/s: 3.3.1 3.3.2 (was: 3.2.0)

[jira] [Resolved] (SPARK-40484) Upgrade log4j2 to 2.19.0

2022-09-19 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-40484. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37926 [https://gith

[jira] [Assigned] (SPARK-40484) Upgrade log4j2 to 2.19.0

2022-09-19 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-40484: --- Assignee: Yang Jie > Upgrade log4j2 to 2.19.0 > > >

[jira] [Commented] (SPARK-40489) Spark 3.3.0 breaks with SFL4J 2.

2022-09-19 Thread Piotr Karwasz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606671#comment-17606671 ] Piotr Karwasz commented on SPARK-40489: --- Since {{StaticLoggerBinder}} is not API,

[jira] [Updated] (SPARK-40489) Spark 3.3.0 breaks with SFL4J 2.

2022-09-19 Thread Garret Wilson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Garret Wilson updated SPARK-40489: -- Description: Spark breaks fundamentally with SLF4J 2.x because it uses {{StaticLoggerBinder}}

[jira] [Updated] (SPARK-40489) Spark 3.3.0 breaks with SFL4J 2.

2022-09-19 Thread Garret Wilson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Garret Wilson updated SPARK-40489: -- Summary: Spark 3.3.0 breaks with SFL4J 2. (was: Spark 3.3.0 breaks SFL4J 2.) > Spark 3.3.0 b

[jira] [Updated] (SPARK-40489) Spark 3.3.0 breaks SFL4J 2.

2022-09-19 Thread Garret Wilson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Garret Wilson updated SPARK-40489: -- Description: Spark breaks fundamentally with SLF4J 2.x because it uses {{StaticLoggerBinder}}

[jira] [Updated] (SPARK-40489) Spark 3.3.0 breaks SFL4J 2.

2022-09-19 Thread Garret Wilson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Garret Wilson updated SPARK-40489: -- Description: Spark breaks fundamentally with SLF4J 2.x because it uses {{StaticLoggerBinder}}

[jira] [Updated] (SPARK-40489) Spark 3.3.0 breaks SFL4J 2.

2022-09-19 Thread Garret Wilson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Garret Wilson updated SPARK-40489: -- Description: Spark breaks fundamentally with SLF4J 2.x because it uses {{StaticLoggerBinder}}

[jira] [Created] (SPARK-40489) Spark 3.3.0 breaks SFL4J 2.

2022-09-19 Thread Garret Wilson (Jira)
Garret Wilson created SPARK-40489: - Summary: Spark 3.3.0 breaks SFL4J 2. Key: SPARK-40489 URL: https://issues.apache.org/jira/browse/SPARK-40489 Project: Spark Issue Type: Bug Compo

[jira] [Resolved] (SPARK-40413) Column.isin produces non-boolean results

2022-09-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40413. -- Resolution: Invalid > Column.isin produces non-boolean results > -

[jira] [Resolved] (SPARK-40456) PartitionIterator.hasNext should be cheap to call repeatedly

2022-09-19 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-40456. - Fix Version/s: 3.4.0 Resolution: Fixed > PartitionIterator.hasNext should be cheap to cal

[jira] [Assigned] (SPARK-40456) PartitionIterator.hasNext should be cheap to call repeatedly

2022-09-19 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-40456: --- Assignee: Richard Chen (was: Wenchen Fan) > PartitionIterator.hasNext should be cheap to c

[jira] [Updated] (SPARK-40456) PartitionIterator.hasNext should be cheap to call repeatedly

2022-09-19 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-40456: - Priority: Minor (was: Major) > PartitionIterator.hasNext should be cheap to call repeatedly > -

[jira] [Resolved] (SPARK-40294) Repeat calls to `PartitionIterator.hasNext` can timeout

2022-09-19 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-40294. -- Resolution: Duplicate > Repeat calls to `PartitionIterator.hasNext` can timeout >

[jira] [Commented] (SPARK-40488) Do not wrap exceptions thrown in FileFormatWriter.write with SparkException

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606544#comment-17606544 ] Apache Spark commented on SPARK-40488: -- User 'bozhang2820' has created a pull reque

[jira] [Assigned] (SPARK-40488) Do not wrap exceptions thrown in FileFormatWriter.write with SparkException

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40488: Assignee: Apache Spark > Do not wrap exceptions thrown in FileFormatWriter.write with Spa

[jira] [Assigned] (SPARK-40488) Do not wrap exceptions thrown in FileFormatWriter.write with SparkException

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40488: Assignee: (was: Apache Spark) > Do not wrap exceptions thrown in FileFormatWriter.wri

[jira] [Resolved] (SPARK-40419) Integrate Grouped Aggregate Pandas UDFs into *.sql test cases

2022-09-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40419. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37873 [https://gi

[jira] [Assigned] (SPARK-40419) Integrate Grouped Aggregate Pandas UDFs into *.sql test cases

2022-09-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-40419: Assignee: Haejoon Lee > Integrate Grouped Aggregate Pandas UDFs into *.sql test cases > -

[jira] [Created] (SPARK-40488) Do not wrap exceptions thrown in FileFormatWriter.write with SparkException

2022-09-19 Thread Bo Zhang (Jira)
Bo Zhang created SPARK-40488: Summary: Do not wrap exceptions thrown in FileFormatWriter.write with SparkException Key: SPARK-40488 URL: https://issues.apache.org/jira/browse/SPARK-40488 Project: Spark

[jira] [Commented] (SPARK-40487) Make defaultJoin in BroadcastNestedLoopJoinExec running in parallel

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606537#comment-17606537 ] Apache Spark commented on SPARK-40487: -- User 'xingczhao' has created a pull request

[jira] [Assigned] (SPARK-40487) Make defaultJoin in BroadcastNestedLoopJoinExec running in parallel

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40487: Assignee: Apache Spark > Make defaultJoin in BroadcastNestedLoopJoinExec running in paral

[jira] [Commented] (SPARK-40487) Make defaultJoin in BroadcastNestedLoopJoinExec running in parallel

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606536#comment-17606536 ] Apache Spark commented on SPARK-40487: -- User 'xingczhao' has created a pull request

[jira] [Assigned] (SPARK-40487) Make defaultJoin in BroadcastNestedLoopJoinExec running in parallel

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40487: Assignee: (was: Apache Spark) > Make defaultJoin in BroadcastNestedLoopJoinExec runni

[jira] [Created] (SPARK-40487) Make defaultJoin in BroadcastNestedLoopJoinExec running in parallel

2022-09-19 Thread Xingchao, Zhang (Jira)
Xingchao, Zhang created SPARK-40487: --- Summary: Make defaultJoin in BroadcastNestedLoopJoinExec running in parallel Key: SPARK-40487 URL: https://issues.apache.org/jira/browse/SPARK-40487 Project: Sp

[jira] [Commented] (SPARK-40486) Implement `spearman` and `kendall` in `DataFrame.corrwith`

2022-09-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606528#comment-17606528 ] Apache Spark commented on SPARK-40486: -- User 'zhengruifeng' has created a pull requ

  1   2   >