[jira] [Resolved] (SPARK-36643) Add more information in ERROR log while SparkConf is modified when spark.sql.legacy.setCommandRejectsSparkCoreConfs is set

2021-09-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-36643. --- Fix Version/s: 3.3.0 Assignee: Senthil Kumar Resolution: Fixed This is resol

[jira] [Assigned] (SPARK-36666) [SQL] Regression in AQEShuffleReadExec

2021-09-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-3: - Assignee: Andy Grove > [SQL] Regression in AQEShuffleReadExec > ---

[jira] [Updated] (SPARK-36666) [SQL] Regression in AQEShuffleReadExec

2021-09-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-3: -- Parent: SPARK-33828 Issue Type: Sub-task (was: Bug) > [SQL] Regression in AQEShuffleR

[jira] [Assigned] (SPARK-36653) Implement Series.__xor__

2021-09-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36653: Assignee: (was: Apache Spark) > Implement Series.__xor__ > >

[jira] [Commented] (SPARK-36653) Implement Series.__xor__

2021-09-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17409857#comment-17409857 ] Apache Spark commented on SPARK-36653: -- User 'dgd-contributor' has created a pull r

[jira] [Commented] (SPARK-36653) Implement Series.__xor__

2021-09-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17409858#comment-17409858 ] Apache Spark commented on SPARK-36653: -- User 'dgd-contributor' has created a pull r

[jira] [Assigned] (SPARK-36653) Implement Series.__xor__

2021-09-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36653: Assignee: Apache Spark > Implement Series.__xor__ > > >

[jira] [Commented] (SPARK-36667) Close resources properly in StateStoreSuite/RocksDBStateStoreSuite

2021-09-03 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17409779#comment-17409779 ] Jungtaek Lim commented on SPARK-36667: -- Will submit a PR soon. > Close resources p

[jira] [Created] (SPARK-36667) Close resources properly in StateStoreSuite/RocksDBStateStoreSuite

2021-09-03 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-36667: Summary: Close resources properly in StateStoreSuite/RocksDBStateStoreSuite Key: SPARK-36667 URL: https://issues.apache.org/jira/browse/SPARK-36667 Project: Spark

[jira] [Updated] (SPARK-36666) [SQL] Regression in AQEShuffleReadExec

2021-09-03 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-3: -- Priority: Blocker (was: Major) > [SQL] Regression in AQEShuffleReadExec > ---

[jira] [Assigned] (SPARK-36666) [SQL] Regression in AQEShuffleReadExec

2021-09-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-3: Assignee: Apache Spark > [SQL] Regression in AQEShuffleReadExec > ---

[jira] [Commented] (SPARK-36666) [SQL] Regression in AQEShuffleReadExec

2021-09-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17409761#comment-17409761 ] Apache Spark commented on SPARK-3: -- User 'andygrove' has created a pull request

[jira] [Assigned] (SPARK-36666) [SQL] Regression in AQEShuffleReadExec

2021-09-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-3: Assignee: (was: Apache Spark) > [SQL] Regression in AQEShuffleReadExec >

[jira] [Created] (SPARK-36666) [SQL] Regression in AQEShuffleReadExec

2021-09-03 Thread Andy Grove (Jira)
Andy Grove created SPARK-3: -- Summary: [SQL] Regression in AQEShuffleReadExec Key: SPARK-3 URL: https://issues.apache.org/jira/browse/SPARK-3 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-36666) [SQL] Regression in AQEShuffleReadExec

2021-09-03 Thread Andy Grove (Jira)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy Grove updated SPARK-3: --- Description: I am currently testing the RAPIDS Accelerator for Apache Spark with the Spark 3.2 rele

[jira] [Commented] (SPARK-36665) Add more Not operator optimizations

2021-09-03 Thread Kazuyuki Tanimura (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17409746#comment-17409746 ] Kazuyuki Tanimura commented on SPARK-36665: --- I am working on this > Add more

[jira] [Created] (SPARK-36665) Add more Not operator optimizations

2021-09-03 Thread Kazuyuki Tanimura (Jira)
Kazuyuki Tanimura created SPARK-36665: - Summary: Add more Not operator optimizations Key: SPARK-36665 URL: https://issues.apache.org/jira/browse/SPARK-36665 Project: Spark Issue Type: Imp

[jira] [Resolved] (SPARK-36655) Add `versionadded` for API added in Spark 3.3.0

2021-09-03 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-36655. --- Fix Version/s: 3.3.0 Assignee: Xinrong Meng Resolution: Fixed Issue resolved

[jira] [Resolved] (SPARK-36401) Implement Series.cov

2021-09-03 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-36401. --- Fix Version/s: 3.3.0 Assignee: dgd_contributor Resolution: Fixed Issue resol

[jira] [Commented] (SPARK-36659) Promote spark.sql.execution.topKSortFallbackThreshold to user-faced config

2021-09-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17409634#comment-17409634 ] Dongjoon Hyun commented on SPARK-36659: --- Although RC2 will fail, I set the fixed v

[jira] [Updated] (SPARK-36659) Promote spark.sql.execution.topKSortFallbackThreshold to user-faced config

2021-09-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36659: -- Fix Version/s: (was: 3.3.0) 3.2.1 > Promote spark.sql.execution.topKSor

[jira] [Updated] (SPARK-36659) Promote spark.sql.execution.topKSortFallbackThreshold to user-faced config

2021-09-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36659: -- Fix Version/s: (was: 3.2.0) 3.3.0 > Promote spark.sql.execution.topKSor

[jira] [Updated] (SPARK-36659) Promote spark.sql.execution.topKSortFallbackThreshold to user-faced config

2021-09-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36659: -- Fix Version/s: (was: 3.3.0) 3.2.0 > Promote spark.sql.execution.topKSor

[jira] [Commented] (SPARK-36639) SQL sequence function with interval returns unexpected error in latest versions

2021-09-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17409620#comment-17409620 ] Apache Spark commented on SPARK-36639: -- User 'sarutak' has created a pull request f

[jira] [Commented] (SPARK-36639) SQL sequence function with interval returns unexpected error in latest versions

2021-09-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17409619#comment-17409619 ] Apache Spark commented on SPARK-36639: -- User 'sarutak' has created a pull request f

[jira] [Created] (SPARK-36664) Log time spent waiting for cluster resources

2021-09-03 Thread Holden Karau (Jira)
Holden Karau created SPARK-36664: Summary: Log time spent waiting for cluster resources Key: SPARK-36664 URL: https://issues.apache.org/jira/browse/SPARK-36664 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-36622) spark.history.kerberos.principal doesn't take value _HOST

2021-09-03 Thread pralabhkumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17409617#comment-17409617 ] pralabhkumar commented on SPARK-36622: -- [~thejdeep]  Its better to have _HOST , it

[jira] [Commented] (SPARK-36639) SQL sequence function with interval returns unexpected error in latest versions

2021-09-03 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17409539#comment-17409539 ] Kousuke Saruta commented on SPARK-36639: Issue resolved in https://github.com/ap

[jira] [Resolved] (SPARK-36639) SQL sequence function with interval returns unexpected error in latest versions

2021-09-03 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-36639. Assignee: Kousuke Saruta Resolution: Fixed > SQL sequence function with interval ret

[jira] [Updated] (SPARK-36639) SQL sequence function with interval returns unexpected error in latest versions

2021-09-03 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-36639: --- Fix Version/s: 3.1.3 3.2.0 > SQL sequence function with interval returns

[jira] [Commented] (SPARK-36663) When the existing field name is a number, an error will be reported when reading the orc file

2021-09-03 Thread mcdull_zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17409492#comment-17409492 ] mcdull_zhang commented on SPARK-36663: -- cc  [~hyukjin.kwon]      [~cloud_fan] > Wh

[jira] [Updated] (SPARK-36663) When the existing field name is a number, an error will be reported when reading the orc file

2021-09-03 Thread mcdull_zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mcdull_zhang updated SPARK-36663: - Description: You can use the following methods to reproduce the problem: {quote}val path = "file

[jira] [Updated] (SPARK-36663) When the existing field name is a number, an error will be reported when reading the orc file

2021-09-03 Thread mcdull_zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mcdull_zhang updated SPARK-36663: - Attachment: image-2021-09-03-20-56-28-846.png > When the existing field name is a number, an err

[jira] [Created] (SPARK-36663) When the existing field name is a number, an error will be reported when reading the orc file

2021-09-03 Thread mcdull_zhang (Jira)
mcdull_zhang created SPARK-36663: Summary: When the existing field name is a number, an error will be reported when reading the orc file Key: SPARK-36663 URL: https://issues.apache.org/jira/browse/SPARK-36663

[jira] [Resolved] (SPARK-36609) Add `errors` argument for `ps.to_numeric`.

2021-09-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36609. -- Fix Version/s: 3.3.0 Assignee: Haejoon Lee Resolution: Fixed Fixed in https://

[jira] [Resolved] (SPARK-36659) Promote spark.sql.execution.topKSortFallbackThreshold to user-faced config

2021-09-03 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-36659. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 33904 [https://github.com

[jira] [Assigned] (SPARK-36659) Promote spark.sql.execution.topKSortFallbackThreshold to user-faced config

2021-09-03 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-36659: Assignee: Kent Yao > Promote spark.sql.execution.topKSortFallbackThreshold to user-faced config >

[jira] [Assigned] (SPARK-36661) Support TimestampNTZ in Py4J

2021-09-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36661: Assignee: (was: Apache Spark) > Support TimestampNTZ in Py4J > --

[jira] [Commented] (SPARK-36661) Support TimestampNTZ in Py4J

2021-09-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17409447#comment-17409447 ] Apache Spark commented on SPARK-36661: -- User 'HyukjinKwon' has created a pull reque

[jira] [Assigned] (SPARK-36661) Support TimestampNTZ in Py4J

2021-09-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36661: Assignee: Apache Spark > Support TimestampNTZ in Py4J > > >

[jira] [Commented] (SPARK-26208) Empty dataframe does not roundtrip for csv with header

2021-09-03 Thread Ranga Reddy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17409427#comment-17409427 ] Ranga Reddy commented on SPARK-26208: - cc [~hyukjin.kwon] > Empty dataframe does no

[jira] [Assigned] (SPARK-36662) special timestamps values support for path filters - modifiedBefore/modifiedAfter

2021-09-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36662: Assignee: Apache Spark > special timestamps values support for path filters - > modified

[jira] [Commented] (SPARK-36662) special timestamps values support for path filters - modifiedBefore/modifiedAfter

2021-09-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17409356#comment-17409356 ] Apache Spark commented on SPARK-36662: -- User 'yaooqinn' has created a pull request

[jira] [Assigned] (SPARK-36662) special timestamps values support for path filters - modifiedBefore/modifiedAfter

2021-09-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36662: Assignee: (was: Apache Spark) > special timestamps values support for path filters -

[jira] [Resolved] (SPARK-36644) Push down boolean column filter

2021-09-03 Thread DB Tsai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-36644. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 33898 [https://github.com/a

[jira] [Assigned] (SPARK-36644) Push down boolean column filter

2021-09-03 Thread DB Tsai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-36644: --- Assignee: Kazuyuki Tanimura > Push down boolean column filter > --- > >

[jira] [Created] (SPARK-36662) special timestamps values support for path filters - modifiedBefore/modifiedAfter

2021-09-03 Thread Kent Yao (Jira)
Kent Yao created SPARK-36662: Summary: special timestamps values support for path filters - modifiedBefore/modifiedAfter Key: SPARK-36662 URL: https://issues.apache.org/jira/browse/SPARK-36662 Project: Sp

[jira] [Assigned] (SPARK-36610) Add `thousands` argument to `ps.read_csv`.

2021-09-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36610: Assignee: Apache Spark > Add `thousands` argument to `ps.read_csv`. > ---

[jira] [Assigned] (SPARK-36610) Add `thousands` argument to `ps.read_csv`.

2021-09-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36610: Assignee: (was: Apache Spark) > Add `thousands` argument to `ps.read_csv`. >

[jira] [Commented] (SPARK-36610) Add `thousands` argument to `ps.read_csv`.

2021-09-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17409328#comment-17409328 ] Apache Spark commented on SPARK-36610: -- User 'itholic' has created a pull request f