[jira] [Assigned] (SPARK-48514) Upgrade kubernetes-client to 6.13.0 for K8s v1.30.0

2024-06-03 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-48514: Assignee: Bjørn Jørgensen > Upgrade kubernetes-client to 6.13.0 for K8s v1.30.0 > ---

[jira] [Resolved] (SPARK-48514) Upgrade kubernetes-client to 6.13.0 for K8s v1.30.0

2024-06-03 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-48514. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46854 [https://github.com

[jira] [Resolved] (SPARK-48517) PythonWorkerFactory does not print error stream in case the daemon fails before the main daemon.py#main()

2024-06-03 Thread Prabhu Joseph (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph resolved SPARK-48517. --- Resolution: Invalid Driver logs have correctly captured the error stream. Apologies for the

[jira] [Updated] (SPARK-48517) PythonWorkerFactory does not print error stream in case the daemon fails before the main daemon.py#main()

2024-06-03 Thread Prabhu Joseph (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated SPARK-48517: -- Description: PythonWorkerFactory does not print the error stream in case the daemon fails bef

[jira] [Updated] (SPARK-48517) PythonWorkerFactory does not print error stream in case the daemon fails before the main daemon.py#main()

2024-06-03 Thread Prabhu Joseph (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated SPARK-48517: -- Summary: PythonWorkerFactory does not print error stream in case the daemon fails before the m

[jira] [Created] (SPARK-48517) PythonWorkerFactory does not print error stream in case the daemon fails before the main daemon.py#method

2024-06-03 Thread Prabhu Joseph (Jira)
Prabhu Joseph created SPARK-48517: - Summary: PythonWorkerFactory does not print error stream in case the daemon fails before the main daemon.py#method Key: SPARK-48517 URL: https://issues.apache.org/jira/browse/SP

[jira] [Assigned] (SPARK-48482) dropDuplicates and dropDuplicatesWithinWatermark should accept varargs

2024-06-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48482: Assignee: Wei Liu > dropDuplicates and dropDuplicatesWithinWatermark should accept vararg

[jira] [Resolved] (SPARK-48482) dropDuplicates and dropDuplicatesWithinWatermark should accept varargs

2024-06-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48482. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46817 [https://gi

[jira] [Assigned] (SPARK-48508) Client Side RPC optimization for Spark Connect

2024-06-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48508: Assignee: Ruifeng Zheng > Client Side RPC optimization for Spark Connect > --

[jira] [Resolved] (SPARK-48508) Client Side RPC optimization for Spark Connect

2024-06-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48508. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46848 [https://gi

[jira] [Assigned] (SPARK-47972) Restrict CAST expression for collations

2024-06-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-47972: --- Assignee: Mihailo Milosevic > Restrict CAST expression for collations > ---

[jira] [Resolved] (SPARK-47972) Restrict CAST expression for collations

2024-06-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-47972. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46474 [https://gith

[jira] [Updated] (SPARK-48515) Enable Arrow optimization for Python UDFs

2024-06-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48515: --- Labels: pull-request-available (was: ) > Enable Arrow optimization for Python UDFs > --

[jira] [Created] (SPARK-48516) Turn on Arrow optimization for Python UDFs by default

2024-06-03 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-48516: Summary: Turn on Arrow optimization for Python UDFs by default Key: SPARK-48516 URL: https://issues.apache.org/jira/browse/SPARK-48516 Project: Spark Issue T

[jira] [Updated] (SPARK-48513) Use NERF framework for state schema compatibility exception

2024-06-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48513: --- Labels: pull-request-available (was: ) > Use NERF framework for state schema compatibility

[jira] [Created] (SPARK-48515) Enable Arrow optimization for Python UDFs

2024-06-03 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-48515: Summary: Enable Arrow optimization for Python UDFs Key: SPARK-48515 URL: https://issues.apache.org/jira/browse/SPARK-48515 Project: Spark Issue Type: Umbrell

[jira] [Updated] (SPARK-48514) Upgrade kubernetes-client to 6.13.0 for K8s v1.30.0

2024-06-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48514: --- Labels: pull-request-available (was: ) > Upgrade kubernetes-client to 6.13.0 for K8s v1.30.

[jira] [Updated] (SPARK-42944) Support Python foreachBatch() in streaming spark connect

2024-06-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-42944: --- Labels: pull-request-available (was: ) > Support Python foreachBatch() in streaming spark c

[jira] [Created] (SPARK-48514) Upgrade kubernetes-client to 6.13.0 for K8s v1.30.0

2024-06-03 Thread Jira
Bjørn Jørgensen created SPARK-48514: --- Summary: Upgrade kubernetes-client to 6.13.0 for K8s v1.30.0 Key: SPARK-48514 URL: https://issues.apache.org/jira/browse/SPARK-48514 Project: Spark Iss

[jira] [Resolved] (SPARK-48413) ALTER COLUMN with collation

2024-06-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48413. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46734 [https://gith

[jira] [Commented] (SPARK-48493) Enhance Python Datasource Reader with Arrow Batch Support for Improved Performance

2024-06-03 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17851794#comment-17851794 ] Luca Canali commented on SPARK-48493: - This work appears related https://issues.apac

[jira] [Assigned] (SPARK-47977) DateTimeUtils.timestampDiff and DateTimeUtils.timestampAdd should not throw INTERNAL_ERROR exception

2024-06-03 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-47977: -- Assignee: Vitalii Li > DateTimeUtils.timestampDiff and DateTimeUtils.timestampAdd sho

[jira] [Resolved] (SPARK-47977) DateTimeUtils.timestampDiff and DateTimeUtils.timestampAdd should not throw INTERNAL_ERROR exception

2024-06-03 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47977. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46210 [https:

[jira] [Updated] (SPARK-48512) Refactor Python tests

2024-06-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48512: --- Labels: pull-request-available (was: ) > Refactor Python tests > - > >

[jira] [Created] (SPARK-48513) Use NERF framework for state schema compatibility exception

2024-06-03 Thread Anish Shrigondekar (Jira)
Anish Shrigondekar created SPARK-48513: -- Summary: Use NERF framework for state schema compatibility exception Key: SPARK-48513 URL: https://issues.apache.org/jira/browse/SPARK-48513 Project: Spar

[jira] [Created] (SPARK-48512) Refactor Python tests

2024-06-03 Thread Rui Wang (Jira)
Rui Wang created SPARK-48512: Summary: Refactor Python tests Key: SPARK-48512 URL: https://issues.apache.org/jira/browse/SPARK-48512 Project: Spark Issue Type: Sub-task Components: PySp

[jira] [Assigned] (SPARK-48503) Scalar subquery with group-by and non-equality predicate incorrectly allowed, wrong results

2024-06-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48503: --- Assignee: Jack Chen > Scalar subquery with group-by and non-equality predicate incorrectly

[jira] [Resolved] (SPARK-48503) Scalar subquery with group-by and non-equality predicate incorrectly allowed, wrong results

2024-06-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48503. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46839 [https://gith

[jira] [Updated] (SPARK-38862) Basic Authentication or Token Based Authentication for The REST Submission Server

2024-06-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-38862: --- Labels: authentication pull-request-available rest spark spark-submit submit (was: authenti

[jira] [Updated] (SPARK-48511) [Arbitrary State Support] Remove TimeMode None

2024-06-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48511: --- Labels: pull-request-available (was: ) > [Arbitrary State Support] Remove TimeMode None > -

[jira] [Created] (SPARK-48511) [Arbitrary State Support] Remove TimeMode None

2024-06-03 Thread Bhuwan Sahni (Jira)
Bhuwan Sahni created SPARK-48511: Summary: [Arbitrary State Support] Remove TimeMode None Key: SPARK-48511 URL: https://issues.apache.org/jira/browse/SPARK-48511 Project: Spark Issue Type: Im

[jira] [Updated] (SPARK-48492) batch-read parquet files written by streaming returns non-nullable fields in schema

2024-06-03 Thread Julien Peloton (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Peloton updated SPARK-48492: --- Issue Type: Bug (was: New Feature) > batch-read parquet files written by streaming returns

[jira] [Updated] (SPARK-48510) Support UDAF.toColumn API in Spark Connect

2024-06-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48510: --- Labels: pull-request-available (was: ) > Support UDAF.toColumn API in Spark Connect > -

[jira] [Created] (SPARK-48510) Support UDAF.toColumn API in Spark Connect

2024-06-03 Thread Pengfei Xu (Jira)
Pengfei Xu created SPARK-48510: -- Summary: Support UDAF.toColumn API in Spark Connect Key: SPARK-48510 URL: https://issues.apache.org/jira/browse/SPARK-48510 Project: Spark Issue Type: New Featur

[jira] [Updated] (SPARK-48508) Client Side RPC optimization for Spark Connect

2024-06-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48508: --- Labels: pull-request-available (was: ) > Client Side RPC optimization for Spark Connect > -

[jira] [Created] (SPARK-48509) Cache user specified schema in DataFrame.{to, mapInPandas, mapInArrow}

2024-06-03 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-48509: - Summary: Cache user specified schema in DataFrame.{to, mapInPandas, mapInArrow} Key: SPARK-48509 URL: https://issues.apache.org/jira/browse/SPARK-48509 Project: Spa

[jira] [Created] (SPARK-48508) Client Side RPC optimization for Spark Connect

2024-06-03 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-48508: - Summary: Client Side RPC optimization for Spark Connect Key: SPARK-48508 URL: https://issues.apache.org/jira/browse/SPARK-48508 Project: Spark Issue Type:

[jira] [Updated] (SPARK-48508) Client Side RPC optimization for Spark Connect

2024-06-03 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-48508: -- Issue Type: Umbrella (was: Improvement) > Client Side RPC optimization for Spark Connect > --

[jira] [Assigned] (SPARK-47690) Hash aggregate support for strings with collation

2024-06-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47690: -- Assignee: Apache Spark > Hash aggregate support for strings with collation >

[jira] [Assigned] (SPARK-47690) Hash aggregate support for strings with collation

2024-06-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47690: -- Assignee: (was: Apache Spark) > Hash aggregate support for strings with collation

[jira] [Assigned] (SPARK-47690) Hash aggregate support for strings with collation

2024-06-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47690: -- Assignee: (was: Apache Spark) > Hash aggregate support for strings with collation

[jira] [Assigned] (SPARK-47690) Hash aggregate support for strings with collation

2024-06-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47690: -- Assignee: Apache Spark > Hash aggregate support for strings with collation >

[jira] [Assigned] (SPARK-47690) Hash aggregate support for strings with collation

2024-06-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47690: -- Assignee: Apache Spark > Hash aggregate support for strings with collation >

[jira] [Assigned] (SPARK-47690) Hash aggregate support for strings with collation

2024-06-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47690: -- Assignee: (was: Apache Spark) > Hash aggregate support for strings with collation

[jira] [Resolved] (SPARK-48507) Use Hadoop 3.3.6 winutils in `build_sparkr_window`

2024-06-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48507. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46846 [https://gi

[jira] [Assigned] (SPARK-48507) Use Hadoop 3.3.6 winutils in `build_sparkr_window`

2024-06-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48507: Assignee: BingKun Pan > Use Hadoop 3.3.6 winutils in `build_sparkr_window` >

[jira] [Assigned] (SPARK-48504) Parent Window class for Spark Connect and Spark Classic

2024-06-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48504: Assignee: Ruifeng Zheng > Parent Window class for Spark Connect and Spark Classic > -

[jira] [Resolved] (SPARK-48504) Parent Window class for Spark Connect and Spark Classic

2024-06-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48504. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46841 [https://gi

[jira] [Updated] (SPARK-48506) Compression codec short names are case insensitive expect for event logging

2024-06-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48506: --- Labels: pull-request-available (was: ) > Compression codec short names are case insensitive

[jira] [Updated] (SPARK-48507) Use Hadoop 3.3.6 winutils in `build_sparkr_window`

2024-06-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48507: --- Labels: pull-request-available (was: ) > Use Hadoop 3.3.6 winutils in `build_sparkr_window`

[jira] [Created] (SPARK-48507) Use Hadoop 3.3.6 winutils in `build_sparkr_window`

2024-06-03 Thread BingKun Pan (Jira)
BingKun Pan created SPARK-48507: --- Summary: Use Hadoop 3.3.6 winutils in `build_sparkr_window` Key: SPARK-48507 URL: https://issues.apache.org/jira/browse/SPARK-48507 Project: Spark Issue Type:

[jira] [Created] (SPARK-48506) Compression codec short names are case insensitive expect for event logging

2024-06-03 Thread Kent Yao (Jira)
Kent Yao created SPARK-48506: Summary: Compression codec short names are case insensitive expect for event logging Key: SPARK-48506 URL: https://issues.apache.org/jira/browse/SPARK-48506 Project: Spark