[jira] [Updated] (SPARK-44564) Refine the documents with LLM

[ https://issues.apache.org/jira/browse/SPARK-44564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-44564: -- Description: Let's first focus on the Documents of *PySpark DataFrame APIs*. *1*, Chose a

[jira] [Updated] (SPARK-44564) Refine the documents with LLM

[ https://issues.apache.org/jira/browse/SPARK-44564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-44564: -- Description: Let's first focus on the Documents of *PySpark DataFrame APIs*. *1*, Chose a

[jira] [Updated] (SPARK-44564) Refine the documents with LLM

[ https://issues.apache.org/jira/browse/SPARK-44564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-44564: -- Description: Let's first focus on the Documents of *PySpark DataFrame APIs*. *1*, Chose a

[jira] [Updated] (SPARK-44564) Refine the documents with LLM

[ https://issues.apache.org/jira/browse/SPARK-44564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-44564: -- Description: Let's first focus on the Documents of *PySpark DataFrame APIs*. *1*, Chose a

[jira] [Updated] (SPARK-44564) Refine the documents with LLM

[ https://issues.apache.org/jira/browse/SPARK-44564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-44564: -- Description: Let's first focus on the Documents of *PySpark DataFrame APIs*. *1*, Chose a

[jira] [Resolved] (SPARK-44557) Flaky PIP packaging test

[ https://issues.apache.org/jira/browse/SPARK-44557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44557. -- Fix Version/s: 3.5.0 4.0.0 3.4.2 Resolution:

[jira] [Assigned] (SPARK-44557) Flaky PIP packaging test

[ https://issues.apache.org/jira/browse/SPARK-44557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-44557: Assignee: Hyukjin Kwon > Flaky PIP packaging test > > >

[jira] [Updated] (SPARK-44564) Refine the documents with LLM

[ https://issues.apache.org/jira/browse/SPARK-44564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-44564: -- Description: Let's first focus on the Documents of *PySpark DataFrame APIs*. 1, Chose a

[jira] [Updated] (SPARK-44565) Example: Refine the docs for Union, UnionAll and unionByName

[ https://issues.apache.org/jira/browse/SPARK-44565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-44565: -- Summary: Example: Refine the docs for Union, UnionAll and unionByName (was: Refine the docs

[jira] [Created] (SPARK-44565) Refine the docs for Union, UnionAll and unionByName

Ruifeng Zheng created SPARK-44565: - Summary: Refine the docs for Union, UnionAll and unionByName Key: SPARK-44565 URL: https://issues.apache.org/jira/browse/SPARK-44565 Project: Spark Issue

[jira] [Updated] (SPARK-44564) Refine the documents with LLM

[ https://issues.apache.org/jira/browse/SPARK-44564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-44564: -- Description: Let's first focus on the Documents of PySpark DataFrame APIs. 1, Chose a subset

[jira] [Updated] (SPARK-44564) Refine the documents with LLM

[ https://issues.apache.org/jira/browse/SPARK-44564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-44564: -- Description: Let's first focus on the Documents of PySpark DataFrame APIs. 1, Chose a subset

[jira] [Created] (SPARK-44564) Refine the documents with LLM

Ruifeng Zheng created SPARK-44564: - Summary: Refine the documents with LLM Key: SPARK-44564 URL: https://issues.apache.org/jira/browse/SPARK-44564 Project: Spark Issue Type: Umbrella

[jira] [Resolved] (SPARK-44533) Add support for accumulator, broadcast, and Spark files in Python UDTF's analyze.

[ https://issues.apache.org/jira/browse/SPARK-44533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-44533. --- Assignee: Takuya Ueshin Resolution: Fixed Issue resolved by pull request 42135

[jira] [Created] (SPARK-44563) Upgrade Apache Arrow to 13.0.0

BingKun Pan created SPARK-44563: --- Summary: Upgrade Apache Arrow to 13.0.0 Key: SPARK-44563 URL: https://issues.apache.org/jira/browse/SPARK-44563 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-43611) Fix unexpected `AnalysisException` from Spark Connect client

[ https://issues.apache.org/jira/browse/SPARK-43611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-43611. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42086

[jira] [Assigned] (SPARK-43611) Fix unexpected `AnalysisException` from Spark Connect client

[ https://issues.apache.org/jira/browse/SPARK-43611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-43611: - Assignee: Ruifeng Zheng > Fix unexpected `AnalysisException` from Spark Connect client

[jira] [Created] (SPARK-44562) Add OptimizeOneRowRelationSubquery in batch of Subquery

Yuming Wang created SPARK-44562: --- Summary: Add OptimizeOneRowRelationSubquery in batch of Subquery Key: SPARK-44562 URL: https://issues.apache.org/jira/browse/SPARK-44562 Project: Spark Issue

[jira] [Resolved] (SPARK-44479) Support Python UDTFs with empty schema

[ https://issues.apache.org/jira/browse/SPARK-44479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-44479. --- Assignee: Takuya Ueshin Resolution: Fixed Issue resolved by pull request 42161

[jira] [Resolved] (SPARK-44553) Ignoring `connect-check-protos` logic in GA testing

[ https://issues.apache.org/jira/browse/SPARK-44553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-44553. --- Fix Version/s: 3.4.2 Resolution: Fixed Issue resolved by pull request 42166

[jira] [Assigned] (SPARK-44553) Ignoring `connect-check-protos` logic in GA testing

[ https://issues.apache.org/jira/browse/SPARK-44553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-44553: - Assignee: BingKun Pan > Ignoring `connect-check-protos` logic in GA testing >

[jira] [Updated] (SPARK-44544) Deduplicate run_python_packaging_tests

[ https://issues.apache.org/jira/browse/SPARK-44544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44544: - Fix Version/s: 3.4.2 > Deduplicate run_python_packaging_tests >

[jira] [Updated] (SPARK-44457) Make ArrowEncoderSuite pass Java 17 daily test

[ https://issues.apache.org/jira/browse/SPARK-44457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-44457: - Priority: Minor (was: Major) > Make ArrowEncoderSuite pass Java 17 daily test >

[jira] [Resolved] (SPARK-44457) Make ArrowEncoderSuite pass Java 17 daily test

[ https://issues.apache.org/jira/browse/SPARK-44457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-44457. -- Fix Version/s: 3.5.0 4.0.0 Assignee: Yang Jie Resolution:

[jira] [Resolved] (SPARK-44522) Upgrade scala-xml to 2.2.0

[ https://issues.apache.org/jira/browse/SPARK-44522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-44522. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42119

[jira] [Assigned] (SPARK-44522) Upgrade scala-xml to 2.2.0

[ https://issues.apache.org/jira/browse/SPARK-44522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-44522: Assignee: Yang Jie > Upgrade scala-xml to 2.2.0 > -- > >

[jira] [Resolved] (SPARK-44528) Spark Connect DataFrame does not allow to add custom instance attributes and check for it

[ https://issues.apache.org/jira/browse/SPARK-44528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44528. -- Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-44528) Spark Connect DataFrame does not allow to add custom instance attributes and check for it

[ https://issues.apache.org/jira/browse/SPARK-44528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-44528: Assignee: Martin Grund > Spark Connect DataFrame does not allow to add custom instance

[jira] [Created] (SPARK-44561) Fix AssertionError when converting UDTF output to a complex type

Allison Wang created SPARK-44561: Summary: Fix AssertionError when converting UDTF output to a complex type Key: SPARK-44561 URL: https://issues.apache.org/jira/browse/SPARK-44561 Project: Spark

[jira] [Created] (SPARK-44560) Improve tests and documentation for Arrow Python UDF

Xinrong Meng created SPARK-44560: Summary: Improve tests and documentation for Arrow Python UDF Key: SPARK-44560 URL: https://issues.apache.org/jira/browse/SPARK-44560 Project: Spark Issue

[jira] [Commented] (SPARK-37562) Add Spark History Server Links for Kubernetes & other CMs

[ https://issues.apache.org/jira/browse/SPARK-37562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17747676#comment-17747676 ] Holden Karau commented on SPARK-37562: -- So (in theory) the cluster administrator has some base

[jira] [Created] (SPARK-44559) Improve error messages for invalid Python UDTF arrow type casts

Allison Wang created SPARK-44559: Summary: Improve error messages for invalid Python UDTF arrow type casts Key: SPARK-44559 URL: https://issues.apache.org/jira/browse/SPARK-44559 Project: Spark

[jira] [Created] (SPARK-44558) Export Pyspark's Spark Connect Log Level

Alice Sayutina created SPARK-44558: -- Summary: Export Pyspark's Spark Connect Log Level Key: SPARK-44558 URL: https://issues.apache.org/jira/browse/SPARK-44558 Project: Spark Issue Type:

[jira] [Commented] (SPARK-44264) DeepSpeed Distrobutor

[ https://issues.apache.org/jira/browse/SPARK-44264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17747612#comment-17747612 ] Ignite TC Bot commented on SPARK-44264: --- User 'mathewjacob1002' has created a pull request for

[jira] [Reopened] (SPARK-44503) Support PARTITION BY and ORDER BY clause for table arguments

[ https://issues.apache.org/jira/browse/SPARK-44503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel reopened SPARK-44503: Reopening since I added the SQL grammar support only in [https://github.com/apache/spark/pull/42100,] and

[jira] [Resolved] (SPARK-44537) Upgrade kubernetes-client to 6.8.0

[ https://issues.apache.org/jira/browse/SPARK-44537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44537. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42142

[jira] [Assigned] (SPARK-44537) Upgrade kubernetes-client to 6.8.0

[ https://issues.apache.org/jira/browse/SPARK-44537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-44537: - Assignee: BingKun Pan > Upgrade kubernetes-client to 6.8.0 >

[jira] [Commented] (SPARK-44557) Flaky PIP packaging test

[ https://issues.apache.org/jira/browse/SPARK-44557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17747471#comment-17747471 ] Nikita Awasthi commented on SPARK-44557: User 'HyukjinKwon' has created a pull request for this

[jira] [Updated] (SPARK-44524) Balancing pyspark-pandas-connect and pyspark-pandas-slow-connect GA testing time

[ https://issues.apache.org/jira/browse/SPARK-44524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-44524: Summary: Balancing pyspark-pandas-connect and pyspark-pandas-slow-connect GA testing time (was:

[jira] [Created] (SPARK-44557) Flaky PIP packaging test

Hyukjin Kwon created SPARK-44557: Summary: Flaky PIP packaging test Key: SPARK-44557 URL: https://issues.apache.org/jira/browse/SPARK-44557 Project: Spark Issue Type: Task

[jira] [Resolved] (SPARK-44531) Move encoder inference to sql/api

[ https://issues.apache.org/jira/browse/SPARK-44531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-44531. --- Fix Version/s: 3.5.0 Resolution: Fixed > Move encoder inference to sql/api >

[jira] [Updated] (SPARK-44555) Use checkError() to check Exception in command Suite & assign some error class names

[ https://issues.apache.org/jira/browse/SPARK-44555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-44555: Summary: Use checkError() to check Exception in command Suite & assign some error class names

[jira] [Updated] (SPARK-44555) Use checkError() to check Exception in command Suite & Assign new error-class

[ https://issues.apache.org/jira/browse/SPARK-44555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-44555: Summary: Use checkError() to check Exception in command Suite & Assign new error-class (was:

[jira] [Created] (SPARK-44556) Reuse `OrcTail` when enable vectorizedReader

dzcxzl created SPARK-44556: -- Summary: Reuse `OrcTail` when enable vectorizedReader Key: SPARK-44556 URL: https://issues.apache.org/jira/browse/SPARK-44556 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-44098) Introduce python breaking change detection

[ https://issues.apache.org/jira/browse/SPARK-44098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17747396#comment-17747396 ] GridGain Integration commented on SPARK-44098: -- User 'StardustDL' has created a pull

[jira] [Assigned] (SPARK-44525) Improve error message when Invoke method is not found

[ https://issues.apache.org/jira/browse/SPARK-44525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-44525: Assignee: Cheng Pan > Improve error message when Invoke method is not found >

[jira] [Resolved] (SPARK-44525) Improve error message when Invoke method is not found

[ https://issues.apache.org/jira/browse/SPARK-44525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-44525. -- Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by pull request

[jira] [Resolved] (SPARK-44544) Deduplicate run_python_packaging_tests

[ https://issues.apache.org/jira/browse/SPARK-44544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-44544. --- Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by

[jira] [Assigned] (SPARK-44544) Deduplicate run_python_packaging_tests

[ https://issues.apache.org/jira/browse/SPARK-44544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-44544: - Assignee: Ruifeng Zheng > Deduplicate run_python_packaging_tests >

[jira] [Updated] (SPARK-44544) Deduplicate run_python_packaging_tests

[ https://issues.apache.org/jira/browse/SPARK-44544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-44544: -- Summary: Deduplicate run_python_packaging_tests (was: Move python packaging tests to a

[jira] [Updated] (SPARK-44554) Install different Python linter dependencies for daily testing of different Spark versions

[ https://issues.apache.org/jira/browse/SPARK-44554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-44554: - Description: Fix daily test python lint check failure for branches 3.3 and 3.4   3.4 :

[jira] [Created] (SPARK-44555) Make branch-3.3 & branch-3.4 daily test happy

BingKun Pan created SPARK-44555: --- Summary: Make branch-3.3 & branch-3.4 daily test happy Key: SPARK-44555 URL: https://issues.apache.org/jira/browse/SPARK-44555 Project: Spark Issue Type:

[jira] [Commented] (SPARK-35914) Driver can't distribute task to executor because NullPointerException

[ https://issues.apache.org/jira/browse/SPARK-35914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17747294#comment-17747294 ] surya commented on SPARK-35914: --- Hey, We are facing similar issue and we are using spark3.1.1 with

[jira] [Created] (SPARK-44554) Install different Python linter dependencies for daily testing of different Spark versions

Yang Jie created SPARK-44554: Summary: Install different Python linter dependencies for daily testing of different Spark versions Key: SPARK-44554 URL: https://issues.apache.org/jira/browse/SPARK-44554

[jira] [Created] (SPARK-44553) Ignoring `connect-check-protos` logic in GA testing

BingKun Pan created SPARK-44553: --- Summary: Ignoring `connect-check-protos` logic in GA testing Key: SPARK-44553 URL: https://issues.apache.org/jira/browse/SPARK-44553 Project: Spark Issue