[jira] [Resolved] (SPARK-43744) Spark Connect scala UDF serialization pulling in unrelated classes not available on server

2023-07-28 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-43744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-43744. --- Fix Version/s: 3.5.0 Assignee: Zhen Li Resolution: Fixed > Spark

[jira] [Resolved] (SPARK-44574) Errors that moved into sq/api should also use Analysis

2023-07-28 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-44574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-44574. --- Fix Version/s: 3.5.0 Resolution: Fixed > Errors that moved into sq/api

[jira] [Created] (SPARK-44595) Make the user session cache number and cache time be configurable in spark connect service

2023-07-28 Thread Min Zhao (Jira)
Min Zhao created SPARK-44595: Summary: Make the user session cache number and cache time be configurable in spark connect service Key: SPARK-44595 URL: https://issues.apache.org/jira/browse/SPARK-44595

[jira] [Commented] (SPARK-44577) INSERT BY NAME returns non-sensical error message

2023-07-28 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17748772#comment-17748772 ] Jia Fan commented on SPARK-44577: - https://github.com/apache/spark/pull/42220 > INSERT BY NAME returns

[jira] [Resolved] (SPARK-44538) Remove ToJsonUtil

2023-07-28 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-44538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-44538. --- Fix Version/s: 3.5.0 Resolution: Fixed > Remove ToJsonUtil >

[jira] [Resolved] (SPARK-41400) Split of API classes from Catalyst

2023-07-28 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-41400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-41400. --- Fix Version/s: 3.5.0 Resolution: Fixed > Split of API classes from Catalyst

[jira] [Assigned] (SPARK-41400) Split of API classes from Catalyst

2023-07-28 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-41400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell reassigned SPARK-41400: - Assignee: Rui Wang > Split of API classes from Catalyst >

[jira] [Commented] (SPARK-44577) INSERT BY NAME returns non-sensical error message

2023-07-28 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17748769#comment-17748769 ] Jia Fan commented on SPARK-44577: - Let me fix this, thanks for report. > INSERT BY NAME returns

[jira] [Created] (SPARK-44594) Remove redundant method parameter in kafka connector

2023-07-28 Thread Min Zhao (Jira)
Min Zhao created SPARK-44594: Summary: Remove redundant method parameter in kafka connector Key: SPARK-44594 URL: https://issues.apache.org/jira/browse/SPARK-44594 Project: Spark Issue Type:

[jira] [Created] (SPARK-44593) Make `breaking-changes-buf` cancelable

2023-07-28 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-44593: - Summary: Make `breaking-changes-buf` cancelable Key: SPARK-44593 URL: https://issues.apache.org/jira/browse/SPARK-44593 Project: Spark Issue Type:

[jira] [Updated] (SPARK-44460) Pass user auth credential to Python workers for foreachBatch and listener

2023-07-28 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-44460: Description: No user specific credentials are sent to Python worker that runs user functions like

[jira] [Updated] (SPARK-44591) Add jobTags to SparkListenerSQLExecutionStart

2023-07-28 Thread Jean-Francois Desjeans Gauthier (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Francois Desjeans Gauthier updated SPARK-44591: Description: As part SPARK-43952, users can define job

[jira] [Updated] (SPARK-44591) Add jobTags to SparkListenerSQLExecutionStart

2023-07-28 Thread Jean-Francois Desjeans Gauthier (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Francois Desjeans Gauthier updated SPARK-44591: Description: As part SPARK-43952, users can define job

[jira] [Updated] (SPARK-44591) Add jobTags to SparkListenerSQLExecutionStart

2023-07-28 Thread Jean-Francois Desjeans Gauthier (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Francois Desjeans Gauthier updated SPARK-44591: Description: As part SPARK-43952, users can define job

[jira] [Updated] (SPARK-44591) Add jobTags to SparkListenerSQLExecutionStart

2023-07-28 Thread Jean-Francois Desjeans Gauthier (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Francois Desjeans Gauthier updated SPARK-44591: Description:  &

[jira] [Updated] (SPARK-44591) Add jobTags to SparkListenerSQLExecutionStart

2023-07-28 Thread Jean-Francois Desjeans Gauthier (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Francois Desjeans Gauthier updated SPARK-44591: Description:  &

[jira] [Updated] (SPARK-44591) Add jobTags to SparkListenerSQLExecutionStart

2023-07-28 Thread Jean-Francois Desjeans Gauthier (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Francois Desjeans Gauthier updated SPARK-44591: Component/s: Connect (was: SQL) >

[jira] [Updated] (SPARK-44591) Add jobTags to SparkListenerSQLExecutionStart

2023-07-28 Thread Jean-Francois Desjeans Gauthier (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Francois Desjeans Gauthier updated SPARK-44591: Description:  &

[jira] [Created] (SPARK-44592) Add history link to previous run on SS HA

2023-07-28 Thread Holden Karau (Jira)
Holden Karau created SPARK-44592: Summary: Add history link to previous run on SS HA Key: SPARK-44592 URL: https://issues.apache.org/jira/browse/SPARK-44592 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-44585) Fix warning condition in MLLib RankingMetrics ndcgAk

2023-07-28 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-44585. -- Fix Version/s: 3.5.0 4.0.0 3.4.2 Resolution:

[jira] [Assigned] (SPARK-44585) Fix warning condition in MLLib RankingMetrics ndcgAk

2023-07-28 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-44585: Assignee: Guilhem Vuillier > Fix warning condition in MLLib RankingMetrics ndcgAk >

[jira] [Created] (SPARK-44591) Add jobTags to SparkListenerSQLExecutionStart

2023-07-28 Thread Jean-Francois Desjeans Gauthier (Jira)
Jean-Francois Desjeans Gauthier created SPARK-44591: --- Summary: Add jobTags to SparkListenerSQLExecutionStart Key: SPARK-44591 URL: https://issues.apache.org/jira/browse/SPARK-44591

[jira] [Commented] (SPARK-44577) INSERT BY NAME returns non-sensical error message

2023-07-28 Thread Linhong Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17748745#comment-17748745 ] Linhong Liu commented on SPARK-44577: - [~fanjia] could you make a followup PR to fix this? The

[jira] [Commented] (SPARK-44577) INSERT BY NAME returns non-sensical error message

2023-07-28 Thread Linhong Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17748746#comment-17748746 ] Linhong Liu commented on SPARK-44577: - cc [~cloud_fan]  > INSERT BY NAME returns non-sensical error

[jira] [Updated] (SPARK-44588) Migrated shuffle blocks when io.encryption is enabled are encrypted multiple times

2023-07-28 Thread Henry Mai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Mai updated SPARK-44588: -- Description: Shuffle blocks upon migration are wrapped for encryption again when being written out

[jira] [Updated] (SPARK-44588) Migrated shuffle blocks are encrypted multiple times when io.encryption is enabled

2023-07-28 Thread Henry Mai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Mai updated SPARK-44588: -- Summary: Migrated shuffle blocks are encrypted multiple times when io.encryption is enabled (was:

[jira] [Created] (SPARK-44590) Remove the arrow batch record limit for SqlCommandResult

2023-07-28 Thread Yihong He (Jira)
Yihong He created SPARK-44590: - Summary: Remove the arrow batch record limit for SqlCommandResult Key: SPARK-44590 URL: https://issues.apache.org/jira/browse/SPARK-44590 Project: Spark Issue

[jira] [Created] (SPARK-44589) Migrate PySpark tests to use PySpark built-in test utils

2023-07-28 Thread Amanda Liu (Jira)
Amanda Liu created SPARK-44589: -- Summary: Migrate PySpark tests to use PySpark built-in test utils Key: SPARK-44589 URL: https://issues.apache.org/jira/browse/SPARK-44589 Project: Spark Issue

[jira] [Updated] (SPARK-44588) Migrated shuffle blocks when io.encryption is enabled are encrypted multiple times

2023-07-28 Thread Henry Mai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Mai updated SPARK-44588: -- Description: Shuffle blocks upon migration are wrapped for encryption again when being written out

[jira] [Updated] (SPARK-44588) Migrated shuffle blocks when io.encryption is enabled are encrypted multiple times

2023-07-28 Thread Henry Mai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Mai updated SPARK-44588: -- Description: Shuffle blocks upon migration are wrapped for encryption again when being written out

[jira] [Created] (SPARK-44588) Migrated shuffle blocks when io.encryption is enabled are encrypted multiple times

2023-07-28 Thread Henry Mai (Jira)
Henry Mai created SPARK-44588: - Summary: Migrated shuffle blocks when io.encryption is enabled are encrypted multiple times Key: SPARK-44588 URL: https://issues.apache.org/jira/browse/SPARK-44588

[jira] [Updated] (SPARK-44218) Customize diff log in assertDataFrameEqual error message format

2023-07-28 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44218: --- Summary: Customize diff log in assertDataFrameEqual error message format (was: Customize

[jira] [Updated] (SPARK-44218) Customize context_diff in assertDataFrameEqual error message format

2023-07-28 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-44218: --- Summary: Customize context_diff in assertDataFrameEqual error message format (was: Add improved

[jira] [Created] (SPARK-44587) Increase Protobuf Marshaller Recursion Limit

2023-07-28 Thread Yihong He (Jira)
Yihong He created SPARK-44587: - Summary: Increase Protobuf Marshaller Recursion Limit Key: SPARK-44587 URL: https://issues.apache.org/jira/browse/SPARK-44587 Project: Spark Issue Type: New

[jira] [Created] (SPARK-44586) TorchDistributor should install cpu-only Torch for testing

2023-07-28 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-44586: - Summary: TorchDistributor should install cpu-only Torch for testing Key: SPARK-44586 URL: https://issues.apache.org/jira/browse/SPARK-44586 Project: Spark

[jira] [Created] (SPARK-44585) Fix warning condition in MLLib RankingMetrics ndcgAk

2023-07-28 Thread Guilhem Vuillier (Jira)
Guilhem Vuillier created SPARK-44585: Summary: Fix warning condition in MLLib RankingMetrics ndcgAk Key: SPARK-44585 URL: https://issues.apache.org/jira/browse/SPARK-44585 Project: Spark

[jira] [Created] (SPARK-44584) AddArtifactsRequest and ArtifactStatusesRequest do not set client_type information

2023-07-28 Thread Venkata Sai Akhil Gudesa (Jira)
Venkata Sai Akhil Gudesa created SPARK-44584: Summary: AddArtifactsRequest and ArtifactStatusesRequest do not set client_type information Key: SPARK-44584 URL:

[jira] [Created] (SPARK-44583) `spark.*.io.connectionCreationTimeout` parameter documentation

2023-07-28 Thread dzcxzl (Jira)
dzcxzl created SPARK-44583: -- Summary: `spark.*.io.connectionCreationTimeout` parameter documentation Key: SPARK-44583 URL: https://issues.apache.org/jira/browse/SPARK-44583 Project: Spark Issue

[jira] [Resolved] (SPARK-44552) Remove useless private object `ParseState` from `IntervalUtils`

2023-07-28 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-44552. -- Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-44552) Remove useless private object `ParseState` from `IntervalUtils`

2023-07-28 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-44552: Assignee: Yang Jie > Remove useless private object `ParseState` from `IntervalUtils` >

[jira] [Updated] (SPARK-44582) JVM crash caused by SMJ and WindowExec

2023-07-28 Thread Wan Kun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wan Kun updated SPARK-44582: Description: After inner SMJ early cleanup offheap memory, when the SMJ call the left Window next

[jira] [Updated] (SPARK-44582) JVM crash caused by SMJ and WindowExec

2023-07-28 Thread Wan Kun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wan Kun updated SPARK-44582: Attachment: screenshot-2.png > JVM crash caused by SMJ and WindowExec >

[jira] [Updated] (SPARK-44582) JVM crash caused by SMJ and WindowExec

2023-07-28 Thread Wan Kun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wan Kun updated SPARK-44582: Attachment: screenshot-1.png > JVM crash caused by SMJ and WindowExec >

[jira] [Created] (SPARK-44582) JVM crash caused by SMJ and WindowExec

2023-07-28 Thread Wan Kun (Jira)
Wan Kun created SPARK-44582: --- Summary: JVM crash caused by SMJ and WindowExec Key: SPARK-44582 URL: https://issues.apache.org/jira/browse/SPARK-44582 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-44581) ShutdownHookManager get wrong hadoop user group information

2023-07-28 Thread liang yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang yu updated SPARK-44581: - Priority: Major (was: Minor) > ShutdownHookManager get wrong hadoop user group information >

[jira] [Updated] (SPARK-44581) ShutdownHookManager get wrong hadoop user group information

2023-07-28 Thread liang yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang yu updated SPARK-44581: - Description:  I use spark 3.2.1 to run a job on yarn in cluster mode.  when the job is finished, there

[jira] [Created] (SPARK-44581) ShutdownHookManager get wrong hadoop user group information

2023-07-28 Thread liang yu (Jira)
liang yu created SPARK-44581: Summary: ShutdownHookManager get wrong hadoop user group information Key: SPARK-44581 URL: https://issues.apache.org/jira/browse/SPARK-44581 Project: Spark Issue

[jira] [Updated] (SPARK-43968) Improve error messages for Python UDTFs with wrong number of outputs

2023-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43968: - Fix Version/s: 3.5.0 > Improve error messages for Python UDTFs with wrong number of outputs >