[jira] [Created] (SPARK-42958) Refactor `CheckConnectJvmClientCompatibility` to compare client and avro

Yang Jie created SPARK-42958: Summary: Refactor `CheckConnectJvmClientCompatibility` to compare client and avro Key: SPARK-42958 URL: https://issues.apache.org/jira/browse/SPARK-42958 Project: Spark

[jira] [Updated] (SPARK-42957) `release-build.sh` should not remove SBOM artifacts

[ https://issues.apache.org/jira/browse/SPARK-42957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-42957: -- Priority: Critical (was: Major) > `release-build.sh` should not remove SBOM artifacts >

[jira] [Created] (SPARK-42957) `release-build.sh` should not remove SBOM artifacts

Dongjoon Hyun created SPARK-42957: - Summary: `release-build.sh` should not remove SBOM artifacts Key: SPARK-42957 URL: https://issues.apache.org/jira/browse/SPARK-42957 Project: Spark Issue

[jira] [Created] (SPARK-42956) avro functions

Yang Jie created SPARK-42956: Summary: avro functions Key: SPARK-42956 URL: https://issues.apache.org/jira/browse/SPARK-42956 Project: Spark Issue Type: Improvement Components: Connect

[jira] [Resolved] (SPARK-42946) Sensitive data could still be exposed by variable substitution

[ https://issues.apache.org/jira/browse/SPARK-42946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-42946. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 40576

[jira] [Assigned] (SPARK-42946) Sensitive data could still be exposed by variable substitution

[ https://issues.apache.org/jira/browse/SPARK-42946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-42946: Assignee: Kent Yao > Sensitive data could still be exposed by variable substitution >

[jira] [Created] (SPARK-42955) Skip classifyException and wrap AnalysisException for SparkThrowable

Kent Yao created SPARK-42955: Summary: Skip classifyException and wrap AnalysisException for SparkThrowable Key: SPARK-42955 URL: https://issues.apache.org/jira/browse/SPARK-42955 Project: Spark

[jira] [Updated] (SPARK-42393) Support for Pandas/Arrow Functions API

[ https://issues.apache.org/jira/browse/SPARK-42393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-42393: - Description: There are derivative APIs which depend on the implementation of Pandas UDFs:

[jira] [Updated] (SPARK-42393) Support for Pandas/Arrow Functions API

[ https://issues.apache.org/jira/browse/SPARK-42393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-42393: - Attachment: image-2023-03-29-11-40-44-318.png > Support for Pandas/Arrow Functions API >

[jira] [Updated] (SPARK-41661) Support for User-defined Functions in Python

[ https://issues.apache.org/jira/browse/SPARK-41661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-41661: - Description: See design doc

[jira] [Updated] (SPARK-41661) Support for User-defined Functions in Python

[ https://issues.apache.org/jira/browse/SPARK-41661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-41661: - Description: User-defined Functions in Python consist of (pickled) Python UDFs and

[jira] [Updated] (SPARK-41661) Support for User-defined Functions in Python

[ https://issues.apache.org/jira/browse/SPARK-41661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-41661: - Description: See design doc

[jira] [Updated] (SPARK-41661) Support for User-defined Functions in Python

[ https://issues.apache.org/jira/browse/SPARK-41661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-41661: - Description: See design doc

[jira] [Updated] (SPARK-41661) Support for User-defined Functions in Python

[ https://issues.apache.org/jira/browse/SPARK-41661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-41661: - Description: See design doc

[jira] [Updated] (SPARK-42393) Support for Pandas/Arrow Functions API

[ https://issues.apache.org/jira/browse/SPARK-42393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-42393: - Description: See design doc

[jira] [Created] (SPARK-42954) Add `YearMonthIntervalType` to PySpark and Spark Connect Python Client

Ruifeng Zheng created SPARK-42954: - Summary: Add `YearMonthIntervalType` to PySpark and Spark Connect Python Client Key: SPARK-42954 URL: https://issues.apache.org/jira/browse/SPARK-42954 Project:

[jira] [Resolved] (SPARK-39153) When we look at spark UI or History, we can see the failed tasks first

[ https://issues.apache.org/jira/browse/SPARK-39153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jingxiong zhong resolved SPARK-39153. - Resolution: Not A Problem > When we look at spark UI or History, we can see the failed

[jira] [Resolved] (SPARK-39967) Instead of using the scalar tasksSuccessful, use the successful array to calculate whether the task is completed

[ https://issues.apache.org/jira/browse/SPARK-39967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jingxiong zhong resolved SPARK-39967. - Resolution: Fixed New version not reproduced   > Instead of using the scalar

[jira] [Created] (SPARK-42953) Impl typed map, flatMap, mapPartitions in Dataset

Zhen Li created SPARK-42953: --- Summary: Impl typed map, flatMap, mapPartitions in Dataset Key: SPARK-42953 URL: https://issues.apache.org/jira/browse/SPARK-42953 Project: Spark Issue Type:

[jira] [Updated] (SPARK-42952) Simplify the parameter of analyzer rule PreprocessTableCreation and DataSourceAnalysis

[ https://issues.apache.org/jira/browse/SPARK-42952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-42952: --- Summary: Simplify the parameter of analyzer rule PreprocessTableCreation and

[jira] [Created] (SPARK-42952) Simplify the parameter of analysis rule PreprocessTableCreation and DataSourceAnalysis

Gengliang Wang created SPARK-42952: -- Summary: Simplify the parameter of analysis rule PreprocessTableCreation and DataSourceAnalysis Key: SPARK-42952 URL: https://issues.apache.org/jira/browse/SPARK-42952

[jira] [Created] (SPARK-42951) Spark Connect: Streaming DataStreamReader API except table()

Wei Liu created SPARK-42951: --- Summary: Spark Connect: Streaming DataStreamReader API except table() Key: SPARK-42951 URL: https://issues.apache.org/jira/browse/SPARK-42951 Project: Spark Issue

[jira] [Created] (SPARK-42950) Add exit code in SparkListenerApplicationEnd

Paul Laffon created SPARK-42950: --- Summary: Add exit code in SparkListenerApplicationEnd Key: SPARK-42950 URL: https://issues.apache.org/jira/browse/SPARK-42950 Project: Spark Issue Type:

[jira] [Created] (SPARK-42949) Simplify code for NAAJ

Cheng Pan created SPARK-42949: - Summary: Simplify code for NAAJ Key: SPARK-42949 URL: https://issues.apache.org/jira/browse/SPARK-42949 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-42927) Make `o.a.spark.util.Iterators#size` as `private[spark]`

[ https://issues.apache.org/jira/browse/SPARK-42927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-42927. -- Fix Version/s: 3.4.1 3.5.0 Assignee: Yang Jie Resolution:

[jira] [Updated] (SPARK-42927) Make `o.a.spark.util.Iterators#size` as `private[spark]`

[ https://issues.apache.org/jira/browse/SPARK-42927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-42927: - Priority: Trivial (was: Major) > Make `o.a.spark.util.Iterators#size` as `private[spark]` >

[jira] [Commented] (SPARK-42947) Spark Thriftserver LDAP should not use DN pattern if user contains domain

[ https://issues.apache.org/jira/browse/SPARK-42947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17705990#comment-17705990 ] Jiayi Liu commented on SPARK-42947: --- issue fixed by https://github.com/apache/spark/pull/40577 >

[jira] [Updated] (SPARK-42937) Join with subquery in condition can fail with wholestage codegen and adaptive execution disabled

[ https://issues.apache.org/jira/browse/SPARK-42937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-42937: -- Affects Version/s: (was: 3.5.0) > Join with subquery in condition can fail with

[jira] [Assigned] (SPARK-42937) Join with subquery in condition can fail with wholestage codegen and adaptive execution disabled

[ https://issues.apache.org/jira/browse/SPARK-42937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-42937: - Assignee: Bruce Robbins > Join with subquery in condition can fail with wholestage

[jira] [Resolved] (SPARK-42937) Join with subquery in condition can fail with wholestage codegen and adaptive execution disabled

[ https://issues.apache.org/jira/browse/SPARK-42937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-42937. --- Fix Version/s: 3.3.3 3.4.1 Resolution: Fixed Issue resolved by

[jira] [Updated] (SPARK-42947) Spark Thriftserver LDAP should not use DN pattern if user contains domain

[ https://issues.apache.org/jira/browse/SPARK-42947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liu updated SPARK-42947: -- Description: When the LDAP provider has domain configuration, such as Active Directory, the

[jira] [Updated] (SPARK-42947) Spark Thriftserver LDAP should not use DN pattern if user contains domain

[ https://issues.apache.org/jira/browse/SPARK-42947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liu updated SPARK-42947: -- Description: When the LDAP provider has domain configuration, such as Active Directory, the

[jira] [Updated] (SPARK-42947) Spark Thriftserver LDAP should not use DN pattern if user contains domain

[ https://issues.apache.org/jira/browse/SPARK-42947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liu updated SPARK-42947: -- Description: When the LDAP provider includes domain configuration, such as Active Directory, the

[jira] [Updated] (SPARK-42947) Spark Thriftserver LDAP should not use DN pattern if user contains domain

[ https://issues.apache.org/jira/browse/SPARK-42947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liu updated SPARK-42947: -- Description: When the LDAP provider includes domain configuration, such as Active Directory, the

[jira] [Commented] (SPARK-41006) ConfigMap has the same name when launching two pods on the same namespace

[ https://issues.apache.org/jira/browse/SPARK-41006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17705965#comment-17705965 ] Dongjoon Hyun commented on SPARK-41006: --- Thank you for reporting and pinging me, [~dhkold]. Let me

[jira] [Commented] (SPARK-42943) Use LONGTEXT instead of TEXT for StringType

[ https://issues.apache.org/jira/browse/SPARK-42943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17705961#comment-17705961 ] Kent Yao commented on SPARK-42943: -- issue resolved by https://github.com/apache/spark/pull/40573 > Use

[jira] [Resolved] (SPARK-42943) Use LONGTEXT instead of TEXT for StringType

[ https://issues.apache.org/jira/browse/SPARK-42943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-42943. -- Fix Version/s: 3.5.0 Target Version/s: 3.5.0 Resolution: Fixed issue fixed by

[jira] [Assigned] (SPARK-42943) Use LONGTEXT instead of TEXT for StringType

[ https://issues.apache.org/jira/browse/SPARK-42943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-42943: Assignee: Kent Yao > Use LONGTEXT instead of TEXT for StringType >

[jira] (SPARK-42127) Spark 3.3.0, Error with java.io.IOException: Mkdirs failed to create file

[ https://issues.apache.org/jira/browse/SPARK-42127 ] shamim deleted comment on SPARK-42127: was (Author: JIRAUSER295634): We are using spark 3.3.0 with hadoop 3  coming with spark. Spark in our application is used as standalone , and we are not

[jira] [Comment Edited] (SPARK-42127) Spark 3.3.0, Error with java.io.IOException: Mkdirs failed to create file

[ https://issues.apache.org/jira/browse/SPARK-42127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17705950#comment-17705950 ] shamim edited comment on SPARK-42127 at 3/28/23 11:39 AM: -- We are using spark

[jira] [Commented] (SPARK-42127) Spark 3.3.0, Error with java.io.IOException: Mkdirs failed to create file

[ https://issues.apache.org/jira/browse/SPARK-42127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17705950#comment-17705950 ] shamim commented on SPARK-42127: We are using spark 3.3.0 with hadoop 3  coming with spark. Spark in

[jira] [Commented] (SPARK-42947) Spark Thriftserver LDAP should not use DN pattern if user contains domain

[ https://issues.apache.org/jira/browse/SPARK-42947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17705938#comment-17705938 ] Jiayi Liu commented on SPARK-42947: --- I will try to fix this. > Spark Thriftserver LDAP should not use

[jira] [Created] (SPARK-42948) Execution plan error, unable to obtain desired results

miaowang created SPARK-42948: Summary: Execution plan error, unable to obtain desired results Key: SPARK-42948 URL: https://issues.apache.org/jira/browse/SPARK-42948 Project: Spark Issue Type:

[jira] [Updated] (SPARK-42947) Spark Thriftserver LDAP should not use DN pattern if user contains domain

[ https://issues.apache.org/jira/browse/SPARK-42947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liu updated SPARK-42947: -- Summary: Spark Thriftserver LDAP should not use DN pattern if user contains domain (was: Spark

[jira] [Created] (SPARK-42947) Spark Thriftserver should not use dn pattern if user contains domain

Jiayi Liu created SPARK-42947: - Summary: Spark Thriftserver should not use dn pattern if user contains domain Key: SPARK-42947 URL: https://issues.apache.org/jira/browse/SPARK-42947 Project: Spark

[jira] [Resolved] (SPARK-42393) Support for Pandas/Arrow Functions API

[ https://issues.apache.org/jira/browse/SPARK-42393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-42393. -- Resolution: Resolved > Support for Pandas/Arrow Functions API >

[jira] [Updated] (SPARK-42393) Support for Pandas/Arrow Functions API

[ https://issues.apache.org/jira/browse/SPARK-42393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-42393: - Affects Version/s: (was: 3.5.0) > Support for Pandas/Arrow Functions API >

[jira] [Commented] (SPARK-41006) ConfigMap has the same name when launching two pods on the same namespace

[ https://issues.apache.org/jira/browse/SPARK-41006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17705918#comment-17705918 ] Cedric van Eetvelde commented on SPARK-41006: - Anyone to check this? [~dongjoon]  (Sorry if

[jira] [Created] (SPARK-42946) Sensitive data could still be exposed by variable substitution

Kent Yao created SPARK-42946: Summary: Sensitive data could still be exposed by variable substitution Key: SPARK-42946 URL: https://issues.apache.org/jira/browse/SPARK-42946 Project: Spark

[jira] [Assigned] (SPARK-42928) Make resolvePersistentFunction synchronized

[ https://issues.apache.org/jira/browse/SPARK-42928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-42928: --- Assignee: Allison Wang > Make resolvePersistentFunction synchronized >

[jira] [Resolved] (SPARK-42928) Make resolvePersistentFunction synchronized

[ https://issues.apache.org/jira/browse/SPARK-42928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-42928. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 40557

[jira] [Resolved] (SPARK-42936) Unresolved having at the end of analysis when using with LCA with the having clause that can be resolved directly by its child Aggregate

[ https://issues.apache.org/jira/browse/SPARK-42936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-42936. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 40558

[jira] [Assigned] (SPARK-42936) Unresolved having at the end of analysis when using with LCA with the having clause that can be resolved directly by its child Aggregate

[ https://issues.apache.org/jira/browse/SPARK-42936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-42936: --- Assignee: Xinyi Yu > Unresolved having at the end of analysis when using with LCA with the

[jira] [Updated] (SPARK-42945) Support PYSPARK_JVM_STACKTRACE_ENABLED in Spark Connect

[ https://issues.apache.org/jira/browse/SPARK-42945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated SPARK-42945: - Summary: Support PYSPARK_JVM_STACKTRACE_ENABLED in Spark Connect (was: Make

[jira] [Created] (SPARK-42945) Make PYSPARK_JVM_STACKTRACE_ENABLED work with Spark Connect

Allison Wang created SPARK-42945: Summary: Make PYSPARK_JVM_STACKTRACE_ENABLED work with Spark Connect Key: SPARK-42945 URL: https://issues.apache.org/jira/browse/SPARK-42945 Project: Spark

[jira] [Commented] (SPARK-42382) Upgrade `cyclonedx-maven-plugin` to 2.7.5

[ https://issues.apache.org/jira/browse/SPARK-42382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17705862#comment-17705862 ] Yang Jie commented on SPARK-42382: -- OK > Upgrade `cyclonedx-maven-plugin` to 2.7.5 >

[jira] [Updated] (SPARK-42895) ValueError when invoking any session operations on a stopped Spark session

[ https://issues.apache.org/jira/browse/SPARK-42895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated SPARK-42895: - Issue Type: Improvement (was: Bug) > ValueError when invoking any session operations on a