[jira] [Commented] (SPARK-44152) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

[ https://issues.apache.org/jira/browse/SPARK-44152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17736365#comment-17736365 ] Hyukjin Kwon commented on SPARK-44152: -- This is from https://issues.apache.org/jira

[jira] [Updated] (SPARK-44152) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

[ https://issues.apache.org/jira/browse/SPARK-44152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44152: - Affects Version/s: 3.4.0 (was: 3.5.0) Description:   I have

[jira] [Commented] (SPARK-44135) Document Spark Connect only API in PySpark

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17736364#comment-17736364 ] Hyukjin Kwon commented on SPARK-44135: -- I made a mistake during the JIRA resolution

[jira] [Assigned] (SPARK-44135) Document Spark Connect only API in PySpark

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-44135: Assignee: Hyukjin Kwon > Document Spark Connect only API in PySpark > --

[jira] [Updated] (SPARK-44135) Document Spark Connect only API in PySpark

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44135: - Issue Type: Documentation (was: Bug) > Document Spark Connect only API in PySpark > --

[jira] [Updated] (SPARK-44135) Document Spark Connect only API in PySpark

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44135: - Epic Link: SPARK-39375 > Document Spark Connect only API in PySpark > -

[jira] [Updated] (SPARK-44152) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

[ https://issues.apache.org/jira/browse/SPARK-44152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44152: - Issue Type: Bug (was: Documentation) > Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thr

[jira] [Resolved] (SPARK-44135) Document Spark Connect only API in PySpark

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44135. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41708 [https://gi

[jira] [Updated] (SPARK-44152) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

[ https://issues.apache.org/jira/browse/SPARK-44152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44152: - Epic Link: (was: SPARK-39375) > Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "m

[jira] [Updated] (SPARK-44152) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

[ https://issues.apache.org/jira/browse/SPARK-44152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44152: - Summary: Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuch

[jira] [Updated] (SPARK-44152) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

[ https://issues.apache.org/jira/browse/SPARK-44152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44152: - Component/s: Spark Core (was: Connect) (was: Documentat

[jira] [Updated] (SPARK-44135) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44135: - Description: https://issues.apache.org/jira/browse/SPARK-41255 https://issues.apache.org/jira/br

[jira] [Updated] (SPARK-44135) Document Spark Connect only API in PySpark

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44135: - Priority: Major (was: Blocker) > Document Spark Connect only API in PySpark >

[jira] [Updated] (SPARK-44135) Document Spark Connect only API in PySpark

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44135: - Component/s: Connect Documentation PySpark

[jira] [Updated] (SPARK-44152) Document Spark Connect only API in PySpark

[ https://issues.apache.org/jira/browse/SPARK-44152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44152: - Description:   I have a spark application that is deployed using k8s and it is of version 3.3

[jira] [Updated] (SPARK-44135) Document Spark Connect only API in PySpark

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44135: - Summary: Document Spark Connect only API in PySpark (was: Upgrade to spark 3.4.0 from 3.3.2 gi

[jira] [Created] (SPARK-44154) Bitmap functions

Entong Shen created SPARK-44154: --- Summary: Bitmap functions Key: SPARK-44154 URL: https://issues.apache.org/jira/browse/SPARK-44154 Project: Spark Issue Type: Improvement Components:

[jira] [Resolved] (SPARK-44139) Discard completely pushed down filters in group-based MERGE operations

[ https://issues.apache.org/jira/browse/SPARK-44139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44139. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41700 [https://

[jira] [Assigned] (SPARK-44139) Discard completely pushed down filters in group-based MERGE operations

[ https://issues.apache.org/jira/browse/SPARK-44139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-44139: - Assignee: Anton Okolnychyi > Discard completely pushed down filters in group-based MERG

[jira] [Created] (SPARK-44153) Support `Heap Histogram` column in Executor tab

Dongjoon Hyun created SPARK-44153: - Summary: Support `Heap Histogram` column in Executor tab Key: SPARK-44153 URL: https://issues.apache.org/jira/browse/SPARK-44153 Project: Spark Issue Type:

[jira] [Updated] (SPARK-44152) Document Spark Connect only API in PySpark

[ https://issues.apache.org/jira/browse/SPARK-44152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44152: - Summary: Document Spark Connect only API in PySpark (was: Document Spark Connect only API) > D

[jira] [Created] (SPARK-44152) Document Spark Connect only API

Hyukjin Kwon created SPARK-44152: Summary: Document Spark Connect only API Key: SPARK-44152 URL: https://issues.apache.org/jira/browse/SPARK-44152 Project: Spark Issue Type: Documentation

[jira] [Assigned] (SPARK-44134) Can't set resources (GPU/FPGA) to 0 when they are set to positive value in spark-defaults.conf

[ https://issues.apache.org/jira/browse/SPARK-44134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-44134: - Assignee: Thomas Graves > Can't set resources (GPU/FPGA) to 0 when they are set to posi

[jira] [Resolved] (SPARK-44134) Can't set resources (GPU/FPGA) to 0 when they are set to positive value in spark-defaults.conf

[ https://issues.apache.org/jira/browse/SPARK-44134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44134. --- Fix Version/s: 3.3.3 3.5.0 3.4.1 Resolution: Fix

[jira] [Resolved] (SPARK-43990) Upgrade kubernetes-client to 6.7.1

[ https://issues.apache.org/jira/browse/SPARK-43990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-43990. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41490 [https://

[jira] [Assigned] (SPARK-43990) Upgrade kubernetes-client to 6.7.1

[ https://issues.apache.org/jira/browse/SPARK-43990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-43990: - Assignee: Bjørn Jørgensen > Upgrade kubernetes-client to 6.7.1 > --

[jira] [Resolved] (SPARK-44110) Forked unit tests don't receive proxy system properties

[ https://issues.apache.org/jira/browse/SPARK-44110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44110. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41678 [https://

[jira] [Assigned] (SPARK-44110) Forked unit tests don't receive proxy system properties

[ https://issues.apache.org/jira/browse/SPARK-44110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-44110: - Assignee: Steve Vaughan > Forked unit tests don't receive proxy system properties > ---

[jira] [Commented] (SPARK-43438) Fix mismatched column list error on INSERT

[ https://issues.apache.org/jira/browse/SPARK-43438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17736317#comment-17736317 ] BingKun Pan commented on SPARK-43438: - Let me to do it. > Fix mismatched column lis

[jira] [Assigned] (SPARK-44142) Utility to convert python types to spark types compares Python "type" object rather than user's "tpe" for categorical data types

[ https://issues.apache.org/jira/browse/SPARK-44142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-44142: Assignee: Ted Chester Jenks > Utility to convert python types to spark types compares Pyt

[jira] [Resolved] (SPARK-44142) Utility to convert python types to spark types compares Python "type" object rather than user's "tpe" for categorical data types

[ https://issues.apache.org/jira/browse/SPARK-44142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44142. -- Fix Version/s: 3.3.3 3.5.0 3.4.1 Resolution: Fixed

[jira] [Created] (SPARK-44151) Upgrade commons-codec from 1.15 to 1.16.0

BingKun Pan created SPARK-44151: --- Summary: Upgrade commons-codec from 1.15 to 1.16.0 Key: SPARK-44151 URL: https://issues.apache.org/jira/browse/SPARK-44151 Project: Spark Issue Type: Improveme

[jira] [Created] (SPARK-44150) Explicit Arrow casting for mismatched return type in Arrow Python UDF

Xinrong Meng created SPARK-44150: Summary: Explicit Arrow casting for mismatched return type in Arrow Python UDF Key: SPARK-44150 URL: https://issues.apache.org/jira/browse/SPARK-44150 Project: Spark

[jira] [Assigned] (SPARK-44138) Prohibit non-deterministic expressions, subqueries and aggregates in MERGE conditions

[ https://issues.apache.org/jira/browse/SPARK-44138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-44138: - Assignee: Anton Okolnychyi > Prohibit non-deterministic expressions, subqueries and agg

[jira] [Resolved] (SPARK-44138) Prohibit non-deterministic expressions, subqueries and aggregates in MERGE conditions

[ https://issues.apache.org/jira/browse/SPARK-44138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44138. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41694 [https://

[jira] [Created] (SPARK-44149) Support DataFrame Merge API

Huaxin Gao created SPARK-44149: -- Summary: Support DataFrame Merge API Key: SPARK-44149 URL: https://issues.apache.org/jira/browse/SPARK-44149 Project: Spark Issue Type: Sub-task Compon

[jira] [Created] (SPARK-44148) Validate NO_BROADCAST_AND_REPLICATION hint is never set on both sides

Anton Okolnychyi created SPARK-44148: Summary: Validate NO_BROADCAST_AND_REPLICATION hint is never set on both sides Key: SPARK-44148 URL: https://issues.apache.org/jira/browse/SPARK-44148 Project

[jira] [Resolved] (SPARK-44147) Add `spark.ui.allowFramingFrom` doc to `Security` page

[ https://issues.apache.org/jira/browse/SPARK-44147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44147. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41702 [https://

[jira] [Assigned] (SPARK-44147) Add `spark.ui.allowFramingFrom` doc to `Security` page

[ https://issues.apache.org/jira/browse/SPARK-44147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-44147: - Assignee: Dongjoon Hyun > Add `spark.ui.allowFramingFrom` doc to `Security` page >

[jira] [Updated] (SPARK-44147) Add `spark.ui.allowFramingFrom` doc to `Security` page

[ https://issues.apache.org/jira/browse/SPARK-44147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44147: -- Summary: Add `spark.ui.allowFramingFrom` doc to `Security` page (was: Add `spark.ui.allowFram

[jira] [Created] (SPARK-44147) Add `spark.ui.allowFramingFrom` doc

Dongjoon Hyun created SPARK-44147: - Summary: Add `spark.ui.allowFramingFrom` doc Key: SPARK-44147 URL: https://issues.apache.org/jira/browse/SPARK-44147 Project: Spark Issue Type: Documentati

[jira] [Updated] (SPARK-44145) Callback prior to execution

[ https://issues.apache.org/jira/browse/SPARK-44145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Francois Desjeans Gauthier updated SPARK-44145: Description: Commands are eagerly executed after analysis p

[jira] [Created] (SPARK-44146) Isolate Spark Connect session/artifacts

Venkata Sai Akhil Gudesa created SPARK-44146: Summary: Isolate Spark Connect session/artifacts Key: SPARK-44146 URL: https://issues.apache.org/jira/browse/SPARK-44146 Project: Spark

[jira] [Created] (SPARK-44145) Callback prior to execution

Jean-Francois Desjeans Gauthier created SPARK-44145: --- Summary: Callback prior to execution Key: SPARK-44145 URL: https://issues.apache.org/jira/browse/SPARK-44145 Project: Spark

[jira] [Created] (SPARK-44144) Enable `spark.authenticate` by default in K8s environment

Dongjoon Hyun created SPARK-44144: - Summary: Enable `spark.authenticate` by default in K8s environment Key: SPARK-44144 URL: https://issues.apache.org/jira/browse/SPARK-44144 Project: Spark I

[jira] [Updated] (SPARK-44134) Can't set resources (GPU/FPGA) to 0 when they are set to positive value in spark-defaults.conf

[ https://issues.apache.org/jira/browse/SPARK-44134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-44134: -- Description: With resource aware scheduling, if you specify a default value in the spark-defa

[jira] [Commented] (SPARK-43816) Spark Corrupts Data In-Transit for High Volume (> 20 TB/hr) of Data

[ https://issues.apache.org/jira/browse/SPARK-43816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17736170#comment-17736170 ] Sai Allu commented on SPARK-43816: -- Hello Yuming, could you please explain a bit more w

[jira] [Created] (SPARK-44143) Use checkError() to check Exception in *DDL*Suite

BingKun Pan created SPARK-44143: --- Summary: Use checkError() to check Exception in *DDL*Suite Key: SPARK-44143 URL: https://issues.apache.org/jira/browse/SPARK-44143 Project: Spark Issue Type: I

[jira] [Resolved] (SPARK-44140) Support positional parameters in PySpark sql()

[ https://issues.apache.org/jira/browse/SPARK-44140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-44140. -- Resolution: Fixed Issue resolved by pull request 41695 [https://github.com/apache/spark/pull/41695] >

[jira] [Commented] (SPARK-44142) Utility to convert python types to spark types compares Python "type" object rather than user's "tpe" for categorical data types

[ https://issues.apache.org/jira/browse/SPARK-44142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17736100#comment-17736100 ] Ted Chester Jenks commented on SPARK-44142: --- https://github.com/apache/spark/p

[jira] [Updated] (SPARK-44142) Utility to convert python types to spark types compares Python "type" object rather than user's "tpe" for categorical data types

[ https://issues.apache.org/jira/browse/SPARK-44142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Chester Jenks updated SPARK-44142: -- Description: In the typehints utility that converts python types to spark types, the l

[jira] [Created] (SPARK-44142) Utility to convert python types to spark types compares Python "type" object rather than user's "tpe" for categorical data types

Ted Chester Jenks created SPARK-44142: - Summary: Utility to convert python types to spark types compares Python "type" object rather than user's "tpe" for categorical data types Key: SPARK-44142 URL: https://i

[jira] [Commented] (SPARK-38230) InsertIntoHadoopFsRelationCommand unnecessarily fetches details of partitions in most cases

[ https://issues.apache.org/jira/browse/SPARK-38230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17736041#comment-17736041 ] ASF GitHub Bot commented on SPARK-38230: User 'jeanlyn' has created a pull reque

[jira] [Commented] (SPARK-36680) Supports Dynamic Table Options for Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-36680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17736039#comment-17736039 ] ASF GitHub Bot commented on SPARK-36680: User 'szehon-ho' has created a pull req

[jira] [Commented] (SPARK-36680) Supports Dynamic Table Options for Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-36680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17736038#comment-17736038 ] ASF GitHub Bot commented on SPARK-36680: User 'szehon-ho' has created a pull req

[jira] [Updated] (SPARK-44141) Remove need to preinstall the buf compiler

[ https://issues.apache.org/jira/browse/SPARK-44141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arnar Pall updated SPARK-44141: --- Description: In order to lower the barrier of entry even further for this project we can remove nee

[jira] [Created] (SPARK-44141) Remove need to preinstall the buf compiler

Arnar Pall created SPARK-44141: -- Summary: Remove need to preinstall the buf compiler Key: SPARK-44141 URL: https://issues.apache.org/jira/browse/SPARK-44141 Project: Spark Issue Type: Improvemen

[jira] [Updated] (SPARK-44140) Support positional parameters in PySpark sql()

[ https://issues.apache.org/jira/browse/SPARK-44140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-44140: - Component/s: PySpark (was: SQL) > Support positional parameters in PySpark sql() >

[jira] [Created] (SPARK-44140) Support positional parameters in PySpark sql()

Max Gekk created SPARK-44140: Summary: Support positional parameters in PySpark sql() Key: SPARK-44140 URL: https://issues.apache.org/jira/browse/SPARK-44140 Project: Spark Issue Type: New Featur