[jira] [Commented] (SPARK-44132) nesting full outer joins confuses code generator

[ https://issues.apache.org/jira/browse/SPARK-44132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736659#comment-17736659 ] Snoot.io commented on SPARK-44132: -- User 'steven-aerts' has created a pull request for this issue:

[jira] [Created] (SPARK-44164) Extract toAttribute method from StructField to Util class

Rui Wang created SPARK-44164: Summary: Extract toAttribute method from StructField to Util class Key: SPARK-44164 URL: https://issues.apache.org/jira/browse/SPARK-44164 Project: Spark Issue

[jira] [Updated] (SPARK-43974) Upgrade buf to v1.22.0

[ https://issues.apache.org/jira/browse/SPARK-43974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-43974: Summary: Upgrade buf to v1.22.0 (was: Upgrade buf to v1.21.0) > Upgrade buf to v1.22.0 >

[jira] [Resolved] (SPARK-44163) Handle ModuleNotFoundError like ImportError

[ https://issues.apache.org/jira/browse/SPARK-44163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44163. --- Resolution: Invalid > Handle ModuleNotFoundError like ImportError >

[jira] [Closed] (SPARK-44163) Handle ModuleNotFoundError like ImportError

[ https://issues.apache.org/jira/browse/SPARK-44163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-44163. - > Handle ModuleNotFoundError like ImportError > --- > >

[jira] [Created] (SPARK-44163) Handle ModuleNotFoundError like ImportError

Dongjoon Hyun created SPARK-44163: - Summary: Handle ModuleNotFoundError like ImportError Key: SPARK-44163 URL: https://issues.apache.org/jira/browse/SPARK-44163 Project: Spark Issue Type:

[jira] [Created] (SPARK-44162) Support G1GC in `spark.eventLog.gcMetrics.*` without warning

Dongjoon Hyun created SPARK-44162: - Summary: Support G1GC in `spark.eventLog.gcMetrics.*` without warning Key: SPARK-44162 URL: https://issues.apache.org/jira/browse/SPARK-44162 Project: Spark

[jira] [Created] (SPARK-44161) Row as UDF inputs causes encoder errors

Zhen Li created SPARK-44161: --- Summary: Row as UDF inputs causes encoder errors Key: SPARK-44161 URL: https://issues.apache.org/jira/browse/SPARK-44161 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-44160) Extract shared code from StructType

[ https://issues.apache.org/jira/browse/SPARK-44160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated SPARK-44160: - Description: The StructType has some methods that require CatalystParser and Catalyst expression. We

[jira] [Created] (SPARK-44160) Extract shared code from StructType

Rui Wang created SPARK-44160: Summary: Extract shared code from StructType Key: SPARK-44160 URL: https://issues.apache.org/jira/browse/SPARK-44160 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-44158) Remove unused `spark.kubernetes.executor.lostCheck.maxAttempts`

[ https://issues.apache.org/jira/browse/SPARK-44158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-44158: - Assignee: Dongjoon Hyun > Remove unused

[jira] [Resolved] (SPARK-44158) Remove unused `spark.kubernetes.executor.lostCheck.maxAttempts`

[ https://issues.apache.org/jira/browse/SPARK-44158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44158. --- Fix Version/s: 3.3.3 3.5.0 3.4.2 Resolution:

[jira] [Created] (SPARK-44159) Commands for writting (InsertIntoHadoopFsRelationCommand and InsertIntoHiveTable) should log what they are doing

Navin Kumar created SPARK-44159: --- Summary: Commands for writting (InsertIntoHadoopFsRelationCommand and InsertIntoHiveTable) should log what they are doing Key: SPARK-44159 URL:

[jira] [Updated] (SPARK-44158) Remove unused `spark.kubernetes.executor.lostCheck.maxAttempts`

[ https://issues.apache.org/jira/browse/SPARK-44158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44158: -- Affects Version/s: 3.2.4 3.1.3 3.0.3

[jira] [Updated] (SPARK-44158) Remove unused `spark.kubernetes.executor.lostCheck.maxAttempts`

[ https://issues.apache.org/jira/browse/SPARK-44158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44158: -- Affects Version/s: 3.4.1 > Remove unused `spark.kubernetes.executor.lostCheck.maxAttempts` >

[jira] [Created] (SPARK-44158) Remove unused `spark.kubernetes.executor.lostCheck.maxAttempts`

Dongjoon Hyun created SPARK-44158: - Summary: Remove unused `spark.kubernetes.executor.lostCheck.maxAttempts` Key: SPARK-44158 URL: https://issues.apache.org/jira/browse/SPARK-44158 Project: Spark

[jira] [Assigned] (SPARK-44151) Upgrade commons-codec from 1.15 to 1.16.0

[ https://issues.apache.org/jira/browse/SPARK-44151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-44151: - Assignee: BingKun Pan > Upgrade commons-codec from 1.15 to 1.16.0 >

[jira] [Resolved] (SPARK-44151) Upgrade commons-codec from 1.15 to 1.16.0

[ https://issues.apache.org/jira/browse/SPARK-44151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44151. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41707

[jira] [Updated] (SPARK-44156) SortAggregation slows down dropDuplicates()

[ https://issues.apache.org/jira/browse/SPARK-44156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Emanuel Velzi updated SPARK-44156: -- Description: TL;DR: SortAggregate makes dropDuplicates slower than HashAggregate. How to

[jira] [Updated] (SPARK-44156) SortAggregation slows down dropDuplicates()

[ https://issues.apache.org/jira/browse/SPARK-44156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Emanuel Velzi updated SPARK-44156: -- Description: TL;DR: SortAggregate makes dropDuplicates slower than HashAggregate. How to

[jira] [Updated] (SPARK-44156) SortAggregation slows down dropDuplicates()

[ https://issues.apache.org/jira/browse/SPARK-44156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Emanuel Velzi updated SPARK-44156: -- Summary: SortAggregation slows down dropDuplicates() (was: SortAggregation slows down

[jira] [Updated] (SPARK-44156) SortAggregation slows down dropDuplicates().

[ https://issues.apache.org/jira/browse/SPARK-44156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Emanuel Velzi updated SPARK-44156: -- Summary: SortAggregation slows down dropDuplicates(). (was: Should HashAggregation improve

[jira] [Created] (SPARK-44157) Outdated JARs in PySpark package

Adrian Gonzalez-Martin created SPARK-44157: -- Summary: Outdated JARs in PySpark package Key: SPARK-44157 URL: https://issues.apache.org/jira/browse/SPARK-44157 Project: Spark Issue

[jira] [Updated] (SPARK-44156) Should HashAggregation improve dropDuplicates()?

[ https://issues.apache.org/jira/browse/SPARK-44156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Emanuel Velzi updated SPARK-44156: -- Description: TL;DR: SortAggregate makes dropDuplicates slower than HashAggregate. How to

[jira] [Created] (SPARK-44156) Should HashAggregation improve dropDuplicates()?

Emanuel Velzi created SPARK-44156: - Summary: Should HashAggregation improve dropDuplicates()? Key: SPARK-44156 URL: https://issues.apache.org/jira/browse/SPARK-44156 Project: Spark Issue

[jira] [Updated] (SPARK-44134) Can't set resources (GPU/FPGA) to 0 when they are set to positive value in spark-defaults.conf

[ https://issues.apache.org/jira/browse/SPARK-44134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-44134: -- Fix Version/s: 3.4.2 (was: 3.4.1) > Can't set resources (GPU/FPGA) to

[jira] [Commented] (SPARK-44132) nesting full outer joins confuses code generator

[ https://issues.apache.org/jira/browse/SPARK-44132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736490#comment-17736490 ] Steven Aerts commented on SPARK-44132: -- [https://github.com/apache/spark/pull/41712] was created

[jira] [Comment Edited] (SPARK-43438) Fix mismatched column list error on INSERT

[ https://issues.apache.org/jira/browse/SPARK-43438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736485#comment-17736485 ] BingKun Pan edited comment on SPARK-43438 at 6/23/23 12:48 PM: --- I checked

[jira] [Comment Edited] (SPARK-43438) Fix mismatched column list error on INSERT

[ https://issues.apache.org/jira/browse/SPARK-43438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736485#comment-17736485 ] BingKun Pan edited comment on SPARK-43438 at 6/23/23 12:44 PM: --- I checked

[jira] [Comment Edited] (SPARK-43438) Fix mismatched column list error on INSERT

[ https://issues.apache.org/jira/browse/SPARK-43438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736485#comment-17736485 ] BingKun Pan edited comment on SPARK-43438 at 6/23/23 12:42 PM: --- I checked

[jira] [Comment Edited] (SPARK-43438) Fix mismatched column list error on INSERT

[ https://issues.apache.org/jira/browse/SPARK-43438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736485#comment-17736485 ] BingKun Pan edited comment on SPARK-43438 at 6/23/23 12:40 PM: --- I checked

[jira] [Comment Edited] (SPARK-43438) Fix mismatched column list error on INSERT

[ https://issues.apache.org/jira/browse/SPARK-43438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736485#comment-17736485 ] BingKun Pan edited comment on SPARK-43438 at 6/23/23 12:39 PM: --- I checked

[jira] [Commented] (SPARK-43438) Fix mismatched column list error on INSERT

[ https://issues.apache.org/jira/browse/SPARK-43438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736485#comment-17736485 ] BingKun Pan commented on SPARK-43438: - I checked and found that after

[jira] [Commented] (SPARK-44155) Adding a dev utility to improve error messages based on LLM

[ https://issues.apache.org/jira/browse/SPARK-44155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736424#comment-17736424 ] ASF GitHub Bot commented on SPARK-44155: User 'itholic' has created a pull request for this

[jira] [Commented] (SPARK-44151) Upgrade commons-codec from 1.15 to 1.16.0

[ https://issues.apache.org/jira/browse/SPARK-44151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736421#comment-17736421 ] ASF GitHub Bot commented on SPARK-44151: User 'panbingkun' has created a pull request for this

[jira] [Commented] (SPARK-44153) Support `Heap Histogram` column in Executor tab

[ https://issues.apache.org/jira/browse/SPARK-44153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736420#comment-17736420 ] ASF GitHub Bot commented on SPARK-44153: User 'dongjoon-hyun' has created a pull request for

[jira] [Created] (SPARK-44155) Adding a dev utility to improve error messages based on LLM

Haejoon Lee created SPARK-44155: --- Summary: Adding a dev utility to improve error messages based on LLM Key: SPARK-44155 URL: https://issues.apache.org/jira/browse/SPARK-44155 Project: Spark

[jira] [Commented] (SPARK-44152) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

[ https://issues.apache.org/jira/browse/SPARK-44152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736404#comment-17736404 ] Ramakrishna commented on SPARK-44152: - [~gurwls223]  Is this an issue spark 3.4.0 ? at least I am

[jira] [Commented] (SPARK-42877) Implement DataFrame.foreach

[ https://issues.apache.org/jira/browse/SPARK-42877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736399#comment-17736399 ] Gurpreet Singh commented on SPARK-42877: [~XinrongM] I am interested in working on this issue. I

[jira] [Resolved] (SPARK-44153) Support `Heap Histogram` column in Executor tab

[ https://issues.apache.org/jira/browse/SPARK-44153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44153. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41709

[jira] [Assigned] (SPARK-44153) Support `Heap Histogram` column in Executor tab

[ https://issues.apache.org/jira/browse/SPARK-44153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-44153: - Assignee: Dongjoon Hyun > Support `Heap Histogram` column in Executor tab >

[jira] [Commented] (SPARK-44152) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

[ https://issues.apache.org/jira/browse/SPARK-44152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736365#comment-17736365 ] Hyukjin Kwon commented on SPARK-44152: -- This is from

[jira] [Updated] (SPARK-44152) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

[ https://issues.apache.org/jira/browse/SPARK-44152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44152: - Affects Version/s: 3.4.0 (was: 3.5.0) Description:   I

[jira] [Commented] (SPARK-44135) Document Spark Connect only API in PySpark

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736364#comment-17736364 ] Hyukjin Kwon commented on SPARK-44135: -- I made a mistake during the JIRA resolution. I move the

[jira] [Assigned] (SPARK-44135) Document Spark Connect only API in PySpark

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-44135: Assignee: Hyukjin Kwon > Document Spark Connect only API in PySpark >

[jira] [Updated] (SPARK-44135) Document Spark Connect only API in PySpark

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44135: - Issue Type: Documentation (was: Bug) > Document Spark Connect only API in PySpark >

[jira] [Updated] (SPARK-44135) Document Spark Connect only API in PySpark

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44135: - Epic Link: SPARK-39375 > Document Spark Connect only API in PySpark >

[jira] [Updated] (SPARK-44152) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

[ https://issues.apache.org/jira/browse/SPARK-44152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44152: - Issue Type: Bug (was: Documentation) > Upgrade to spark 3.4.0 from 3.3.2 gives Exception in

[jira] [Resolved] (SPARK-44135) Document Spark Connect only API in PySpark

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44135. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41708

[jira] [Updated] (SPARK-44152) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

[ https://issues.apache.org/jira/browse/SPARK-44152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44152: - Epic Link: (was: SPARK-39375) > Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread

[jira] [Updated] (SPARK-44152) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

[ https://issues.apache.org/jira/browse/SPARK-44152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44152: - Summary: Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main"

[jira] [Updated] (SPARK-44152) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

[ https://issues.apache.org/jira/browse/SPARK-44152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44152: - Component/s: Spark Core (was: Connect) (was:

[jira] [Updated] (SPARK-44135) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44135: - Description: https://issues.apache.org/jira/browse/SPARK-41255

[jira] [Updated] (SPARK-44135) Document Spark Connect only API in PySpark

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44135: - Priority: Major (was: Blocker) > Document Spark Connect only API in PySpark >

[jira] [Updated] (SPARK-44135) Document Spark Connect only API in PySpark

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44135: - Component/s: Connect Documentation PySpark

[jira] [Updated] (SPARK-44152) Document Spark Connect only API in PySpark

[ https://issues.apache.org/jira/browse/SPARK-44152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44152: - Description:   I have a spark application that is deployed using k8s and it is of version

[jira] [Updated] (SPARK-44135) Document Spark Connect only API in PySpark

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44135: - Summary: Document Spark Connect only API in PySpark (was: Upgrade to spark 3.4.0 from 3.3.2