[jira] [Resolved] (SPARK-44066) Support positional parameters in parameterized query

[ https://issues.apache.org/jira/browse/SPARK-44066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-44066. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41568 [https://github.com

[jira] [Commented] (SPARK-44110) Forked unit tests don't receive proxy system properties

[ https://issues.apache.org/jira/browse/SPARK-44110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735978#comment-17735978 ] Snoot.io commented on SPARK-44110: -- User 'snmvaughan' has created a pull request for th

[jira] [Commented] (SPARK-44132) nesting full outer joins confuses code generator

[ https://issues.apache.org/jira/browse/SPARK-44132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735976#comment-17735976 ] Bruce Robbins commented on SPARK-44132: --- [~steven.aerts] Go for it! > nesting ful

[jira] [Commented] (SPARK-44132) nesting full outer joins confuses code generator

[ https://issues.apache.org/jira/browse/SPARK-44132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735973#comment-17735973 ] Steven Aerts commented on SPARK-44132: -- [~bersprockets] I did not think of that. A

[jira] [Comment Edited] (SPARK-44132) nesting full outer joins confuses code generator

[ https://issues.apache.org/jira/browse/SPARK-44132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735944#comment-17735944 ] Bruce Robbins edited comment on SPARK-44132 at 6/22/23 1:51 AM: --

[jira] [Commented] (SPARK-44132) nesting full outer joins confuses code generator

[ https://issues.apache.org/jira/browse/SPARK-44132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735944#comment-17735944 ] Bruce Robbins commented on SPARK-44132: --- You may have this figured out already, bu

[jira] [Resolved] (SPARK-44078) Add support for classloader/resource isolation

[ https://issues.apache.org/jira/browse/SPARK-44078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-44078. --- Fix Version/s: 3.5.0 Assignee: Venkata Sai Akhil Gudesa Resolution: F

[jira] [Assigned] (SPARK-44136) StateManager may get materialized in executor instead of driver in FlatMapGroupsWithStateExec

[ https://issues.apache.org/jira/browse/SPARK-44136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-44136: Assignee: Bo Gao > StateManager may get materialized in executor instead of driver in >

[jira] [Resolved] (SPARK-44136) StateManager may get materialized in executor instead of driver in FlatMapGroupsWithStateExec

[ https://issues.apache.org/jira/browse/SPARK-44136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44136. -- Fix Version/s: 3.5.0 3.4.1 Resolution: Fixed Issue resolved by pull

[jira] [Created] (SPARK-44139) Discard completely pushed down filters in group-based MERGE operations

Anton Okolnychyi created SPARK-44139: Summary: Discard completely pushed down filters in group-based MERGE operations Key: SPARK-44139 URL: https://issues.apache.org/jira/browse/SPARK-44139 Projec

[jira] [Assigned] (SPARK-43710) Support functions.date_part for Spark Connect

[ https://issues.apache.org/jira/browse/SPARK-43710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43710: Assignee: Haejoon Lee > Support functions.date_part for Spark Connect > -

[jira] [Updated] (SPARK-44128) Upgrade netty from 4.1.92 to 4.1.93

[ https://issues.apache.org/jira/browse/SPARK-44128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-44128: Summary: Upgrade netty from 4.1.92 to 4.1.93 (was: Upgrade netty from 4.1.92 to 4.1.94) > Upgrad

[jira] [Commented] (SPARK-44136) StateManager may get materialized in executor instead of driver in FlatMapGroupsWithStateExec

[ https://issues.apache.org/jira/browse/SPARK-44136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735926#comment-17735926 ] Jungtaek Lim commented on SPARK-44136: -- PR: https://github.com/apache/spark/pull/41

[jira] [Updated] (SPARK-44138) Prohibit non-deterministic expressions, subqueries and aggregates in MERGE conditions

[ https://issues.apache.org/jira/browse/SPARK-44138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anton Okolnychyi updated SPARK-44138: - Summary: Prohibit non-deterministic expressions, subqueries and aggregates in MERGE cond

[jira] [Created] (SPARK-44138) Prohibit non-deterministic conditions, subqueries and aggregates in MERGE conditions

Anton Okolnychyi created SPARK-44138: Summary: Prohibit non-deterministic conditions, subqueries and aggregates in MERGE conditions Key: SPARK-44138 URL: https://issues.apache.org/jira/browse/SPARK-44138

[jira] [Created] (SPARK-44137) Change handling of iterable objects for on field in joins

John Haberstroh created SPARK-44137: --- Summary: Change handling of iterable objects for on field in joins Key: SPARK-44137 URL: https://issues.apache.org/jira/browse/SPARK-44137 Project: Spark

[jira] [Created] (SPARK-44136) StateManager may get materialized in executor instead of driver in FlatMapGroupsWithStateExec

Bo Gao created SPARK-44136: -- Summary: StateManager may get materialized in executor instead of driver in FlatMapGroupsWithStateExec Key: SPARK-44136 URL: https://issues.apache.org/jira/browse/SPARK-44136 Pro

[jira] [Assigned] (SPARK-44133) Upgrade MyPy from 0.920 to 0.982

[ https://issues.apache.org/jira/browse/SPARK-44133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-44133: - Assignee: Hyukjin Kwon > Upgrade MyPy from 0.920 to 0.982 > ---

[jira] [Resolved] (SPARK-44133) Upgrade MyPy from 0.920 to 0.982

[ https://issues.apache.org/jira/browse/SPARK-44133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44133. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41690 [https://

[jira] [Commented] (SPARK-39375) SPIP: Spark Connect - A client and server interface for Apache Spark

[ https://issues.apache.org/jira/browse/SPARK-39375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735876#comment-17735876 ] Michael Allman commented on SPARK-39375: To be clear, Spark Connect will be an a

[jira] [Commented] (SPARK-42962) Allow access to stopped streaming queries

[ https://issues.apache.org/jira/browse/SPARK-42962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735867#comment-17735867 ] Wei Liu commented on SPARK-42962: - Closing this as it's duplicate with SPARK-42940. Plea

[jira] [Resolved] (SPARK-42962) Allow access to stopped streaming queries

[ https://issues.apache.org/jira/browse/SPARK-42962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu resolved SPARK-42962. - Resolution: Duplicate > Allow access to stopped streaming queries >

[jira] [Commented] (SPARK-42941) Add support for streaming listener in Python

[ https://issues.apache.org/jira/browse/SPARK-42941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735866#comment-17735866 ] Wei Liu commented on SPARK-42941: - Hmmm this is actually still in progress. I was thinki

[jira] [Reopened] (SPARK-42941) Add support for streaming listener in Python

[ https://issues.apache.org/jira/browse/SPARK-42941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu reopened SPARK-42941: - > Add support for streaming listener in Python > > >

[jira] [Assigned] (SPARK-43915) Assign names to the error class _LEGACY_ERROR_TEMP_[2438-2445]

[ https://issues.apache.org/jira/browse/SPARK-43915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-43915: Assignee: jiaan.geng > Assign names to the error class _LEGACY_ERROR_TEMP_[2438-2445] > -

[jira] [Resolved] (SPARK-43915) Assign names to the error class _LEGACY_ERROR_TEMP_[2438-2445]

[ https://issues.apache.org/jira/browse/SPARK-43915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-43915. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41553 [https://github.com

[jira] [Commented] (SPARK-43924) Add misc functions to Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-43924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735789#comment-17735789 ] Ignite TC Bot commented on SPARK-43924: --- User 'panbingkun' has created a pull requ

[jira] [Updated] (SPARK-44135) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramakrishna updated SPARK-44135: Description:   I have a spark application that is deployed using k8s and it is of version 3.3.2 R

[jira] [Updated] (SPARK-44135) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramakrishna updated SPARK-44135: Priority: Blocker (was: Critical) > Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "

[jira] [Updated] (SPARK-44135) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramakrishna updated SPARK-44135: Issue Type: Bug (was: Improvement) > Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread

[jira] [Updated] (SPARK-44135) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramakrishna updated SPARK-44135: Description:   I have a spark application that is deployed using k8s and it is of version 3.3.2 R

[jira] [Updated] (SPARK-44135) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramakrishna updated SPARK-44135: Description:   I have a spark application that is deployed using k8s and it is of version 3.3.2 R

[jira] [Updated] (SPARK-44135) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramakrishna updated SPARK-44135: Component/s: Spark Core (was: Shuffle) > Upgrade to spark 3.4.0 from 3.3.2 gi

[jira] [Updated] (SPARK-44135) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramakrishna updated SPARK-44135: Affects Version/s: (was: 3.2.0) > Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread

[jira] [Updated] (SPARK-44135) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramakrishna updated SPARK-44135: Description:   I have a spark application that is deployed using k8s and it is of version 3.3.2 R

[jira] [Updated] (SPARK-44135) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

[ https://issues.apache.org/jira/browse/SPARK-44135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramakrishna updated SPARK-44135: Description: (was: In our production environment, _finalizeShuffleMerge_ processing took longe

[jira] [Created] (SPARK-44135) Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location

Ramakrishna created SPARK-44135: --- Summary: Upgrade to spark 3.4.0 from 3.3.2 gives Exception in thread "main" java.nio.file.NoSuchFileException: , although jar is present in the location Key: SPARK-44135 URL: https

[jira] [Assigned] (SPARK-44056) Improve error message when UDF execution fails

[ https://issues.apache.org/jira/browse/SPARK-44056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-44056: Assignee: Rob Reeves > Improve error message when UDF execution fails > -

[jira] [Resolved] (SPARK-44056) Improve error message when UDF execution fails

[ https://issues.apache.org/jira/browse/SPARK-44056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-44056. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41599 [https://github.com

[jira] [Resolved] (SPARK-44109) Remove duplicate preferred locations of each RDD partition

[ https://issues.apache.org/jira/browse/SPARK-44109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44109. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41676 [https://

[jira] [Assigned] (SPARK-44109) Remove duplicate preferred locations of each RDD partition

[ https://issues.apache.org/jira/browse/SPARK-44109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-44109: - Assignee: Wan Kun > Remove duplicate preferred locations of each RDD partition > --

[jira] [Commented] (SPARK-44042) SPIP: PySpark Test Framework

[ https://issues.apache.org/jira/browse/SPARK-44042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735747#comment-17735747 ] Amanda Liu commented on SPARK-44042: [~ste...@apache.org] thank you for the comment!

[jira] [Created] (SPARK-44134) Can't set resources (GPU/FPGA) to 0 when they are set to positive value in spark-defaults.conf

Thomas Graves created SPARK-44134: - Summary: Can't set resources (GPU/FPGA) to 0 when they are set to positive value in spark-defaults.conf Key: SPARK-44134 URL: https://issues.apache.org/jira/browse/SPARK-44134

[jira] [Commented] (SPARK-44134) Can't set resources (GPU/FPGA) to 0 when they are set to positive value in spark-defaults.conf

[ https://issues.apache.org/jira/browse/SPARK-44134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735746#comment-17735746 ] Thomas Graves commented on SPARK-44134: --- I'm working on a fix for this > Can't se

[jira] [Commented] (SPARK-41599) Memory leak in FileSystem.CACHE when submitting apps to secure cluster using InProcessLauncher

[ https://issues.apache.org/jira/browse/SPARK-41599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735743#comment-17735743 ] Xieming Li commented on SPARK-41599: Hi, [~ste...@apache.org], Thank you very much f

[jira] [Resolved] (SPARK-42740) Fix the bug that pushdown offset or paging is invalid for some built-in dialect

[ https://issues.apache.org/jira/browse/SPARK-42740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng resolved SPARK-42740. Resolution: Resolved > Fix the bug that pushdown offset or paging is invalid for some built-in >

[jira] [Commented] (SPARK-42740) Fix the bug that pushdown offset or paging is invalid for some built-in dialect

[ https://issues.apache.org/jira/browse/SPARK-42740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735688#comment-17735688 ] jiaan.geng commented on SPARK-42740: Resolved by https://github.com/apache/spark/pul

[jira] [Commented] (SPARK-44042) SPIP: PySpark Test Framework

[ https://issues.apache.org/jira/browse/SPARK-44042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735646#comment-17735646 ] Steve Loughran commented on SPARK-44042: * you can create an independent git rep

[jira] [Commented] (SPARK-43903) Improve ArrayType input support in Arrow Python UDF

[ https://issues.apache.org/jira/browse/SPARK-43903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735623#comment-17735623 ] ASF GitHub Bot commented on SPARK-43903: User 'xinrong-meng' has created a pull

[jira] [Resolved] (SPARK-43903) Improve ArrayType input support in Arrow Python UDF

[ https://issues.apache.org/jira/browse/SPARK-43903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-43903. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41603 [https://

[jira] [Assigned] (SPARK-43903) Improve ArrayType input support in Arrow Python UDF

[ https://issues.apache.org/jira/browse/SPARK-43903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-43903: - Assignee: Xinrong Meng > Improve ArrayType input support in Arrow Python UDF >

[jira] [Created] (SPARK-44133) Upgrade MyPy from 0.920 to 0.982

Hyukjin Kwon created SPARK-44133: Summary: Upgrade MyPy from 0.920 to 0.982 Key: SPARK-44133 URL: https://issues.apache.org/jira/browse/SPARK-44133 Project: Spark Issue Type: Task C

[jira] [Updated] (SPARK-44132) nesting full outer joins confuses code generator

[ https://issues.apache.org/jira/browse/SPARK-44132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Aerts updated SPARK-44132: - Description: We are seeing issues with the code generator when querying java bean encoded data

[jira] [Updated] (SPARK-44132) nesting full outer joins confuses code generator

[ https://issues.apache.org/jira/browse/SPARK-44132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Aerts updated SPARK-44132: - Description: We are seeing issues with the code generator when querying java bean encoded data

[jira] [Updated] (SPARK-44132) nesting full outer joins confuses code generator

[ https://issues.apache.org/jira/browse/SPARK-44132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Aerts updated SPARK-44132: - Summary: nesting full outer joins confuses code generator (was: nesting full outer joins confus

[jira] [Updated] (SPARK-44004) Assign name & improve error message for frequent LEGACY errors.

[ https://issues.apache.org/jira/browse/SPARK-44004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-44004: - Fix Version/s: 3.5.0 (was: 4.0.0) > Assign name & improve error message for frequ

[jira] [Assigned] (SPARK-44004) Assign name & improve error message for frequent LEGACY errors.

[ https://issues.apache.org/jira/browse/SPARK-44004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-44004: Assignee: Haejoon Lee > Assign name & improve error message for frequent LEGACY errors. > ---

[jira] [Resolved] (SPARK-44004) Assign name & improve error message for frequent LEGACY errors.

[ https://issues.apache.org/jira/browse/SPARK-44004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-44004. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 41504 [https://github.com

[jira] [Created] (SPARK-44132) nesting full outer joins confuses confuses code generator

Steven Aerts created SPARK-44132: Summary: nesting full outer joins confuses confuses code generator Key: SPARK-44132 URL: https://issues.apache.org/jira/browse/SPARK-44132 Project: Spark Iss