[jira] [Resolved] (SPARK-43339) LEFT JOIN is treated as INNER JOIN when being in a middle of double join

[ https://issues.apache.org/jira/browse/SPARK-43339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-43339. - Resolution: Not A Problem It is optimized by EliminateOuterJoin. > LEFT JOIN is treated as

[jira] [Assigned] (SPARK-44052) Add util to get proper Column or DataFrame class for Spark Connect.

[ https://issues.apache.org/jira/browse/SPARK-44052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-44052: - Assignee: Haejoon Lee > Add util to get proper Column or DataFrame class for Spark

[jira] [Resolved] (SPARK-44052) Add util to get proper Column or DataFrame class for Spark Connect.

[ https://issues.apache.org/jira/browse/SPARK-44052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-44052. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41570

[jira] [Assigned] (SPARK-43928) Add bit operations to Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-43928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-43928: - Assignee: jiaan.geng > Add bit operations to Scala and Python >

[jira] [Resolved] (SPARK-43928) Add bit operations to Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-43928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-43928. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41608

[jira] [Commented] (SPARK-43816) Spark Corrupts Data In-Transit for High Volume (> 20 TB/hr) of Data

[ https://issues.apache.org/jira/browse/SPARK-43816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17733690#comment-17733690 ] Yuming Wang commented on SPARK-43816: - You can also set another config:

[jira] [Updated] (SPARK-44080) Update Spark SQL config default value for thriftserver

[ https://issues.apache.org/jira/browse/SPARK-44080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-44080: Description: To support updating the Spark SQL config default value for new connections without

[jira] [Created] (SPARK-44084) Dynamic allocation pending tasks should not include finished ones

Zhongwei Zhu created SPARK-44084: Summary: Dynamic allocation pending tasks should not include finished ones Key: SPARK-44084 URL: https://issues.apache.org/jira/browse/SPARK-44084 Project: Spark

[jira] [Updated] (SPARK-44083) Spark streaming: Add max pending microbatches conf to skip scheduling new mircobatch

[ https://issues.apache.org/jira/browse/SPARK-44083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anil Dasari updated SPARK-44083: Description: In the case of uneven incoming rates and high scheduling delays, streaming will

[jira] [Updated] (SPARK-44083) Spark streaming: Add max pending microbatches conf to skip scheduling new mircobatch

[ https://issues.apache.org/jira/browse/SPARK-44083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anil Dasari updated SPARK-44083: Issue Type: Improvement (was: New Feature) > Spark streaming: Add max pending microbatches conf

[jira] [Updated] (SPARK-40307) Introduce Arrow Python UDFs

[ https://issues.apache.org/jira/browse/SPARK-40307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-40307: - Affects Version/s: (was: 3.4.0) > Introduce Arrow Python UDFs > ---

[jira] [Updated] (SPARK-44083) Spark streaming: Add max pending microbatches conf to skip scheduling new mircobatch

[ https://issues.apache.org/jira/browse/SPARK-44083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anil Dasari updated SPARK-44083: Description: In the case of uneven incoming rates and high scheduling delays, streaming will

[jira] [Updated] (SPARK-44083) Spark streaming: Add max pending microbatches conf to avoid scheduling new mircobatch

[ https://issues.apache.org/jira/browse/SPARK-44083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anil Dasari updated SPARK-44083: Summary: Spark streaming: Add max pending microbatches conf to avoid scheduling new mircobatch

[jira] [Updated] (SPARK-44083) Spark streaming: Add max pending microbatches conf to skip scheduling new mircobatch

[ https://issues.apache.org/jira/browse/SPARK-44083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anil Dasari updated SPARK-44083: Summary: Spark streaming: Add max pending microbatches conf to skip scheduling new mircobatch

[jira] [Updated] (SPARK-44083) Spark streaming: Add max pending microbatches conf to avoid new pending mircobatch

[ https://issues.apache.org/jira/browse/SPARK-44083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anil Dasari updated SPARK-44083: Description: In the case of uneven incoming rates and high scheduling delays, streaming will

[jira] [Created] (SPARK-44083) Spark streaming: Add max pending microbatches conf to avoid new pending mircobatch

Anil Dasari created SPARK-44083: --- Summary: Spark streaming: Add max pending microbatches conf to avoid new pending mircobatch Key: SPARK-44083 URL: https://issues.apache.org/jira/browse/SPARK-44083

[jira] [Resolved] (SPARK-44071) Define UnresolvedNode trait to reduce redundancy

[ https://issues.apache.org/jira/browse/SPARK-44071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-44071. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41617

[jira] [Assigned] (SPARK-44071) Define UnresolvedNode trait to reduce redundancy

[ https://issues.apache.org/jira/browse/SPARK-44071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-44071: Assignee: Ryan Johnson > Define UnresolvedNode trait to reduce redundancy >

[jira] [Resolved] (SPARK-44081) Simplify PartitionedFileUtil API a little

[ https://issues.apache.org/jira/browse/SPARK-44081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-44081. Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41632

[jira] [Assigned] (SPARK-44081) Simplify PartitionedFileUtil API a little

[ https://issues.apache.org/jira/browse/SPARK-44081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-44081: -- Assignee: Ryan Johnson > Simplify PartitionedFileUtil API a little >

[jira] [Assigned] (SPARK-42618) Support pandas 2.0.0.

[ https://issues.apache.org/jira/browse/SPARK-42618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-42618: - Assignee: Haejoon Lee > Support pandas 2.0.0. > - > >

[jira] [Resolved] (SPARK-42618) Support pandas 2.0.0.

[ https://issues.apache.org/jira/browse/SPARK-42618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-42618. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41612

[jira] [Created] (SPARK-44082) Generate operator does not update reference set properly

Rui Wang created SPARK-44082: Summary: Generate operator does not update reference set properly Key: SPARK-44082 URL: https://issues.apache.org/jira/browse/SPARK-44082 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-44041) Upgrade ammonite to 2.5.9

[ https://issues.apache.org/jira/browse/SPARK-44041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-44041: - Assignee: Yang Jie > Upgrade ammonite to 2.5.9 > - > >

[jira] [Resolved] (SPARK-44041) Upgrade ammonite to 2.5.9

[ https://issues.apache.org/jira/browse/SPARK-44041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44041. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41624

[jira] [Updated] (SPARK-44081) Simplify PartitionedFileUtil API a little

[ https://issues.apache.org/jira/browse/SPARK-44081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Johnson updated SPARK-44081: - Summary: Simplify PartitionedFileUtil API a little (was: Simplify PartitionedFileUtil API) >

[jira] [Created] (SPARK-44081) Simplify PartitionedFileUtil API

Ryan Johnson created SPARK-44081: Summary: Simplify PartitionedFileUtil API Key: SPARK-44081 URL: https://issues.apache.org/jira/browse/SPARK-44081 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-44070) Bump snappy-java 1.1.10.1

[ https://issues.apache.org/jira/browse/SPARK-44070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-44070: --- Assignee: Cheng Pan > Bump snappy-java 1.1.10.1 > - > >

[jira] [Resolved] (SPARK-44070) Bump snappy-java 1.1.10.1

[ https://issues.apache.org/jira/browse/SPARK-44070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-44070. - Fix Version/s: 3.5.0 3.4.1 Resolution: Fixed Issue resolved by pull

[jira] [Commented] (SPARK-44080) Update Spark SQL config default value for thriftserver

[ https://issues.apache.org/jira/browse/SPARK-44080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17733556#comment-17733556 ] Yuming Wang commented on SPARK-44080: - https://github.com/apache/spark/pull/41630 > Update Spark

[jira] [Created] (SPARK-44080) Update Spark SQL config default value for thriftserver

Yuming Wang created SPARK-44080: --- Summary: Update Spark SQL config default value for thriftserver Key: SPARK-44080 URL: https://issues.apache.org/jira/browse/SPARK-44080 Project: Spark Issue

[jira] [Updated] (SPARK-44079) Json reader crashes when a different schema is present

[ https://issues.apache.org/jira/browse/SPARK-44079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] charlotte van der scheun updated SPARK-44079: - Description: When using pyspark 3.4, we noticed that when reading a

[jira] [Created] (SPARK-44079) Json reader crashes when a different schema is present

charlotte van der scheun created SPARK-44079: Summary: Json reader crashes when a different schema is present Key: SPARK-44079 URL: https://issues.apache.org/jira/browse/SPARK-44079

[jira] [Commented] (SPARK-43438) Fix mismatched column list error on INSERT

[ https://issues.apache.org/jira/browse/SPARK-43438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17733496#comment-17733496 ] Max Gekk commented on SPARK-43438: -- [~beliefer] [~panbingkun] Would you like to work on this? > Fix

[jira] [Created] (SPARK-44078) Add support for classloader/resource isolation

Venkata Sai Akhil Gudesa created SPARK-44078: Summary: Add support for classloader/resource isolation Key: SPARK-44078 URL: https://issues.apache.org/jira/browse/SPARK-44078 Project: Spark

[jira] [Resolved] (SPARK-43290) Support IV and AAD optional parameters for aes_encrypt / ExpressionImplUtil

[ https://issues.apache.org/jira/browse/SPARK-43290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-43290. -- Resolution: Fixed Issue resolved by pull request 41488 [https://github.com/apache/spark/pull/41488]

[jira] [Commented] (SPARK-44073) Add date time functions to Scala and Python - part 2

[ https://issues.apache.org/jira/browse/SPARK-44073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17733476#comment-17733476 ] jiaan.geng commented on SPARK-44073: I will fix this one. > Add date time functions to Scala and

[jira] [Commented] (SPARK-43929) Add date time functions to Scala and Python - part 1

[ https://issues.apache.org/jira/browse/SPARK-43929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17733474#comment-17733474 ] jiaan.geng commented on SPARK-43929: I will take over this one. > Add date time functions to Scala

[jira] [Commented] (SPARK-41599) Memory leak in FileSystem.CACHE when submitting apps to secure cluster using InProcessLauncher

[ https://issues.apache.org/jira/browse/SPARK-41599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17733454#comment-17733454 ] Steve Loughran commented on SPARK-41599: correct. remember, all the source of hadoop is there

[jira] [Commented] (SPARK-39740) vis-timeline @ 4.2.1 vulnerable to XSS attacks

[ https://issues.apache.org/jira/browse/SPARK-39740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17733413#comment-17733413 ] ASF GitHub Bot commented on SPARK-39740: User 'shrprasa' has created a pull request for this

[jira] [Commented] (SPARK-39740) vis-timeline @ 4.2.1 vulnerable to XSS attacks

[ https://issues.apache.org/jira/browse/SPARK-39740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17733412#comment-17733412 ] ASF GitHub Bot commented on SPARK-39740: User 'shrprasa' has created a pull request for this

[jira] [Commented] (SPARK-43999) Data is still fetched even though result was returned

[ https://issues.apache.org/jira/browse/SPARK-43999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17733406#comment-17733406 ] Kamil Kliczbor commented on SPARK-43999: I checked against the version 3.4.0 and the problem

[jira] [Updated] (SPARK-43999) Data is still fetched even though result was returned

[ https://issues.apache.org/jira/browse/SPARK-43999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Kliczbor updated SPARK-43999: --- Affects Version/s: 3.4.0 (was: 3.3.2) > Data is still fetched

[jira] [Resolved] (SPARK-44075) Make 'transformStatCorr' lazy

[ https://issues.apache.org/jira/browse/SPARK-44075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44075. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41621

[jira] [Assigned] (SPARK-44075) Make 'transformStatCorr' lazy

[ https://issues.apache.org/jira/browse/SPARK-44075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-44075: Assignee: Ruifeng Zheng > Make 'transformStatCorr' lazy > -

[jira] [Assigned] (SPARK-43925) Add some, bool_or,bool_and,every to Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-43925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-43925: - Assignee: Ruifeng Zheng > Add some, bool_or,bool_and,every to Scala and Python >

[jira] [Resolved] (SPARK-43925) Add some, bool_or,bool_and,every to Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-43925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-43925. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41539