[jira] [Created] (SPARK-44034) Add a new test group for sql module

2023-06-12 Thread Yang Jie (Jira)
Yang Jie created SPARK-44034: Summary: Add a new test group for sql module Key: SPARK-44034 URL: https://issues.apache.org/jira/browse/SPARK-44034 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-43885) DataSource V2: Handle MERGE commands for delta-based sources

2023-06-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-43885. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41448

[jira] [Assigned] (SPARK-43885) DataSource V2: Handle MERGE commands for delta-based sources

2023-06-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-43885: - Assignee: Anton Okolnychyi > DataSource V2: Handle MERGE commands for delta-based

[jira] [Updated] (SPARK-44032) Remove the bytecode version exclusion of threeten-extra when orc-core fixes the violation

2023-06-12 Thread Bowen Liang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Liang updated SPARK-44032: Affects Version/s: 3.4.0 (was: 3.4.1) > Remove the bytecode version

[jira] [Updated] (SPARK-44032) Remove the bytecode version exclusion of threeten-extra when orc-core fixes the violation

2023-06-12 Thread Bowen Liang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Liang updated SPARK-44032: Target Version/s: (was: 3.4.1) > Remove the bytecode version exclusion of threeten-extra when

[jira] [Updated] (SPARK-44022) Enforce max bytecode version on Maven dependencies

2023-06-12 Thread Bowen Liang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Liang updated SPARK-44022: Target Version/s: (was: 3.4.1) > Enforce max bytecode version on Maven dependencies >

[jira] [Commented] (SPARK-44025) CSV Table Read Error with CharType(length) column

2023-06-12 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17731835#comment-17731835 ] BingKun Pan commented on SPARK-44025: - [~cloud_fan] Can I try to fix it? > CSV Table Read Error

[jira] [Created] (SPARK-44033) Support list-like for binary ops

2023-06-12 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-44033: --- Summary: Support list-like for binary ops Key: SPARK-44033 URL: https://issues.apache.org/jira/browse/SPARK-44033 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-43798) Initial support for Python UDTFs

2023-06-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43798: Assignee: Allison Wang > Initial support for Python UDTFs >

[jira] [Resolved] (SPARK-43798) Initial support for Python UDTFs

2023-06-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43798. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41316

[jira] [Commented] (SPARK-43944) Add string functions to Scala and Python - part 2

2023-06-12 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17731820#comment-17731820 ] BingKun Pan commented on SPARK-43944: - I work on it. > Add string functions to Scala and Python -

[jira] [Updated] (SPARK-44022) Enforce max bytecode version on Maven dependencies

2023-06-12 Thread Bowen Liang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Liang updated SPARK-44022: Summary: Enforce max bytecode version on Maven dependencies (was: Enforce Java max bytecode

[jira] [Updated] (SPARK-44022) Enforce Java max bytecode version to maven dependencies

2023-06-12 Thread Bowen Liang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Liang updated SPARK-44022: Target Version/s: 3.4.1 Affects Version/s: 3.4.0 (was: 3.5.0) >

[jira] [Updated] (SPARK-44032) Remove the bytecode version exclusion of threeten-extra when orc-core fixes the violation

2023-06-12 Thread Bowen Liang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Liang updated SPARK-44032: Target Version/s: 3.4.1 > Remove the bytecode version exclusion of threeten-extra when orc-core

[jira] [Updated] (SPARK-44032) Remove the bytecode version exclusion of threeten-extra when orc-core fixes the violation

2023-06-12 Thread Bowen Liang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Liang updated SPARK-44032: Description: Remove the exclusion of threeten-extra when orc-core fixes the violation.  

[jira] [Updated] (SPARK-44032) Remove the bytecode version exclusion of threeten-extra when orc-core fixes the violation

2023-06-12 Thread Bowen Liang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Liang updated SPARK-44032: Description: Remove the exclusion of threeten-extra when orc-core fixes the violation.  

[jira] [Created] (SPARK-44032) Remove the bytecode version exclusion of threeten-extra when orc-core fixes the violation

2023-06-12 Thread Bowen Liang (Jira)
Bowen Liang created SPARK-44032: --- Summary: Remove the bytecode version exclusion of threeten-extra when orc-core fixes the violation Key: SPARK-44032 URL: https://issues.apache.org/jira/browse/SPARK-44032

[jira] [Resolved] (SPARK-44023) Add System.gc at beforeEach in PruneFileSourcePartitionsSuite

2023-06-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44023. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41533

[jira] [Assigned] (SPARK-44023) Add System.gc at beforeEach in PruneFileSourcePartitionsSuite

2023-06-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-44023: Assignee: Yang Jie > Add System.gc at beforeEach in PruneFileSourcePartitionsSuite >

[jira] [Resolved] (SPARK-44029) Observation.py supports streaming dataframes

2023-06-12 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu resolved SPARK-44029. - Resolution: Not A Problem > Observation.py supports streaming dataframes >

[jira] [Resolved] (SPARK-42298) Assign name to _LEGACY_ERROR_TEMP_2132

2023-06-12 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-42298. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40632

[jira] [Assigned] (SPARK-42298) Assign name to _LEGACY_ERROR_TEMP_2132

2023-06-12 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-42298: Assignee: Jia Fan > Assign name to _LEGACY_ERROR_TEMP_2132 >

[jira] [Created] (SPARK-44031) Upgrade silencer to 1.7.13

2023-06-12 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-44031: - Summary: Upgrade silencer to 1.7.13 Key: SPARK-44031 URL: https://issues.apache.org/jira/browse/SPARK-44031 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-44030) Move out json and expression private methods from DataType

2023-06-12 Thread Rui Wang (Jira)
Rui Wang created SPARK-44030: Summary: Move out json and expression private methods from DataType Key: SPARK-44030 URL: https://issues.apache.org/jira/browse/SPARK-44030 Project: Spark Issue

[jira] [Comment Edited] (SPARK-43511) Implemented State APIs for Spark Connect Scala

2023-06-12 Thread Bo Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17722891#comment-17722891 ] Bo Gao edited comment on SPARK-43511 at 6/12/23 6:59 PM: - Created PR

[jira] [Created] (SPARK-44029) Observation.py supports streaming dataframes

2023-06-12 Thread Wei Liu (Jira)
Wei Liu created SPARK-44029: --- Summary: Observation.py supports streaming dataframes Key: SPARK-44029 URL: https://issues.apache.org/jira/browse/SPARK-44029 Project: Spark Issue Type: Documentation

[jira] [Created] (SPARK-44028) Upgrade commons-io to 2.13.0

2023-06-12 Thread Yang Jie (Jira)
Yang Jie created SPARK-44028: Summary: Upgrade commons-io to 2.13.0 Key: SPARK-44028 URL: https://issues.apache.org/jira/browse/SPARK-44028 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-44027) create *permanent* Spark View from DataFrame via PySpark & Scala DataFrame API

2023-06-12 Thread Martin Bode (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Bode updated SPARK-44027: Description: currently only *_temporary_ Spark Views* can be created from a DataFrame: *

[jira] [Updated] (SPARK-44027) create *permanent* Spark View from DataFrame via PySpark & Scala DataFrame API

2023-06-12 Thread Martin Bode (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Bode updated SPARK-44027: Affects Version/s: 3.5.0 (was: 3.4.0) > create *permanent* Spark View

[jira] [Updated] (SPARK-44027) create *permanent* Spark View from DataFrame via PySpark & Scala DataFrame API

2023-06-12 Thread Martin Bode (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Bode updated SPARK-44027: Description: currently only *_temporary_ Spark Views* can be created from a DataFrame: *

[jira] [Created] (SPARK-44027) create *permanent* Spark View from DataFrame via PySpark & Scala DataFrame API

2023-06-12 Thread Martin Bode (Jira)
Martin Bode created SPARK-44027: --- Summary: create *permanent* Spark View from DataFrame via PySpark & Scala DataFrame API Key: SPARK-44027 URL: https://issues.apache.org/jira/browse/SPARK-44027

[jira] [Commented] (SPARK-44018) Improve the hashCode for Some DS V2 Expression

2023-06-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17731695#comment-17731695 ] Dongjoon Hyun commented on SPARK-44018: --- Hi, [~beliefer]. Thank you for reporting. BTW, it seems

[jira] [Updated] (SPARK-43744) Spark Connect scala UDF serialization pulling in unrelated classes not available on server

2023-06-12 Thread Juliusz Sompolski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juliusz Sompolski updated SPARK-43744: -- Summary: Spark Connect scala UDF serialization pulling in unrelated classes not

[jira] [Updated] (SPARK-43744) Maven test failure when "interrupt all" tests are moved to ClientE2ETestSuite

2023-06-12 Thread Juliusz Sompolski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juliusz Sompolski updated SPARK-43744: -- Description: [https://github.com/apache/spark/pull/41487] moved "interrupt all -

[jira] [Reopened] (SPARK-43744) Maven test failure of ClientE2ETestSuite "interrupt all" tests

2023-06-12 Thread Juliusz Sompolski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juliusz Sompolski reopened SPARK-43744: --- [https://github.com/apache/spark/pull/41487] has swept the issue under the rug by

[jira] [Updated] (SPARK-43744) Maven test failure when "interrupt all" tests are moved to ClientE2ETestSuite

2023-06-12 Thread Juliusz Sompolski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juliusz Sompolski updated SPARK-43744: -- Summary: Maven test failure when "interrupt all" tests are moved to

[jira] [Created] (SPARK-44026) Update helper methods to create SQLMetric with initial value

2023-06-12 Thread Johan Lasperas (Jira)
Johan Lasperas created SPARK-44026: -- Summary: Update helper methods to create SQLMetric with initial value Key: SPARK-44026 URL: https://issues.apache.org/jira/browse/SPARK-44026 Project: Spark

[jira] [Updated] (SPARK-43781) IllegalStateException when cogrouping two datasets derived from the same source

2023-06-12 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-43781: Affects Version/s: 3.4.0 > IllegalStateException when cogrouping two datasets derived from the same >

[jira] [Updated] (SPARK-43915) Assign names to the error class _LEGACY_ERROR_TEMP_[2438-2445]

2023-06-12 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-43915: --- Summary: Assign names to the error class _LEGACY_ERROR_TEMP_[2438-2445] (was: Assign a name to the

[jira] [Commented] (SPARK-38200) [SQL] Spark JDBC Savemode Supports Upsert

2023-06-12 Thread Enrico Minack (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17731503#comment-17731503 ] Enrico Minack commented on SPARK-38200: --- Related: SPARK-19335 > [SQL] Spark JDBC Savemode

[jira] [Commented] (SPARK-38200) [SQL] Spark JDBC Savemode Supports Upsert

2023-06-12 Thread Enrico Minack (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17731502#comment-17731502 ] Enrico Minack commented on SPARK-38200: --- Sadly, MERGE is shown to perform worse than UPDATE+INSERT

[jira] [Resolved] (SPARK-43398) Executor timeout should be max of idleTimeout rddTimeout shuffleTimeout

2023-06-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-43398. --- Fix Version/s: 3.3.3 3.5.0 3.4.1 Resolution:

[jira] [Assigned] (SPARK-43398) Executor timeout should be max of idleTimeout rddTimeout shuffleTimeout

2023-06-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-43398: - Assignee: Zhongwei Zhu > Executor timeout should be max of idleTimeout rddTimeout

[jira] [Resolved] (SPARK-43617) Enable pyspark.pandas.spark.functions.product in Spark Connect.

2023-06-12 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-43617. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41547