[jira] [Assigned] (SPARK-48704) Update `build_sparkr_window.yml` to use `windows-2022`

2024-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48704: Assignee: BingKun Pan > Update `build_sparkr_window.yml` to use `windows-2022` >

[jira] [Resolved] (SPARK-48704) Update `build_sparkr_window.yml` to use `windows-2022`

2024-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48704. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47076 [https://gi

[jira] [Resolved] (SPARK-48629) Migrate the remaining code to structured logging framework

2024-06-24 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-48629. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46986 [https:

[jira] [Created] (SPARK-48707) RocksDB setLevelCompactionDynamicLevelBytes should be set to false

2024-06-24 Thread Neil Ramaswamy (Jira)
Neil Ramaswamy created SPARK-48707: -- Summary: RocksDB setLevelCompactionDynamicLevelBytes should be set to false Key: SPARK-48707 URL: https://issues.apache.org/jira/browse/SPARK-48707 Project: Spark

[jira] [Resolved] (SPARK-48692) Upgrade `rocksdbjni` to 9.2.1

2024-06-24 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-48692. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46146 [https://github.com

[jira] [Updated] (SPARK-48706) Python UDF in higher order functions should not throw internal error

2024-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-48706: - Description: {code} from pyspark.sql.functions import transform, udf, col, array spark.range(1).

[jira] [Created] (SPARK-48706) Python UDF in higher order functions should not throw internal error

2024-06-24 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-48706: Summary: Python UDF in higher order functions should not throw internal error Key: SPARK-48706 URL: https://issues.apache.org/jira/browse/SPARK-48706 Project: Spark

[jira] [Created] (SPARK-48705) Explicitly use worker_main when it starts with pyspark

2024-06-24 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-48705: Summary: Explicitly use worker_main when it starts with pyspark Key: SPARK-48705 URL: https://issues.apache.org/jira/browse/SPARK-48705 Project: Spark Issue

[jira] [Created] (SPARK-48704) Update `build_sparkr_window.yml` to use `windows-2022`

2024-06-24 Thread BingKun Pan (Jira)
BingKun Pan created SPARK-48704: --- Summary: Update `build_sparkr_window.yml` to use `windows-2022` Key: SPARK-48704 URL: https://issues.apache.org/jira/browse/SPARK-48704 Project: Spark Issue Ty

[jira] [Assigned] (SPARK-48702) Fix `Python CodeGen check`

2024-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48702: Assignee: BingKun Pan > Fix `Python CodeGen check` > -- > >

[jira] [Resolved] (SPARK-48702) Fix `Python CodeGen check`

2024-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48702. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47074 [https://gi

[jira] [Created] (SPARK-48703) Upgrade `mssql-jdbc` to 12.6.3.jre11

2024-06-24 Thread BingKun Pan (Jira)
BingKun Pan created SPARK-48703: --- Summary: Upgrade `mssql-jdbc` to 12.6.3.jre11 Key: SPARK-48703 URL: https://issues.apache.org/jira/browse/SPARK-48703 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-48702) Fix `Python CodeGen check`

2024-06-24 Thread BingKun Pan (Jira)
BingKun Pan created SPARK-48702: --- Summary: Fix `Python CodeGen check` Key: SPARK-48702 URL: https://issues.apache.org/jira/browse/SPARK-48702 Project: Spark Issue Type: Improvement Co

[jira] [Updated] (SPARK-47046) Apache Spark 4.0.0 Dependency Audit and Cleanup

2024-06-24 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47046: -- Labels: releasenotes (was: pull-request-available releasenotes) > Apache Spark 4.0.0 Dependen

[jira] [Created] (SPARK-48701) PandasMode (all collations)

2024-06-24 Thread Jira
Uroš Bojanić created SPARK-48701: Summary: PandasMode (all collations) Key: SPARK-48701 URL: https://issues.apache.org/jira/browse/SPARK-48701 Project: Spark Issue Type: Sub-task Co

[jira] [Created] (SPARK-48700) Mode expression for complex types (all collations)

2024-06-24 Thread Jira
Uroš Bojanić created SPARK-48700: Summary: Mode expression for complex types (all collations) Key: SPARK-48700 URL: https://issues.apache.org/jira/browse/SPARK-48700 Project: Spark Issue Type

[jira] [Updated] (SPARK-47353) Mode expression for strings (all collations)

2024-06-24 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-47353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uroš Bojanić updated SPARK-47353: - Summary: Mode expression for strings (all collations) (was: Mode (all collations)) > Mode expr

[jira] [Updated] (SPARK-39627) DS V2 pushdown should unify the compile API

2024-06-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-39627: --- Labels: pull-request-available (was: ) > DS V2 pushdown should unify the compile API >

[jira] [Created] (SPARK-48699) Refine collation API

2024-06-24 Thread Jira
Uroš Bojanić created SPARK-48699: Summary: Refine collation API Key: SPARK-48699 URL: https://issues.apache.org/jira/browse/SPARK-48699 Project: Spark Issue Type: Sub-task Component

[jira] [Updated] (SPARK-48670) Providing suggestion as part of error message when invalid collation name is given

2024-06-24 Thread Aleksandar Tomic (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksandar Tomic updated SPARK-48670: - Summary: Providing suggestion as part of error message when invalid collation name is gi

[jira] [Updated] (SPARK-18523) OOM killer may leave SparkContext in broken state causing Connection Refused errors

2024-06-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-18523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-18523: --- Labels: pull-request-available (was: ) > OOM killer may leave SparkContext in broken state

[jira] [Updated] (SPARK-48698) Support analyze column stats for tables with collated columns

2024-06-24 Thread Nikola Mandic (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikola Mandic updated SPARK-48698: -- Epic Link: SPARK-46830 > Support analyze column stats for tables with collated columns > -

[jira] [Created] (SPARK-48698) Support analyze column stats for tables with collated columns

2024-06-24 Thread Nikola Mandic (Jira)
Nikola Mandic created SPARK-48698: - Summary: Support analyze column stats for tables with collated columns Key: SPARK-48698 URL: https://issues.apache.org/jira/browse/SPARK-48698 Project: Spark

[jira] [Resolved] (SPARK-48658) Encode/Decode functions report coding error instead of mojibake

2024-06-24 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-48658. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47017 [https://github.com

[jira] [Assigned] (SPARK-48658) Encode/Decode functions report coding error instead of mojibake

2024-06-24 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-48658: Assignee: Kent Yao > Encode/Decode functions report coding error instead of mojibake > --

[jira] [Assigned] (SPARK-48695) TimestampNTZType.fromInternal not use the deprecated methods

2024-06-24 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-48695: - Assignee: Ruifeng Zheng > TimestampNTZType.fromInternal not use the deprecated methods

[jira] [Resolved] (SPARK-48695) TimestampNTZType.fromInternal not use the deprecated methods

2024-06-24 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-48695. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47068 [https://

[jira] [Updated] (SPARK-48697) Fix consumption of filters that have widened parts of the predicate to be AlwaysTrue

2024-06-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48697: --- Labels: pull-request-available (was: ) > Fix consumption of filters that have widened parts

[jira] [Created] (SPARK-48697) Fix consumption of filters that have widened parts of the predicate to be AlwaysTrue

2024-06-24 Thread Stefan Kandic (Jira)
Stefan Kandic created SPARK-48697: - Summary: Fix consumption of filters that have widened parts of the predicate to be AlwaysTrue Key: SPARK-48697 URL: https://issues.apache.org/jira/browse/SPARK-48697

[jira] [Created] (SPARK-48696) Also truncate the schema row for show function

2024-06-24 Thread Kent Yao (Jira)
Kent Yao created SPARK-48696: Summary: Also truncate the schema row for show function Key: SPARK-48696 URL: https://issues.apache.org/jira/browse/SPARK-48696 Project: Spark Issue Type: Improvemen

[jira] [Updated] (SPARK-48695) TimestampNTZType.fromInternal not use the deprecated methods

2024-06-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48695: --- Labels: pull-request-available (was: ) > TimestampNTZType.fromInternal not use the deprecat

[jira] [Created] (SPARK-48695) TimestampNTZType.fromInternal not use the deprecated methods

2024-06-24 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-48695: - Summary: TimestampNTZType.fromInternal not use the deprecated methods Key: SPARK-48695 URL: https://issues.apache.org/jira/browse/SPARK-48695 Project: Spark

[jira] [Updated] (SPARK-48691) Upgrade `scalatest` related dependencies to the 3.2.18 series

2024-06-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48691: --- Labels: pull-request-available (was: ) > Upgrade `scalatest` related dependencies to the 3.

[jira] [Updated] (SPARK-48639) Add Origin to RelationCommon in protobuf defnition

2024-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-48639: - Fix Version/s: (was: 3.5.2) > Add Origin to RelationCommon in protobuf defnition > -

[jira] [Resolved] (SPARK-48639) Add Origin to RelationCommon in protobuf defnition

2024-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48639. -- Fix Version/s: 3.5.2 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-48639) Add Origin to RelationCommon in protobuf defnition

2024-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48639: Assignee: Hyukjin Kwon > Add Origin to RelationCommon in protobuf defnition > ---

[jira] [Updated] (SPARK-48639) Add Origin to RelationCommon in protobuf defnition

2024-06-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48639: --- Labels: pull-request-available (was: ) > Add Origin to RelationCommon in protobuf defnition

[jira] [Assigned] (SPARK-48658) Encode/Decode functions report coding error instead of mojibake

2024-06-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-48658: -- Assignee: (was: Apache Spark) > Encode/Decode functions report coding error inste

[jira] [Assigned] (SPARK-48658) Encode/Decode functions report coding error instead of mojibake

2024-06-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-48658: -- Assignee: Apache Spark > Encode/Decode functions report coding error instead of mojib

[jira] [Updated] (SPARK-42495) Scala Client: Add 2nd batch of functions

2024-06-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-42495: --- Labels: pull-request-available (was: ) > Scala Client: Add 2nd batch of functions > ---

[jira] [Updated] (SPARK-39041) Mapping Spark Query ResultSet/Schema to TRowSet/TTableSchema directly

2024-06-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-39041: --- Labels: pull-request-available (was: ) > Mapping Spark Query ResultSet/Schema to TRowSet/TT

[jira] [Updated] (SPARK-48658) Encode/Decode functions report coding error instead of mojibake

2024-06-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48658: --- Labels: pull-request-available (was: ) > Encode/Decode functions report coding error instea

[jira] [Resolved] (SPARK-48683) Schema evolution with `df.mergeInto` losing `when` clauses

2024-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48683. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47055 [https://gi

[jira] [Assigned] (SPARK-48683) Schema evolution with `df.mergeInto` losing `when` clauses

2024-06-24 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48683: Assignee: Pengfei Xu > Schema evolution with `df.mergeInto` losing `when` clauses > -

[jira] [Resolved] (SPARK-48680) Add char/varchar doc to language specific tables

2024-06-24 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-48680. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47052 [https://github.com

[jira] [Updated] (SPARK-48683) Schema evolution with `df.mergeInto` losing `when` clauses

2024-06-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48683: --- Labels: pull-request-available (was: ) > Schema evolution with `df.mergeInto` losing `when`

[jira] [Resolved] (SPARK-48681) Use ICU in Lower/Upper expressions (UTF8_BINARY collation)

2024-06-24 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48681. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47043 [https://gith

[jira] [Updated] (SPARK-48687) Add changes to implement state schema validation in planning phase on driver for stateful streaming queries

2024-06-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48687: --- Labels: pull-request-available (was: ) > Add changes to implement state schema validation i

[jira] [Updated] (SPARK-48694) Manage memory used by external cache

2024-06-24 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48694: --- Labels: pull-request-available (was: ) > Manage memory used by external cache > ---

[jira] [Updated] (SPARK-48694) Manage memory used by external cache

2024-06-24 Thread Yan Ma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yan Ma updated SPARK-48694: --- Environment: (was: We have a scenario that use Spark together with a 3rd party file source cache, which

[jira] [Updated] (SPARK-48694) Manage memory used by external cache

2024-06-24 Thread Yan Ma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yan Ma updated SPARK-48694: --- Description: We have a scenario that use Spark together with a 3rd party file source cache, which is an ind

[jira] [Created] (SPARK-48694) Manage memory used by external cache

2024-06-24 Thread Yan Ma (Jira)
Yan Ma created SPARK-48694: -- Summary: Manage memory used by external cache Key: SPARK-48694 URL: https://issues.apache.org/jira/browse/SPARK-48694 Project: Spark Issue Type: Improvement Co