[jira] [Updated] (SPARK-49773) uncaught Java exception from make_timestamp() with bad timezone

2024-09-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49773: --- Labels: pull-request-available (was: ) > uncaught Java exception from make_timestamp() with

[jira] [Resolved] (SPARK-49786) Lower `KubernetesClusterSchedulerBackend.onDisconnected` log level to debug

2024-09-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-49786. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48249 [https://

[jira] [Updated] (SPARK-49776) Support pie plots

2024-09-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49776: --- Labels: pull-request-available (was: ) > Support pie plots > - > >

[jira] [Updated] (SPARK-49792) Upgrade to numpy 2 for building and testing Spark branches

2024-09-25 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-49792: - Summary: Upgrade to numpy 2 for building and testing Spark branches (was: Upgrade NumPy to 2.1.

[jira] [Updated] (SPARK-49792) Upgrade NumPy to 2.1.0

2024-09-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49792: --- Labels: pull-request-available (was: ) > Upgrade NumPy to 2.1.0 > -- >

[jira] [Updated] (SPARK-49793) Enable PredictBatchUDFTests.test_caching for NumPy 2

2024-09-25 Thread Xinrong Meng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-49793: - Description:   {code:java} import numpy as np import pandas as pd from pyspark.ml.functions impo

[jira] [Created] (SPARK-49793) Enable PredictBatchUDFTests.test_caching for NumPy 2

2024-09-25 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-49793: Summary: Enable PredictBatchUDFTests.test_caching for NumPy 2 Key: SPARK-49793 URL: https://issues.apache.org/jira/browse/SPARK-49793 Project: Spark Issue Ty

[jira] [Updated] (SPARK-49791) Make DelegatingCatalogExtension more extendable

2024-09-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49791: --- Labels: pull-request-available (was: ) > Make DelegatingCatalogExtension more extendable >

[jira] [Created] (SPARK-49791) Make DelegatingCatalogExtension more extendable

2024-09-25 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-49791: --- Summary: Make DelegatingCatalogExtension more extendable Key: SPARK-49791 URL: https://issues.apache.org/jira/browse/SPARK-49791 Project: Spark Issue Type: Imp

[jira] [Resolved] (SPARK-49790) Support `HPA` template for `SparkCluster`

2024-09-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-49790. --- Fix Version/s: kubernetes-operator-0.1.0 Resolution: Fixed Issue resolved by pull req

[jira] [Updated] (SPARK-49744) [PySpark] State TTL support - ListState

2024-09-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49744: --- Labels: pull-request-available (was: ) > [PySpark] State TTL support - ListState >

[jira] [Assigned] (SPARK-49790) Support `HPA` template for `SparkCluster`

2024-09-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-49790: - Assignee: Dongjoon Hyun > Support `HPA` template for `SparkCluster` > -

[jira] [Updated] (SPARK-49790) Support `HPA` template for `SparkCluster`

2024-09-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49790: --- Labels: pull-request-available (was: ) > Support `HPA` template for `SparkCluster` > --

[jira] [Created] (SPARK-49790) Support `HPA` template for `SparkCluster`

2024-09-25 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-49790: - Summary: Support `HPA` template for `SparkCluster` Key: SPARK-49790 URL: https://issues.apache.org/jira/browse/SPARK-49790 Project: Spark Issue Type: Sub-t

[jira] [Updated] (SPARK-49789) org.apache.spark.SparkUnsupportedOperationException: [ENCODER_NOT_FOUND] Not found an encoder of the type T

2024-09-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49789: --- Labels: pull-request-available spark-sql (was: spark-sql) > org.apache.spark.SparkUnsupport

[jira] [Commented] (SPARK-49788) Add spark.cleaner.ttl functionality for long lived jobs

2024-09-25 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17884785#comment-17884785 ] Holden Karau commented on SPARK-49788: -- Rough thoughts on how we could make a new a

[jira] [Created] (SPARK-49789) org.apache.spark.SparkUnsupportedOperationException: [ENCODER_NOT_FOUND] Not found an encoder of the type T

2024-09-25 Thread Asif (Jira)
Asif created SPARK-49789: Summary: org.apache.spark.SparkUnsupportedOperationException: [ENCODER_NOT_FOUND] Not found an encoder of the type T Key: SPARK-49789 URL: https://issues.apache.org/jira/browse/SPARK-49789

[jira] [Commented] (SPARK-46679) Encoders with multiple inheritance - Key not found: T

2024-09-25 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17884769#comment-17884769 ] Asif commented on SPARK-46679: -- [~andoni.teso] It has been brought to my notice by my colle

[jira] [Commented] (SPARK-49788) Add spark.cleaner.ttl functionality for long lived jobs

2024-09-25 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17884768#comment-17884768 ] Holden Karau commented on SPARK-49788: -- Original functionality was removed in 8e19c

[jira] [Updated] (SPARK-49787) Cast between UDT and other types

2024-09-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49787: --- Labels: pull-request-available (was: ) > Cast between UDT and other types > ---

[jira] [Commented] (SPARK-49788) Add spark.cleaner.ttl functionality for long lived jobs

2024-09-25 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17884760#comment-17884760 ] Holden Karau commented on SPARK-49788: -- CC [~joshrosen] for thoughts. > Add spark.

[jira] [Updated] (SPARK-49788) Add spark.cleaner.ttl functionality for long lived jobs

2024-09-25 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-49788: - Description: We should add a TTL (and maybe a threshold?) to clean shuffle files which have bee

[jira] [Assigned] (SPARK-49787) Cast between UDT and other types

2024-09-25 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-49787: --- Assignee: L. C. Hsieh > Cast between UDT and other types >

[jira] [Created] (SPARK-49788) Add spark.cleaner.ttl functionality for long lived jobs

2024-09-25 Thread Holden Karau (Jira)
Holden Karau created SPARK-49788: Summary: Add spark.cleaner.ttl functionality for long lived jobs Key: SPARK-49788 URL: https://issues.apache.org/jira/browse/SPARK-49788 Project: Spark Issue

[jira] [Created] (SPARK-49787) Cast between UDT and other types

2024-09-25 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-49787: --- Summary: Cast between UDT and other types Key: SPARK-49787 URL: https://issues.apache.org/jira/browse/SPARK-49787 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-49770) RocksDB Hardening: Improve SST file mapping management

2024-09-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49770: --- Labels: pull-request-available (was: ) > RocksDB Hardening: Improve SST file mapping manage

[jira] [Assigned] (SPARK-49786) Lower `KubernetesClusterSchedulerBackend.onDisconnected` log level to debug

2024-09-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-49786: - Assignee: Dongjoon Hyun > Lower `KubernetesClusterSchedulerBackend.onDisconnected` log

[jira] [Updated] (SPARK-49786) Lower `KubernetesClusterSchedulerBackend.onDisconnected` log level to debug

2024-09-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49786: --- Labels: pull-request-available (was: ) > Lower `KubernetesClusterSchedulerBackend.onDisconn

[jira] [Created] (SPARK-49786) Lower `KubernetesClusterSchedulerBackend.onDisconnected` log level to debug

2024-09-25 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-49786: - Summary: Lower `KubernetesClusterSchedulerBackend.onDisconnected` log level to debug Key: SPARK-49786 URL: https://issues.apache.org/jira/browse/SPARK-49786 Project

[jira] [Updated] (SPARK-49731) Support K8s volume `mount.subPathExpr` and `hostPath` volume `type`

2024-09-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-49731: -- Parent: SPARK-49524 Issue Type: Sub-task (was: New Feature) > Support K8s volume `mou

[jira] [Resolved] (SPARK-49731) Support Kubernetes subPathExpr and hostPath volume type options

2024-09-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-49731. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48181 [https://

[jira] [Assigned] (SPARK-49731) Support Kubernetes subPathExpr and hostPath volume type options

2024-09-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-49731: - Assignee: Enrico Minack > Support Kubernetes subPathExpr and hostPath volume type optio

[jira] [Updated] (SPARK-49731) Support K8s volume `mount.subPathExpr` and `hostPath` volume `type`

2024-09-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-49731: -- Summary: Support K8s volume `mount.subPathExpr` and `hostPath` volume `type` (was: Support Ku

[jira] [Resolved] (SPARK-49746) Upgrade Scala to 2.13.15

2024-09-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-49746. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48192 [https://

[jira] [Created] (SPARK-49785) Spark 3.5.3 does not compile

2024-09-25 Thread lione Herbet (Jira)
lione Herbet created SPARK-49785: Summary: Spark 3.5.3 does not compile Key: SPARK-49785 URL: https://issues.apache.org/jira/browse/SPARK-49785 Project: Spark Issue Type: Bug Compon

[jira] [Resolved] (SPARK-49745) Add support for reading timer values using state data source reader and transformWithState

2024-09-25 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-49745. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48205 [https://gi

[jira] [Assigned] (SPARK-49745) Add support for reading timer values using state data source reader and transformWithState

2024-09-25 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-49745: Assignee: Anish Shrigondekar > Add support for reading timer values using state data sour

[jira] [Updated] (SPARK-49784) Add more test for spark.sql

2024-09-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49784: --- Labels: pull-request-available (was: ) > Add more test for spark.sql >

[jira] [Created] (SPARK-49784) Add more test for spark.sql

2024-09-25 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-49784: - Summary: Add more test for spark.sql Key: SPARK-49784 URL: https://issues.apache.org/jira/browse/SPARK-49784 Project: Spark Issue Type: Test Comp

[jira] [Updated] (SPARK-49768) Provide error Conditions for make_date/make_timestamp errors _LEGACY_ERROR_TEMP_2000

2024-09-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49768: --- Labels: pull-request-available (was: ) > Provide error Conditions for make_date/make_timest

[jira] [Resolved] (SPARK-49775) Make tests of INVALID_PARAMETER_VALUE.CHARSET deterministic

2024-09-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-49775. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48235 [https://gi

[jira] [Resolved] (SPARK-49767) Refactor the internal function invocation

2024-09-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-49767. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48227 [https://gi

[jira] [Assigned] (SPARK-49767) Refactor the internal function invocation

2024-09-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-49767: Assignee: Ruifeng Zheng > Refactor the internal function invocation > ---

[jira] [Assigned] (SPARK-49764) Support area plots

2024-09-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-49764: Assignee: Xinrong Meng > Support area plots > -- > > Key:

[jira] [Resolved] (SPARK-49764) Support area plots

2024-09-25 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-49764. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48236 [https://gi

[jira] [Updated] (SPARK-49782) ResolveDataFrameDropColumns rule mistakenly handles UnresolvedAttribute

2024-09-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49782: --- Labels: pull-request-available (was: ) > ResolveDataFrameDropColumns rule mistakenly handle

[jira] [Assigned] (SPARK-49666) Put trim sensitivity under a flag

2024-09-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-49666: -- Assignee: (was: Apache Spark) > Put trim sensitivity under a flag > -

[jira] [Assigned] (SPARK-49666) Put trim sensitivity under a flag

2024-09-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-49666: -- Assignee: Apache Spark > Put trim sensitivity under a flag >

[jira] [Updated] (SPARK-49783) Resource leak of spark yarn allocator

2024-09-25 Thread Junfan Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junfan Zhang updated SPARK-49783: - Description: When the target < running containers number, the assigned containers from the reso

[jira] [Updated] (SPARK-49783) Resource leak of spark yarn allocator

2024-09-25 Thread Junfan Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junfan Zhang updated SPARK-49783: - Description: When the target < running containers number, the assigned containers from the reso

[jira] [Updated] (SPARK-49783) Resource leak of spark yarn allocator

2024-09-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49783: --- Labels: pull-request-available (was: ) > Resource leak of spark yarn allocator > --

[jira] [Created] (SPARK-49783) Resource leak of spark yarn allocator

2024-09-25 Thread Junfan Zhang (Jira)
Junfan Zhang created SPARK-49783: Summary: Resource leak of spark yarn allocator Key: SPARK-49783 URL: https://issues.apache.org/jira/browse/SPARK-49783 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-49782) ResolveDataFrameDropColumns rule mistakenly handles UnresolvedAttribute

2024-09-25 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-49782: -- Summary: ResolveDataFrameDropColumns rule mistakenly handles UnresolvedAttribute Key: SPARK-49782 URL: https://issues.apache.org/jira/browse/SPARK-49782 Project: Spark

[jira] [Updated] (SPARK-49781) Optimize size of `MapFromArrays` / `MapFromEntries` to be the size of its children

2024-09-25 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49781: --- Labels: pull-request-available (was: ) > Optimize size of `MapFromArrays` / `MapFromEntries

[jira] [Created] (SPARK-49781) Optimize size of `MapFromArrays` / `MapFromEntries` to be the size of its children

2024-09-25 Thread BingKun Pan (Jira)
BingKun Pan created SPARK-49781: --- Summary: Optimize size of `MapFromArrays` / `MapFromEntries` to be the size of its children Key: SPARK-49781 URL: https://issues.apache.org/jira/browse/SPARK-49781 Pro

[jira] [Resolved] (SPARK-49778) Remove (master|worker) prefix from field names of `(Master|Worker)Spec`

2024-09-25 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-49778. --- Fix Version/s: kubernetes-operator-0.1.0 Resolution: Fixed Issue resolved by pull req