[jira] [Updated] (SPARK-50379) Fix DayTimeIntevalType handling in WindowExecBase

[ https://issues.apache.org/jira/browse/SPARK-50379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50379: --- Labels: pull-request-available (was: ) > Fix DayTimeIntevalType handling in WindowExecBase

[jira] [Commented] (SPARK-50379) Fix DayTimeIntevalType handling in WindowExecBase

[ https://issues.apache.org/jira/browse/SPARK-50379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17899946#comment-17899946 ] Mihailo Milosevic commented on SPARK-50379: --- Working on this. > Fix DayTimeIn

[jira] [Updated] (SPARK-50379) Fix DayTimeIntevalType handling in WindowExecBase

[ https://issues.apache.org/jira/browse/SPARK-50379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mihailo Milosevic updated SPARK-50379: -- Description: It was noticed that we cannot add INTERVAL DAY as range type to DATE colu

[jira] [Created] (SPARK-50379) Fix DayTimeIntevalType handling in WindowExecBase

Mihailo Milosevic created SPARK-50379: - Summary: Fix DayTimeIntevalType handling in WindowExecBase Key: SPARK-50379 URL: https://issues.apache.org/jira/browse/SPARK-50379 Project: Spark I

[jira] [Updated] (SPARK-50133) Support conversion to table argument

[ https://issues.apache.org/jira/browse/SPARK-50133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50133: --- Labels: pull-request-available (was: ) > Support conversion to table argument > ---

[jira] [Assigned] (SPARK-50376) Centralize the dependency check in ML tests

[ https://issues.apache.org/jira/browse/SPARK-50376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-50376: Assignee: Ruifeng Zheng > Centralize the dependency check in ML tests > -

[jira] [Created] (SPARK-50378) Add initialState metric for transformWithState

Anish Shrigondekar created SPARK-50378: -- Summary: Add initialState metric for transformWithState Key: SPARK-50378 URL: https://issues.apache.org/jira/browse/SPARK-50378 Project: Spark Is

[jira] [Created] (SPARK-50377) Allow to evaluate foldable RuntimeReplaceable

Wenchen Fan created SPARK-50377: --- Summary: Allow to evaluate foldable RuntimeReplaceable Key: SPARK-50377 URL: https://issues.apache.org/jira/browse/SPARK-50377 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-50376) Centralize the dependency check in ML tests

[ https://issues.apache.org/jira/browse/SPARK-50376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50376: --- Labels: pull-request-available (was: ) > Centralize the dependency check in ML tests >

[jira] [Created] (SPARK-50376) Centralize the dependency check in ML tests

Ruifeng Zheng created SPARK-50376: - Summary: Centralize the dependency check in ML tests Key: SPARK-50376 URL: https://issues.apache.org/jira/browse/SPARK-50376 Project: Spark Issue Type: Imp

[jira] [Updated] (SPARK-50258) Fix output column order changed issue after AQE optimization

[ https://issues.apache.org/jira/browse/SPARK-50258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-50258: Summary: Fix output column order changed issue after AQE optimization (was: Keep the output order

[jira] [Updated] (SPARK-41344) Reading V2 datasource masks underlying error

[ https://issues.apache.org/jira/browse/SPARK-41344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-41344: --- Labels: pull-request-available (was: ) > Reading V2 datasource masks underlying error > ---

[jira] [Updated] (SPARK-47193) Converting dataframe to rdd results in data loss

[ https://issues.apache.org/jira/browse/SPARK-47193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47193: --- Labels: correctness pull-request-available (was: correctness) > Converting dataframe to rdd

[jira] [Updated] (SPARK-50374) SubqueryBroadcast is not reused in the SparkPlanGraph

[ https://issues.apache.org/jira/browse/SPARK-50374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated SPARK-50374: -- Attachment: spark-ui-expected-subquery-broadcast-2024-11-20.png > SubqueryBroadcast is not reu

[jira] [Updated] (SPARK-50374) SubqueryBroadcast is not reused in the SparkPlanGraph

[ https://issues.apache.org/jira/browse/SPARK-50374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated SPARK-50374: -- Description: Currently, the SparkPlanGraph construction ignores that SubqueryBroadcast can be

[jira] [Updated] (SPARK-50374) SubqueryBroadcast is not reused in the SparkPlanGraph

[ https://issues.apache.org/jira/browse/SPARK-50374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated SPARK-50374: -- Attachment: (was: spark-ui-expected-subquery-broadcast-2024-11-20.png) > SubqueryBroadcast

[jira] [Updated] (SPARK-50374) SubqueryBroadcast is not reused in the SparkPlanGraph

[ https://issues.apache.org/jira/browse/SPARK-50374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated SPARK-50374: -- Attachment: spark-ui-expected-subquery-broadcast-2024-11-20.png > SubqueryBroadcast is not reu

[jira] [Updated] (SPARK-50374) SubqueryBroadcast is not reused in the SparkPlanGraph

[ https://issues.apache.org/jira/browse/SPARK-50374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated SPARK-50374: -- Attachment: spark-ui-invalid-subquery-broadcast-2024-11-20.png > SubqueryBroadcast is not reus

[jira] [Created] (SPARK-50374) SubqueryBroadcast is not reused in the SparkPlanGraph

Ahmed Hussein created SPARK-50374: - Summary: SubqueryBroadcast is not reused in the SparkPlanGraph Key: SPARK-50374 URL: https://issues.apache.org/jira/browse/SPARK-50374 Project: Spark Issue

[jira] [Updated] (SPARK-50373) Prohibit set operations on Variant type

[ https://issues.apache.org/jira/browse/SPARK-50373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50373: --- Labels: pull-request-available (was: ) > Prohibit set operations on Variant type >

[jira] [Updated] (SPARK-50258) Keep the output order after AQE optimization

[ https://issues.apache.org/jira/browse/SPARK-50258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-50258: -- Affects Version/s: 3.4.4 > Keep the output order after AQE optimization >

[jira] [Created] (SPARK-50373) Prohibit set operations on Variant type

Harsh Motwani created SPARK-50373: - Summary: Prohibit set operations on Variant type Key: SPARK-50373 URL: https://issues.apache.org/jira/browse/SPARK-50373 Project: Spark Issue Type: Task

[jira] [Assigned] (SPARK-50364) Row.jsonValue is not able to serialize LocalDateTime

[ https://issues.apache.org/jira/browse/SPARK-50364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-50364: Assignee: Karim Ramadan > Row.jsonValue is not able to serialize LocalDateTime >

[jira] [Resolved] (SPARK-50364) Row.jsonValue is not able to serialize LocalDateTime

[ https://issues.apache.org/jira/browse/SPARK-50364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-50364. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48905 [https://github.com

[jira] [Resolved] (SPARK-50362) Skip CrossValidatorTests if `torch/torcheval` not installed

[ https://issues.apache.org/jira/browse/SPARK-50362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-50362. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48901 [https://

[jira] [Created] (SPARK-50372) Bugfix: not all DF execute paths notify Observation.get

Pengfei Xu created SPARK-50372: -- Summary: Bugfix: not all DF execute paths notify Observation.get Key: SPARK-50372 URL: https://issues.apache.org/jira/browse/SPARK-50372 Project: Spark Issue Typ

[jira] [Updated] (SPARK-50370) Codegen Support for `json_tuple`

[ https://issues.apache.org/jira/browse/SPARK-50370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50370: --- Labels: pull-request-available (was: ) > Codegen Support for `json_tuple` > ---

[jira] [Updated] (SPARK-50356) Skipped stages are inconsitent on UI

[ https://issues.apache.org/jira/browse/SPARK-50356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-50356: --- Description: The following snipett produces inconsitent UI state: {noformat} import scala.collection

[jira] [Assigned] (SPARK-50370) Codegen Support for `json_tuple`

[ https://issues.apache.org/jira/browse/SPARK-50370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pan Bingkun reassigned SPARK-50370: --- Assignee: Pan Bingkun > Codegen Support for `json_tuple` >

[jira] [Created] (SPARK-50371) Introduce a shared plan representation

Herman van Hövell created SPARK-50371: - Summary: Introduce a shared plan representation Key: SPARK-50371 URL: https://issues.apache.org/jira/browse/SPARK-50371 Project: Spark Issue Type:

[jira] [Created] (SPARK-50370) Codegen Support for `json_tuple`

Pan Bingkun created SPARK-50370: --- Summary: Codegen Support for `json_tuple` Key: SPARK-50370 URL: https://issues.apache.org/jira/browse/SPARK-50370 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-50368) Validate shading rules for connect client and connect server

Herman van Hövell created SPARK-50368: - Summary: Validate shading rules for connect client and connect server Key: SPARK-50368 URL: https://issues.apache.org/jira/browse/SPARK-50368 Project: Spark

[jira] [Commented] (SPARK-49710) Make sure internal/dev API is properly annotated in SparkSession

[ https://issues.apache.org/jira/browse/SPARK-49710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17899772#comment-17899772 ] Herman van Hövell commented on SPARK-49710: --- I think most of this is done. I n

[jira] [Created] (SPARK-50369) Document UDF classpath differences between Classic and Connect

Herman van Hövell created SPARK-50369: - Summary: Document UDF classpath differences between Classic and Connect Key: SPARK-50369 URL: https://issues.apache.org/jira/browse/SPARK-50369 Project: Spa

[jira] [Updated] (SPARK-49428) Rename/Reorg Connect Client Packages

[ https://issues.apache.org/jira/browse/SPARK-49428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell updated SPARK-49428: -- Description: Rename sql/connect/common to sql/connect/client Move connector/connect/c

[jira] [Updated] (SPARK-49428) Rename/Reorg Connect Client Packages

[ https://issues.apache.org/jira/browse/SPARK-49428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell updated SPARK-49428: -- Summary: Rename/Reorg Connect Client Packages (was: Move Connect Scala Client to conn

[jira] [Assigned] (SPARK-50258) Keep the output order after AQE optimization

[ https://issues.apache.org/jira/browse/SPARK-50258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-50258: --- Assignee: Yuming Wang > Keep the output order after AQE optimization >

[jira] [Resolved] (SPARK-50258) Keep the output order after AQE optimization

[ https://issues.apache.org/jira/browse/SPARK-50258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-50258. - Fix Version/s: 3.5.4 4.0.0 Resolution: Fixed Issue resolved by pull re

[jira] [Updated] (SPARK-50352) Refine docstrings for window/aggregation functions - part 3

[ https://issues.apache.org/jira/browse/SPARK-50352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50352: --- Labels: pull-request-available (was: ) > Refine docstrings for window/aggregation functions

[jira] [Updated] (SPARK-50362) Skip CrossValidatorTests if `torch/torcheval` not installed

[ https://issues.apache.org/jira/browse/SPARK-50362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50362: --- Labels: pull-request-available (was: ) > Skip CrossValidatorTests if `torch/torcheval` not

[jira] [Updated] (SPARK-50366) Isolate tags on thread level for SparkSession in Classic

[ https://issues.apache.org/jira/browse/SPARK-50366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50366: --- Labels: pull-request-available (was: ) > Isolate tags on thread level for SparkSession in C

[jira] [Updated] (SPARK-50366) Isolate tags on thread level for SparkSession in Classic

[ https://issues.apache.org/jira/browse/SPARK-50366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengfei Xu updated SPARK-50366: --- Description: Follow-up from the discussion in https://github.com/apache/spark/pull/47815#discussion_

[jira] [Created] (SPARK-50366) Isolate tags on thread level for SparkSession in Classic

Pengfei Xu created SPARK-50366: -- Summary: Isolate tags on thread level for SparkSession in Classic Key: SPARK-50366 URL: https://issues.apache.org/jira/browse/SPARK-50366 Project: Spark Issue Ty

[jira] [Resolved] (SPARK-50363) Refine the docstring for datetime functions - part 1

[ https://issues.apache.org/jira/browse/SPARK-50363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-50363. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48904 [https://gi

[jira] [Assigned] (SPARK-50363) Refine the docstring for datetime functions - part 1

[ https://issues.apache.org/jira/browse/SPARK-50363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-50363: Assignee: Ruifeng Zheng > Refine the docstring for datetime functions - part 1 >

[jira] [Updated] (SPARK-50363) Refine the docstring for datetime functions - part 1

[ https://issues.apache.org/jira/browse/SPARK-50363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50363: --- Labels: pull-request-available (was: ) > Refine the docstring for datetime functions - part

[jira] [Commented] (SPARK-50364) Row.jsonValue is not able to serialize LocalDateTime

[ https://issues.apache.org/jira/browse/SPARK-50364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17899720#comment-17899720 ] Karim Ramadan commented on SPARK-50364: --- Simple fix at [https://github.com/apache

[jira] [Updated] (SPARK-50364) Row.jsonValue is not able to serialize LocalDateTime

[ https://issues.apache.org/jira/browse/SPARK-50364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50364: --- Labels: pull-request-available (was: ) > Row.jsonValue is not able to serialize LocalDateTi

[jira] [Created] (SPARK-50364) Row.jsonValue is not able to serialize LocalDateTime

Karim Ramadan created SPARK-50364: - Summary: Row.jsonValue is not able to serialize LocalDateTime Key: SPARK-50364 URL: https://issues.apache.org/jira/browse/SPARK-50364 Project: Spark Issue

[jira] [Resolved] (SPARK-50365) Row.jsonValue is not able to serialize LocalDateTime

[ https://issues.apache.org/jira/browse/SPARK-50365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karim Ramadan resolved SPARK-50365. --- Resolution: Duplicate > Row.jsonValue is not able to serialize LocalDateTime > -

[jira] [Resolved] (SPARK-50365) Row.jsonValue is not able to serialize LocalDateTime

[ https://issues.apache.org/jira/browse/SPARK-50365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karim Ramadan resolved SPARK-50365. --- Resolution: Duplicate > Row.jsonValue is not able to serialize LocalDateTime > -

[jira] [Created] (SPARK-50365) Row.jsonValue is not able to serialize LocalDateTime

Karim Ramadan created SPARK-50365: - Summary: Row.jsonValue is not able to serialize LocalDateTime Key: SPARK-50365 URL: https://issues.apache.org/jira/browse/SPARK-50365 Project: Spark Issue

[jira] [Reopened] (SPARK-50365) Row.jsonValue is not able to serialize LocalDateTime

[ https://issues.apache.org/jira/browse/SPARK-50365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karim Ramadan reopened SPARK-50365: --- > Row.jsonValue is not able to serialize LocalDateTime > ---

[jira] [Created] (SPARK-50362) Skip CrossValidatorTests if `torch/torcheval` not installed

Yang Jie created SPARK-50362: Summary: Skip CrossValidatorTests if `torch/torcheval` not installed Key: SPARK-50362 URL: https://issues.apache.org/jira/browse/SPARK-50362 Project: Spark Issue Ty

[jira] [Created] (SPARK-50363) Refine the docstring for datetime functions - part 1

Ruifeng Zheng created SPARK-50363: - Summary: Refine the docstring for datetime functions - part 1 Key: SPARK-50363 URL: https://issues.apache.org/jira/browse/SPARK-50363 Project: Spark Issue

[jira] [Assigned] (SPARK-50361) Agg using Lateral Column Alias should throw LATERAL_COLUMN_ALIAS_IN_AG GREGATE_FUNC

[ https://issues.apache.org/jira/browse/SPARK-50361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-50361: -- Assignee: Apache Spark > Agg using Lateral Column Alias should throw LATERAL_COLUMN_A

[jira] [Updated] (SPARK-50361) Agg using Lateral Column Alias in ORDER BY should throw LATERAL_COLUMN_ALIAS_IN_AG GREGATE_FUNC

[ https://issues.apache.org/jira/browse/SPARK-50361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-50361: - Summary: Agg using Lateral Column Alias in ORDER BY should throw LATERAL_COLUMN_ALIAS_IN_AG GREGATE_FUNC

[jira] [Resolved] (SPARK-50353) Refactor ResolveSQLOnFile

[ https://issues.apache.org/jira/browse/SPARK-50353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-50353. -- Resolution: Fixed Issue resolved by pull request 48894 [https://github.com/apache/spark/pull/48894] >

[jira] [Assigned] (SPARK-50361) Agg using Lateral Column Alias should throw LATERAL_COLUMN_ALIAS_IN_AG GREGATE_FUNC

[ https://issues.apache.org/jira/browse/SPARK-50361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-50361: -- Assignee: (was: Apache Spark) > Agg using Lateral Column Alias should throw LATER

[jira] [Assigned] (SPARK-49907) [Connect] Support spark.ml on Connect

[ https://issues.apache.org/jira/browse/SPARK-49907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-49907: -- Assignee: (was: Apache Spark) > [Connect] Support spark.ml on Connect > -

[jira] [Assigned] (SPARK-50361) Agg using Lateral Column Alias should throw LATERAL_COLUMN_ALIAS_IN_AG GREGATE_FUNC

[ https://issues.apache.org/jira/browse/SPARK-50361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-50361: -- Assignee: Apache Spark > Agg using Lateral Column Alias should throw LATERAL_COLUMN_A

[jira] [Assigned] (SPARK-50361) Agg using Lateral Column Alias should throw LATERAL_COLUMN_ALIAS_IN_AG GREGATE_FUNC

[ https://issues.apache.org/jira/browse/SPARK-50361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-50361: -- Assignee: (was: Apache Spark) > Agg using Lateral Column Alias should throw LATER

[jira] [Updated] (SPARK-50361) Agg using Lateral Column Alias should throw LATERAL_COLUMN_ALIAS_IN_AG GREGATE_FUNC

[ https://issues.apache.org/jira/browse/SPARK-50361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-50361: --- Labels: pull-request-available (was: ) > Agg using Lateral Column Alias should throw LATERA

[jira] [Assigned] (SPARK-49907) [Connect] Support spark.ml on Connect

[ https://issues.apache.org/jira/browse/SPARK-49907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-49907: -- Assignee: Apache Spark > [Connect] Support spark.ml on Connect >

[jira] [Created] (SPARK-50361) Agg using Lateral Column Alias should throw LATERAL_COLUMN_ALIAS_IN_AG GREGATE_FUNC

Kent Yao created SPARK-50361: Summary: Agg using Lateral Column Alias should throw LATERAL_COLUMN_ALIAS_IN_AG GREGATE_FUNC Key: SPARK-50361 URL: https://issues.apache.org/jira/browse/SPARK-50361 Project:

[jira] [Updated] (SPARK-50351) javaoptions execution method.

[ https://issues.apache.org/jira/browse/SPARK-50351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] TiSong Zhang updated SPARK-50351: - Description: Executing javaOptions as bash -c has an execution problem that causes the commands