[jira] [Commented] (SPARK-43170) The spark sql like statement is pushed down to parquet for execution, but the data cannot be queried

2023-04-17 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713412#comment-17713412 ] Hyukjin Kwon commented on SPARK-43170: -- There'd be no more releases in Spark 3.2.X

[jira] [Commented] (SPARK-43170) The spark sql like statement is pushed down to parquet for execution, but the data cannot be queried

2023-04-17 Thread todd (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713403#comment-17713403 ] todd commented on SPARK-43170: -- Spark3.2.x is currently used in production, and there is no

[jira] [Comment Edited] (SPARK-42663) Fix `default_session` to work properly

2023-04-17 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713391#comment-17713391 ] Haejoon Lee edited comment on SPARK-42663 at 4/18/23 5:20 AM:

[jira] [Resolved] (SPARK-42663) Fix `default_session` to work properly

2023-04-17 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-42663. - Resolution: Fixed Fixed from SPARK-42859 > Fix `default_session` to work properly > ---

[jira] [Commented] (SPARK-42663) Fix `default_session` to work properly

2023-04-17 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713391#comment-17713391 ] Haejoon Lee commented on SPARK-42663: - Sorry for the late responding, [~Wencong Liu]

[jira] [Resolved] (SPARK-43078) Separate test into `pyspark-conenct-pandas` and `pyspark-connect-pandas-slow`

2023-04-17 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-43078. - Resolution: Duplicate > Separate test into `pyspark-conenct-pandas` and `pyspark-connect-pandas-

[jira] [Created] (SPARK-43171) Support dynamic changing unix user in Pod

2023-04-17 Thread Cheng Pan (Jira)
Cheng Pan created SPARK-43171: - Summary: Support dynamic changing unix user in Pod Key: SPARK-43171 URL: https://issues.apache.org/jira/browse/SPARK-43171 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-43111) Merge nested if statements into single if statements

2023-04-17 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43111: Assignee: Bjørn Jørgensen > Merge nested if statements into single if statements > --

[jira] [Resolved] (SPARK-43111) Merge nested if statements into single if statements

2023-04-17 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43111. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40759 [https://gi

[jira] [Resolved] (SPARK-43113) Codegen error when full outer join's bound condition has multiple references to the same stream-side column

2023-04-17 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43113. -- Fix Version/s: 3.4.1 3.5.0 Assignee: Bruce Robbins Resolutio

[jira] [Updated] (SPARK-43112) Spark may use a column other than the actual specified partitioning column for partitioning, for Hive format tables

2023-04-17 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43112: - Target Version/s: (was: 3.3.3) > Spark may use a column other than the actual specified parti

[jira] [Updated] (SPARK-43112) Spark may use a column other than the actual specified partitioning column for partitioning, for Hive format tables

2023-04-17 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43112: - Priority: Major (was: Critical) > Spark may use a column other than the actual specified parti

[jira] [Commented] (SPARK-43154) Pyspark 3.4 fails when running "pivot" function on a dataframe using the values arguement

2023-04-17 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713377#comment-17713377 ] Hyukjin Kwon commented on SPARK-43154: -- Would be great if we have a self-cotained r

[jira] [Resolved] (SPARK-43163) An exception occurred while hive table join tidb table

2023-04-17 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43163. -- Resolution: Invalid > An exception occurred while hive table join tidb table > ---

[jira] [Commented] (SPARK-43170) The spark sql like statement is pushed down to parquet for execution, but the data cannot be queried

2023-04-17 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713373#comment-17713373 ] Hyukjin Kwon commented on SPARK-43170: -- Spark 3.2.X is EOL. Mind trying if the same

[jira] [Updated] (SPARK-43170) The spark sql like statement is pushed down to parquet for execution, but the data cannot be queried

2023-04-17 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43170: - Priority: Major (was: Blocker) > The spark sql like statement is pushed down to parquet for exe

[jira] [Resolved] (SPARK-43122) Reenable TorchDistributorLocalUnitTestsOnConnect and TorchDistributorLocalUnitTestsIIOnConnect

2023-04-17 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-43122. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40793 [https://

[jira] [Assigned] (SPARK-43148) Add official image dockerfile for Spark v3.4.0

2023-04-17 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang reassigned SPARK-43148: --- Assignee: Yikun Jiang > Add official image dockerfile for Spark v3.4.0 > --

[jira] [Updated] (SPARK-43170) The spark sql like statement is pushed down to parquet for execution, but the data cannot be queried

2023-04-17 Thread todd (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] todd updated SPARK-43170: - Description: --DDL CREATE TABLE `ecom_dwm`.`dwm_user_app_action_sum_all` (   `gaid` STRING COMMENT '',   `beyla

[jira] [Updated] (SPARK-43170) The spark sql like statement is pushed down to parquet for execution, but the data cannot be queried

2023-04-17 Thread todd (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] todd updated SPARK-43170: - Description: --DDL CREATE TABLE `ecom_dwm`.`dwm_user_app_action_sum_all` (   `gaid` STRING COMMENT '',   `beyla

[jira] [Updated] (SPARK-43170) The spark sql like statement is pushed down to parquet for execution, but the data cannot be queried

2023-04-17 Thread todd (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] todd updated SPARK-43170: - Attachment: image-2023-04-18-10-59-30-199.png > The spark sql like statement is pushed down to parquet for execu

[jira] [Resolved] (SPARK-43148) Add official image dockerfile for Spark v3.4.0

2023-04-17 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang resolved SPARK-43148. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 33 [https://github.

[jira] [Created] (SPARK-43170) The spark sql like statement is pushed down to parquet for execution, but the data cannot be queried

2023-04-17 Thread todd (Jira)
todd created SPARK-43170: Summary: The spark sql like statement is pushed down to parquet for execution, but the data cannot be queried Key: SPARK-43170 URL: https://issues.apache.org/jira/browse/SPARK-43170

[jira] [Created] (SPARK-43169) Update mima's previousSparkVersion to 3.4.0

2023-04-17 Thread Yang Jie (Jira)
Yang Jie created SPARK-43169: Summary: Update mima's previousSparkVersion to 3.4.0 Key: SPARK-43169 URL: https://issues.apache.org/jira/browse/SPARK-43169 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-43168) Remove get PhysicalDataType method from Datatype class

2023-04-17 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-43168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-43168. --- Fix Version/s: 3.5.0 Resolution: Fixed > Remove get PhysicalDataType method f

[jira] [Resolved] (SPARK-42984) Fix test_createDataFrame_with_single_data_type.

2023-04-17 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-42984. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40828 [https://

[jira] [Assigned] (SPARK-42984) Fix test_createDataFrame_with_single_data_type.

2023-04-17 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-42984: - Assignee: Takuya Ueshin > Fix test_createDataFrame_with_single_data_type. > ---

[jira] [Resolved] (SPARK-41210) Window based executor failure tracking mechanism

2023-04-17 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-41210. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40774 [https://github.com

[jira] [Assigned] (SPARK-41210) Window based executor failure tracking mechanism

2023-04-17 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-41210: Assignee: Cheng Pan > Window based executor failure tracking mechanism >

[jira] [Resolved] (SPARK-43084) Add Python state API (applyInPandasWithState) and verify UDFs

2023-04-17 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43084. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40736 [https://gi

[jira] [Assigned] (SPARK-43084) Add Python state API (applyInPandasWithState) and verify UDFs

2023-04-17 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43084: Assignee: Peng Zhong > Add Python state API (applyInPandasWithState) and verify UDFs > --

[jira] [Commented] (SPARK-43146) Implement eager evaluation.

2023-04-17 Thread GridGain Integration (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713265#comment-17713265 ] GridGain Integration commented on SPARK-43146: -- User 'ueshin' has created a

[jira] [Updated] (SPARK-43165) Move canWrite to DataTypeUtils

2023-04-17 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated SPARK-43165: - Affects Version/s: 3.5.0 (was: 3.4.0) > Move canWrite to DataTypeUtils >

[jira] [Created] (SPARK-43168) Remove get PhysicalDataType method from Datatype class

2023-04-17 Thread Rui Wang (Jira)
Rui Wang created SPARK-43168: Summary: Remove get PhysicalDataType method from Datatype class Key: SPARK-43168 URL: https://issues.apache.org/jira/browse/SPARK-43168 Project: Spark Issue Type: Su

[jira] [Created] (SPARK-43167) Streaming Connect console output format support

2023-04-17 Thread Wei Liu (Jira)
Wei Liu created SPARK-43167: --- Summary: Streaming Connect console output format support Key: SPARK-43167 URL: https://issues.apache.org/jira/browse/SPARK-43167 Project: Spark Issue Type: Task

[jira] [Commented] (SPARK-43166) Docker images are missing passwd entry for UID 185

2023-04-17 Thread Derek Brown (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713232#comment-17713232 ] Derek Brown commented on SPARK-43166: - I created a PR for this here: https://github

[jira] [Created] (SPARK-43166) Docker images are missing passwd entry for UID 185

2023-04-17 Thread Derek Brown (Jira)
Derek Brown created SPARK-43166: --- Summary: Docker images are missing passwd entry for UID 185 Key: SPARK-43166 URL: https://issues.apache.org/jira/browse/SPARK-43166 Project: Spark Issue Type:

[jira] [Created] (SPARK-43165) Move canWrite to DataTypeUtils

2023-04-17 Thread Rui Wang (Jira)
Rui Wang created SPARK-43165: Summary: Move canWrite to DataTypeUtils Key: SPARK-43165 URL: https://issues.apache.org/jira/browse/SPARK-43165 Project: Spark Issue Type: Sub-task Compone

[jira] [Resolved] (SPARK-42657) Support to find and transfer client-side REPL classfiles to server as artifacts

2023-04-17 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-42657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-42657. --- Fix Version/s: 3.5.0 Assignee: Venkata Sai Akhil Gudesa Resolution: F

[jira] [Commented] (SPARK-43164) Spark overwrites existing FILES/ARCHIVES/... settings instead of merging them

2023-04-17 Thread YE (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713084#comment-17713084 ] YE commented on SPARK-43164: hi [~vanzin], [~kabhwan] do you guys have any input for this is

[jira] [Created] (SPARK-43164) Spark overwrites existing FILES/ARCHIVES/... settings instead of merging them

2023-04-17 Thread YE (Jira)
YE created SPARK-43164: -- Summary: Spark overwrites existing FILES/ARCHIVES/... settings instead of merging them Key: SPARK-43164 URL: https://issues.apache.org/jira/browse/SPARK-43164 Project: Spark Is

[jira] [Commented] (SPARK-43163) An exception occurred while hive table join tidb table

2023-04-17 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713069#comment-17713069 ] Yuming Wang commented on SPARK-43163: - It seems like TiSpark issue. > An exception

[jira] [Commented] (SPARK-43152) Parametrisable output metadata path (_spark_metadata)

2023-04-17 Thread Wojciech Indyk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713044#comment-17713044 ] Wojciech Indyk commented on SPARK-43152: I raised a PR here: https://github.com/

[jira] [Created] (SPARK-43163) An exception occurred while hive table join tidb table

2023-04-17 Thread Zhang Qi (Jira)
Zhang Qi created SPARK-43163: Summary: An exception occurred while hive table join tidb table Key: SPARK-43163 URL: https://issues.apache.org/jira/browse/SPARK-43163 Project: Spark Issue Type: Bu

[jira] [Updated] (SPARK-42869) can not analyze window exp on sub query

2023-04-17 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-42869: Attachment: (was: image-2023-04-17-19-06-54-728.png) > can not analyze window exp on sub query > -

[jira] [Updated] (SPARK-42869) can not analyze window exp on sub query

2023-04-17 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-42869: Attachment: image-2023-04-17-19-09-41-485.png > can not analyze window exp on sub query >

[jira] [Commented] (SPARK-42869) can not analyze window exp on sub query

2023-04-17 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713029#comment-17713029 ] Jia Fan commented on SPARK-42869: - Can you try with master branch? I tested without any

[jira] [Updated] (SPARK-42869) can not analyze window exp on sub query

2023-04-17 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-42869: Attachment: image-2023-04-17-19-06-54-728.png > can not analyze window exp on sub query >

[jira] [Updated] (SPARK-42869) can not analyze window exp on sub query

2023-04-17 Thread Jia Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan updated SPARK-42869: Attachment: image-2023-04-17-19-06-28-069.png > can not analyze window exp on sub query >

[jira] [Updated] (SPARK-43162) Runtime filter should extracts multiple filter creation side plans

2023-04-17 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-43162: --- Description: Currently, runtime filter only extract one filter creation side plan. In some scene, in

[jira] [Updated] (SPARK-43162) Runtime filter should extracts multiple filter creation side plans

2023-04-17 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-43162: --- Description: Currently, Runtime filter only extract one > Runtime filter should extracts multiple f

[jira] [Created] (SPARK-43162) Runtime filter should extracts multiple filter creation side plans

2023-04-17 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-43162: -- Summary: Runtime filter should extracts multiple filter creation side plans Key: SPARK-43162 URL: https://issues.apache.org/jira/browse/SPARK-43162 Project: Spark

[jira] [Updated] (SPARK-43160) Remove typing.io namespace references as it is being removed

2023-04-17 Thread Aimilios Tsouvelekakis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aimilios Tsouvelekakis updated SPARK-43160: --- Summary: Remove typing.io namespace references as it is being removed (was:

[jira] [Updated] (SPARK-43160) Remove typing.io references as it is being deprecated from python 3.12

2023-04-17 Thread Aimilios Tsouvelekakis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aimilios Tsouvelekakis updated SPARK-43160: --- Description: Python 3.11 gives a deprecation warning to the following: {code

[jira] [Updated] (SPARK-43160) Remove typing.io references as it is being deprecated from python 3.12

2023-04-17 Thread Aimilios Tsouvelekakis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aimilios Tsouvelekakis updated SPARK-43160: --- Description: Python 3.11 gives a deprecation warning to the following: {code

[jira] [Created] (SPARK-43161) Throw error in planning phase when the query redefines watermark for a stream

2023-04-17 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-43161: Summary: Throw error in planning phase when the query redefines watermark for a stream Key: SPARK-43161 URL: https://issues.apache.org/jira/browse/SPARK-43161 Project

[jira] [Created] (SPARK-43160) Remove typing.io references as it is being deprecated from python 3.12

2023-04-17 Thread Aimilios Tsouvelekakis (Jira)
Aimilios Tsouvelekakis created SPARK-43160: -- Summary: Remove typing.io references as it is being deprecated from python 3.12 Key: SPARK-43160 URL: https://issues.apache.org/jira/browse/SPARK-43160