[jira] [Updated] (SPARK-45568) WholeStageCodegenSparkSubmitSuite flakiness

2023-10-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45568: --- Labels: pull-request-available (was: ) > WholeStageCodegenSparkSubmitSuite flakiness >

[jira] [Created] (SPARK-45568) WholeStageCodegenSparkSubmitSuite flakiness

2023-10-16 Thread Kent Yao (Jira)
Kent Yao created SPARK-45568: Summary: WholeStageCodegenSparkSubmitSuite flakiness Key: SPARK-45568 URL: https://issues.apache.org/jira/browse/SPARK-45568 Project: Spark Issue Type: Test

[jira] [Updated] (SPARK-45558) Introduce a metadata file for streaming stateful operator

2023-10-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45558: --- Labels: pull-request-available (was: ) > Introduce a metadata file for streaming stateful

[jira] [Commented] (SPARK-45511) SPIP: State Data Source - Reader

2023-10-16 Thread Chaoqin Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776009#comment-17776009 ] Chaoqin Li commented on SPARK-45511: +1. This will be very useful for debugging stateful streaming 

[jira] [Updated] (SPARK-45567) Remove redundant if in org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender#run

2023-10-16 Thread Min Zhao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Min Zhao updated SPARK-45567: - Summary: Remove redundant if in org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender#run

[jira] [Updated] (SPARK-45567) Remove redundant if block in org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender#run

2023-10-16 Thread Min Zhao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Min Zhao updated SPARK-45567: - Attachment: image-2023-10-17-11-46-31-897.png > Remove redundant if block in >

[jira] [Commented] (SPARK-45567) Remove redundant if block in org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender#run

2023-10-16 Thread Min Zhao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776005#comment-17776005 ] Min Zhao commented on SPARK-45567: -- [~juliuszsompolski] Hello,this ```if``` seems to be redundant

[jira] [Created] (SPARK-45567) Remove redundant if block in org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender#run

2023-10-16 Thread Min Zhao (Jira)
Min Zhao created SPARK-45567: Summary: Remove redundant if block in org.apache.spark.sql.connect.execution.ExecuteGrpcResponseSender#run Key: SPARK-45567 URL: https://issues.apache.org/jira/browse/SPARK-45567

[jira] [Created] (SPARK-45566) Introduce Pandas-like testing utils for Pandas API on Spark

2023-10-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45566: --- Summary: Introduce Pandas-like testing utils for Pandas API on Spark Key: SPARK-45566 URL: https://issues.apache.org/jira/browse/SPARK-45566 Project: Spark

[jira] [Updated] (SPARK-45565) Unnecessary JSON.stringify and JSON.parse loop for task list

2023-10-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45565: --- Labels: pull-request-available (was: ) > Unnecessary JSON.stringify and JSON.parse loop

[jira] [Created] (SPARK-45565) Unnecessary JSON.stringify and JSON.parse loop for task list

2023-10-16 Thread Kent Yao (Jira)
Kent Yao created SPARK-45565: Summary: Unnecessary JSON.stringify and JSON.parse loop for task list Key: SPARK-45565 URL: https://issues.apache.org/jira/browse/SPARK-45565 Project: Spark Issue

[jira] [Updated] (SPARK-45553) Deprecate assertPandasOnSparkEqual

2023-10-16 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45553: Description: We will add new APIs for DataFrame, Series and Index separately, and we should

[jira] [Updated] (SPARK-45553) Deprecate assertPandasOnSparkEqual

2023-10-16 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45553: Summary: Deprecate assertPandasOnSparkEqual (was: Introduce flexible parameters to

[jira] [Resolved] (SPARK-45484) Fix the bug that uses incorrect parquet compression codec lz4raw

2023-10-16 Thread Jiaan Geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaan Geng resolved SPARK-45484. Fix Version/s: 3.5.1 Resolution: Fixed Issue resolved by pull request 43330

[jira] [Updated] (SPARK-45543) WindowGroupLimit causes bug if the other window functions haven't the same window frame as the rank-like functions

2023-10-16 Thread Jiaan Geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaan Geng updated SPARK-45543: --- Summary: WindowGroupLimit causes bug if the other window functions haven't the same window frame as

[jira] [Commented] (SPARK-45543) WindowGroupLimit causes bug if the other window functions haven't the same window frame as the rank-like functions

2023-10-16 Thread Jiaan Geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775974#comment-17775974 ] Jiaan Geng commented on SPARK-45543: [~ronserruya] Thank you. > WindowGroupLimit causes bug if the

[jira] [Updated] (SPARK-45564) Simplify 'DataFrameStatFunctions.bloomFilter' function with 'BloomFilterAggregate' expression

2023-10-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45564: --- Labels: pull-request-available (was: ) > Simplify 'DataFrameStatFunctions.bloomFilter'

[jira] [Created] (SPARK-45564) Simplify 'DataFrameStatFunctions.bloomFilter' function with 'BloomFilterAggregate' expression

2023-10-16 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-45564: - Summary: Simplify 'DataFrameStatFunctions.bloomFilter' function with 'BloomFilterAggregate' expression Key: SPARK-45564 URL: https://issues.apache.org/jira/browse/SPARK-45564

[jira] [Updated] (SPARK-42942) Support coalesce table cache stage partitions

2023-10-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-42942: --- Labels: pull-request-available (was: ) > Support coalesce table cache stage partitions >

[jira] [Assigned] (SPARK-45517) Expand more exception constructors to support error framework parameters

2023-10-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-45517: Assignee: Yihong He > Expand more exception constructors to support error framework

[jira] [Resolved] (SPARK-45517) Expand more exception constructors to support error framework parameters

2023-10-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-45517. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43368

[jira] [Updated] (SPARK-45561) Convert TINYINT catalyst properly in MySQL and MsSQLServer JDBCv2 connector

2023-10-16 Thread Michael Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Zhang updated SPARK-45561: -- Description: MySQL dialect currently incorrectly converts catalyst types `TINYINT` to BYTE.

[jira] [Commented] (SPARK-45527) Task fraction resource request is not expected

2023-10-16 Thread Bobby Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775948#comment-17775948 ] Bobby Wang commented on SPARK-45527: Working on a PR to fix it. > Task fraction resource request is

[jira] [Updated] (SPARK-45563) Spark history files backend currently depend on polling for loading into the history server

2023-10-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-45563: - Affects Version/s: 4.0.0 (was: 3.3.0)

[jira] [Updated] (SPARK-45563) Spark history files backend currently depend on polling for loading into the history server

2023-10-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-45563: - Issue Type: Improvement (was: Bug) > Spark history files backend currently depend on polling

[jira] [Updated] (SPARK-45560) Support spark.read.load() with non-empty path for Python data source API

2023-10-16 Thread Allison Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated SPARK-45560: - Summary: Support spark.read.load() with non-empty path for Python data source API (was:

[jira] [Updated] (SPARK-45563) Spark history files backend currently depend on polling for loading into the history server

2023-10-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-45563: - Description: The spark history server FS  currently depends on polling for loading history

[jira] [Updated] (SPARK-45563) Spark history files backend currently depend on polling for loading into the history server

2023-10-16 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-45563: - Summary: Spark history files backend currently depend on polling for loading into the history

[jira] [Updated] (SPARK-45559) Support spark.read.schema(...) for Python data source API

2023-10-16 Thread Allison Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated SPARK-45559: - Summary: Support spark.read.schema(...) for Python data source API (was: Support

[jira] [Created] (SPARK-45563) Spark rolling history files currently depend on polling for loading into the history server

2023-10-16 Thread Holden Karau (Jira)
Holden Karau created SPARK-45563: Summary: Spark rolling history files currently depend on polling for loading into the history server Key: SPARK-45563 URL: https://issues.apache.org/jira/browse/SPARK-45563

[jira] [Updated] (SPARK-45561) Convert TINYINT and SMALLINT catalyst properly in MySQL JDBCv2 connector

2023-10-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45561: --- Labels: pull-request-available (was: ) > Convert TINYINT and SMALLINT catalyst properly in

[jira] [Updated] (SPARK-45562) XML: Make 'rowTag' a required option

2023-10-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45562: --- Labels: pull-request-available (was: ) > XML: Make 'rowTag' a required option >

[jira] [Created] (SPARK-45562) XML: Make 'rowTag' a required option

2023-10-16 Thread Sandip Agarwala (Jira)
Sandip Agarwala created SPARK-45562: --- Summary: XML: Make 'rowTag' a required option Key: SPARK-45562 URL: https://issues.apache.org/jira/browse/SPARK-45562 Project: Spark Issue Type:

[jira] [Created] (SPARK-45561) Convert TINYINT and SMALLINT catalyst properly in MySQL JDBCv2 connector

2023-10-16 Thread Michael Zhang (Jira)
Michael Zhang created SPARK-45561: - Summary: Convert TINYINT and SMALLINT catalyst properly in MySQL JDBCv2 connector Key: SPARK-45561 URL: https://issues.apache.org/jira/browse/SPARK-45561 Project:

[jira] [Created] (SPARK-45560) Support spark.read.load() with paths for Python data source API

2023-10-16 Thread Allison Wang (Jira)
Allison Wang created SPARK-45560: Summary: Support spark.read.load() with paths for Python data source API Key: SPARK-45560 URL: https://issues.apache.org/jira/browse/SPARK-45560 Project: Spark

[jira] [Created] (SPARK-45559) Support df.read.schema(...) for Python data source API

2023-10-16 Thread Allison Wang (Jira)
Allison Wang created SPARK-45559: Summary: Support df.read.schema(...) for Python data source API Key: SPARK-45559 URL: https://issues.apache.org/jira/browse/SPARK-45559 Project: Spark Issue

[jira] [Commented] (SPARK-34910) JDBC - Add an option for different stride orders

2023-10-16 Thread Jason Yarbrough (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775922#comment-17775922 ] Jason Yarbrough commented on SPARK-34910: - Hey guys, Sorry for commenting on such an old issue,

[jira] [Commented] (SPARK-45511) SPIP: State Data Source - Reader

2023-10-16 Thread Praveen Gattu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775886#comment-17775886 ] Praveen Gattu commented on SPARK-45511: --- Thank you Jungtaek for starting this discussion. Made a

[jira] [Updated] (SPARK-45557) Spark Connect can not be started because of missing user home dir in Docker container

2023-10-16 Thread Niels Pardon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niels Pardon updated SPARK-45557: - Description: I was trying to start Spark Connect within a container using the Spark Docker

[jira] [Updated] (SPARK-45558) Introduce a metadata file for streaming stateful operator

2023-10-16 Thread Chaoqin Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoqin Li updated SPARK-45558: --- Description: The information to store in the metadata file: * operator name (no need to be unique

[jira] [Created] (SPARK-45558) Introduce a metadata file for streaming stateful operator

2023-10-16 Thread Chaoqin Li (Jira)
Chaoqin Li created SPARK-45558: -- Summary: Introduce a metadata file for streaming stateful operator Key: SPARK-45558 URL: https://issues.apache.org/jira/browse/SPARK-45558 Project: Spark Issue

[jira] [Created] (SPARK-45557) Spark Connect can not be started because of missing user home dir in Docker container

2023-10-16 Thread Niels Pardon (Jira)
Niels Pardon created SPARK-45557: Summary: Spark Connect can not be started because of missing user home dir in Docker container Key: SPARK-45557 URL: https://issues.apache.org/jira/browse/SPARK-45557

[jira] [Updated] (SPARK-45545) [CORE] Pass SSLOptions wherever we create a SparkTransportConf

2023-10-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45545: --- Labels: pull-request-available (was: ) > [CORE] Pass SSLOptions wherever we create a

[jira] [Updated] (SPARK-45541) [CORE] Add SSLFactory

2023-10-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45541: --- Labels: pull-request-available (was: ) > [CORE] Add SSLFactory > - > >

[jira] [Commented] (SPARK-44594) Remove redundant method parameter in kafka connector

2023-10-16 Thread Philip Dakin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775805#comment-17775805 ] Philip Dakin commented on SPARK-44594: -- PR merged, good to close ticket [~zhaomin]. > Remove

[jira] [Updated] (SPARK-45543) WindowGroupLimit causes bug if window expressions exists non-rank function

2023-10-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45543: --- Labels: correctness data-loss pull-request-available (was: correctness data-loss) >

[jira] [Resolved] (SPARK-45273) Http header Attack【HttpSecurityFilter】

2023-10-16 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-45273. -- Resolution: Invalid > Http header Attack【HttpSecurityFilter】 >

[jira] [Assigned] (SPARK-45540) Upgrade jetty to 9.4.53.v20231009

2023-10-16 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-45540: Assignee: Yang Jie > Upgrade jetty to 9.4.53.v20231009 > -- > >

[jira] [Resolved] (SPARK-45540) Upgrade jetty to 9.4.53.v20231009

2023-10-16 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-45540. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43375

[jira] [Created] (SPARK-45556) Inconsistent status code between web page and REST API when exception is thrown

2023-10-16 Thread wy (Jira)
wy created SPARK-45556: -- Summary: Inconsistent status code between web page and REST API when exception is thrown Key: SPARK-45556 URL: https://issues.apache.org/jira/browse/SPARK-45556 Project: Spark

[jira] [Updated] (SPARK-43035) Add error class in Spark Connect server's ErrorInfo

2023-10-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-43035: --- Labels: pull-request-available (was: ) > Add error class in Spark Connect server's

[jira] [Updated] (SPARK-45543) WindowGroupLimit causes bug if window expressions exists non-rank function

2023-10-16 Thread Jiaan Geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaan Geng updated SPARK-45543: --- Summary: WindowGroupLimit causes bug if window expressions exists non-rank function (was:

[jira] [Updated] (SPARK-45543) WindowGroupLimit causes bug if window expressions exists non-rank window function

2023-10-16 Thread Jiaan Geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaan Geng updated SPARK-45543: --- Summary: WindowGroupLimit causes bug if window expressions exists non-rank window function (was:

[jira] [Updated] (SPARK-45543) Cannot insert WindowGroupLimit if window expressions exists non-rank window function

2023-10-16 Thread Jiaan Geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaan Geng updated SPARK-45543: --- Summary: Cannot insert WindowGroupLimit if window expressions exists non-rank window function

[jira] [Updated] (SPARK-45543) WindowGroupLimit Causes incorrect results when window expressions exists non-rank function

2023-10-16 Thread Jiaan Geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaan Geng updated SPARK-45543: --- Summary: WindowGroupLimit Causes incorrect results when window expressions exists non-rank function

[jira] [Updated] (SPARK-45543) WindowGroupLimit causes incorrect results if window expressions exists non-rank window function

2023-10-16 Thread Jiaan Geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaan Geng updated SPARK-45543: --- Summary: WindowGroupLimit causes incorrect results if window expressions exists non-rank window

[jira] [Updated] (SPARK-39420) Support ANALYZE TABLE on v2 tables

2023-10-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-39420: --- Labels: pull-request-available (was: ) > Support ANALYZE TABLE on v2 tables >

[jira] [Created] (SPARK-45555) Returning a debuggable object for failed assertion

2023-10-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-4: --- Summary: Returning a debuggable object for failed assertion Key: SPARK-4 URL: https://issues.apache.org/jira/browse/SPARK-4 Project: Spark Issue Type:

[jira] [Created] (SPARK-45554) Introduce flexible parameter to assertSchemaEqual

2023-10-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45554: --- Summary: Introduce flexible parameter to assertSchemaEqual Key: SPARK-45554 URL: https://issues.apache.org/jira/browse/SPARK-45554 Project: Spark Issue Type:

[jira] [Created] (SPARK-45553) Introduce flexible parameters to assertPandasOnSparkEqual

2023-10-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45553: --- Summary: Introduce flexible parameters to assertPandasOnSparkEqual Key: SPARK-45553 URL: https://issues.apache.org/jira/browse/SPARK-45553 Project: Spark

[jira] [Created] (SPARK-45552) Introduce flexible parameters to assertDataFrameEqual

2023-10-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45552: --- Summary: Introduce flexible parameters to assertDataFrameEqual Key: SPARK-45552 URL: https://issues.apache.org/jira/browse/SPARK-45552 Project: Spark Issue

[jira] [Created] (SPARK-45551) Enchance PySpark testing utils

2023-10-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45551: --- Summary: Enchance PySpark testing utils Key: SPARK-45551 URL: https://issues.apache.org/jira/browse/SPARK-45551 Project: Spark Issue Type: Umbrella

[jira] [Updated] (SPARK-45550) Remove deprecated APIs from Pandas API on Spark

2023-10-16 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45550: Summary: Remove deprecated APIs from Pandas API on Spark (was: Remove to_spark_io) > Remove

[jira] [Updated] (SPARK-45550) Remove to_spark_io

2023-10-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45550: --- Labels: pull-request-available (was: ) > Remove to_spark_io > -- > >

[jira] [Updated] (SPARK-45550) Remove to_spark_io

2023-10-16 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-45550: Summary: Remove to_spark_io (was: Remove deprecated PySpark APIs) > Remove to_spark_io >

[jira] [Updated] (SPARK-45549) Remove unused `numExistingExecutors` in `CoarseGrainedSchedulerBackend`

2023-10-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45549: --- Labels: pull-request-available (was: ) > Remove unused `numExistingExecutors` in

[jira] [Created] (SPARK-45550) Remove deprecated PySpark APIs

2023-10-16 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-45550: --- Summary: Remove deprecated PySpark APIs Key: SPARK-45550 URL: https://issues.apache.org/jira/browse/SPARK-45550 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-45549) Remove unused `numExistingExecutors` in `CoarseGrainedSchedulerBackend`

2023-10-16 Thread xiaoping.huang (Jira)
xiaoping.huang created SPARK-45549: -- Summary: Remove unused `numExistingExecutors` in `CoarseGrainedSchedulerBackend` Key: SPARK-45549 URL: https://issues.apache.org/jira/browse/SPARK-45549 Project:

[jira] [Commented] (SPARK-45533) Use `j.l.r.Cleaner` instead of `finalize` for `RocksDBIterator/LevelDBIterator`

2023-10-16 Thread Min Zhao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775672#comment-17775672 ] Min Zhao commented on SPARK-45533: -- Could you like to assign it to me? I will complete it after

[jira] [Updated] (SPARK-44619) Free up disk space for container jobs

2023-10-16 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-44619: - Fix Version/s: 3.5.1 > Free up disk space for container jobs > - >

[jira] [Updated] (SPARK-44619) Free up disk space for container jobs

2023-10-16 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-44619: - Affects Version/s: 3.5.1 > Free up disk space for container jobs >

[jira] [Assigned] (SPARK-45543) WindowGroupLimit Causes incorrect results when multiple windows are used

2023-10-16 Thread Jiaan Geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaan Geng reassigned SPARK-45543: -- Assignee: Jiaan Geng > WindowGroupLimit Causes incorrect results when multiple windows are

[jira] [Created] (SPARK-45548) Failure to generate pyspark documents due to capitalization in the MacOS environment

2023-10-16 Thread BingKun Pan (Jira)
BingKun Pan created SPARK-45548: --- Summary: Failure to generate pyspark documents due to capitalization in the MacOS environment Key: SPARK-45548 URL: https://issues.apache.org/jira/browse/SPARK-45548

[jira] [Assigned] (SPARK-44262) JdbcUtils hardcodes some SQL statements

2023-10-16 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-44262: Assignee: Jia Fan > JdbcUtils hardcodes some SQL statements >

[jira] [Resolved] (SPARK-44262) JdbcUtils hardcodes some SQL statements

2023-10-16 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-44262. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 41855

[jira] [Assigned] (SPARK-45538) pyspark connect overwrite_partitions bug

2023-10-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-45538: Assignee: xie shuiahu > pyspark connect overwrite_partitions bug >

[jira] [Resolved] (SPARK-45538) pyspark connect overwrite_partitions bug

2023-10-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-45538. -- Fix Version/s: 4.0.0 3.5.1 Resolution: Fixed Fixed in

[jira] [Updated] (SPARK-45486) make add_artifact idempotent

2023-10-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-45486: - Language: (was: Scala) > make add_artifact idempotent > > >

[jira] [Updated] (SPARK-45485) make add_artifact idempotent

2023-10-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-45485: - Description: Similar to the [Python

[jira] [Updated] (SPARK-45486) make add_artifact idempotent

2023-10-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-45486: - Priority: Minor (was: Major) > make add_artifact idempotent > > >

[jira] [Updated] (SPARK-45485) [CONNECT][SCALA] Update user agent to recognize SPARK_CONNECT_USER_AGENT and include environment specific attributes

2023-10-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45485: --- Labels: pull-request-available (was: ) > [CONNECT][SCALA] Update user agent to recognize

[jira] [Updated] (SPARK-45486) make add_artifact idempotent

2023-10-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-45486: - Summary: make add_artifact idempotent (was: [CONNECT][SCALA] Update user agent to recognize

[jira] [Resolved] (SPARK-45531) Add more comments and rename some variable name for InjectRuntimeFilter

2023-10-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-45531. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43359

[jira] [Commented] (SPARK-45543) WindowGroupLimit Causes incorrect results when multiple windows are used

2023-10-16 Thread Jiaan Geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775566#comment-17775566 ] Jiaan Geng commented on SPARK-45543: [~ronserruya]I see. > WindowGroupLimit Causes incorrect

[jira] [Resolved] (SPARK-45491) Replace: _LEGACY_ERROR_TEMP_2196 with a better error class

2023-10-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-45491. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43376