[jira] [Resolved] (SPARK-46753) Fix `pypy3` python test

2024-01-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-46753. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44778 [https://gi

[jira] [Assigned] (SPARK-46753) Fix `pypy3` python test

2024-01-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-46753: Assignee: BingKun Pan > Fix `pypy3` python test > --- > >

[jira] [Assigned] (SPARK-46926) Add `convert_dtypes`, `infer_objects` and `set_axis` in fallback list

2024-01-30 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-46926: - Assignee: Ruifeng Zheng > Add `convert_dtypes`, `infer_objects` and `set_axis` in fallb

[jira] [Resolved] (SPARK-46926) Add `convert_dtypes`, `infer_objects` and `set_axis` in fallback list

2024-01-30 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-46926. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44965 [https://

[jira] [Created] (SPARK-46931) Implement {Frame, Series}.to_hdf

2024-01-30 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-46931: - Summary: Implement {Frame, Series}.to_hdf Key: SPARK-46931 URL: https://issues.apache.org/jira/browse/SPARK-46931 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-46926) Add `convert_dtypes`, `infer_objects` and `set_axis` in fallback list

2024-01-30 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-46926: -- Summary: Add `convert_dtypes`, `infer_objects` and `set_axis` in fallback list (was: Support

[jira] [Updated] (SPARK-46926) Support `convert_dtypes`, `infer_objects` and `set_axis` in fallback mode

2024-01-30 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-46926: -- Summary: Support `convert_dtypes`, `infer_objects` and `set_axis` in fallback mode (was: Supp

[jira] [Updated] (SPARK-46930) Add support for a custom prefix for fields of Avro union type when enableStableIdentifiersForUnionType is enabled

2024-01-30 Thread Ivan Sadikov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-46930: - Description: {{enableStableIdentifiersForUnionType}} allows to enable stable identifiers in Avro

[jira] [Created] (SPARK-46930) Add support for a custom prefix for fields of Avro union type when enableStableIdentifiersForUnionType is enabled

2024-01-30 Thread Ivan Sadikov (Jira)
Ivan Sadikov created SPARK-46930: Summary: Add support for a custom prefix for fields of Avro union type when enableStableIdentifiersForUnionType is enabled Key: SPARK-46930 URL: https://issues.apache.org/jira/br

[jira] [Updated] (SPARK-46929) Use ThreadUtils.shutdown to close thread pools

2024-01-30 Thread Jiaan Geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaan Geng updated SPARK-46929: --- Component/s: Connect Spark Core SS (was: SQL)

[jira] [Updated] (SPARK-46747) Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1

2024-01-30 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-46747: - Fix Version/s: 3.5.1 3.4.3 > Too Many Shared Locks due to PostgresDialect.getTableExi

[jira] [Resolved] (SPARK-46924) Fix `Load New` button in `Master/HistoryServer` Log UI

2024-01-30 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-46924. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44957 [https://github.com

[jira] [Resolved] (SPARK-46927) Make `assertDataFrameEqual` work properly without PyArrow

2024-01-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-46927. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44960 [https://gi

[jira] [Created] (SPARK-46929) Use ThreadUtils.shutdown to close thread pools

2024-01-30 Thread Jiaan Geng (Jira)
Jiaan Geng created SPARK-46929: -- Summary: Use ThreadUtils.shutdown to close thread pools Key: SPARK-46929 URL: https://issues.apache.org/jira/browse/SPARK-46929 Project: Spark Issue Type: Improv

[jira] [Assigned] (SPARK-46927) Make `assertDataFrameEqual` work properly without PyArrow

2024-01-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-46927: Assignee: Haejoon Lee > Make `assertDataFrameEqual` work properly without PyArrow > -

[jira] [Created] (SPARK-46927) Make `assertDataFrameEqual` work properly without PyArrow

2024-01-30 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46927: --- Summary: Make `assertDataFrameEqual` work properly without PyArrow Key: SPARK-46927 URL: https://issues.apache.org/jira/browse/SPARK-46927 Project: Spark Issue

[jira] [Created] (SPARK-46928) Support ListState in Arbitrary State API v2

2024-01-30 Thread Bhuwan Sahni (Jira)
Bhuwan Sahni created SPARK-46928: Summary: Support ListState in Arbitrary State API v2 Key: SPARK-46928 URL: https://issues.apache.org/jira/browse/SPARK-46928 Project: Spark Issue Type: New F

[jira] [Created] (SPARK-46926) Support `convert_dtypes` and `infer_objects` in fallback mode

2024-01-30 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-46926: - Summary: Support `convert_dtypes` and `infer_objects` in fallback mode Key: SPARK-46926 URL: https://issues.apache.org/jira/browse/SPARK-46926 Project: Spark

[jira] [Assigned] (SPARK-46925) Add a warning that instructs to install memory_profiler for memory profiling

2024-01-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-46925: Assignee: Xinrong Meng > Add a warning that instructs to install memory_profiler for memo

[jira] [Resolved] (SPARK-46925) Add a warning that instructs to install memory_profiler for memory profiling

2024-01-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-46925. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44958 [https://gi

[jira] [Resolved] (SPARK-46192) failed to insert the table using the default value of union

2024-01-30 Thread zengxl (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zengxl resolved SPARK-46192. Resolution: Fixed > failed to insert the table using the default value of union >

[jira] [Commented] (SPARK-46192) failed to insert the table using the default value of union

2024-01-30 Thread zengxl (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17812518#comment-17812518 ] zengxl commented on SPARK-46192: This patch solves all of these problems https://issues

[jira] [Resolved] (SPARK-46923) Limit width of config tables in documentation and style them consistently

2024-01-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-46923. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44955 [https://gi

[jira] [Assigned] (SPARK-46923) Limit width of config tables in documentation and style them consistently

2024-01-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-46923: Assignee: Nicholas Chammas > Limit width of config tables in documentation and style them

[jira] [Created] (SPARK-46925) Add a warning that instructs to install memory_profiler for memory profiling

2024-01-30 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-46925: Summary: Add a warning that instructs to install memory_profiler for memory profiling Key: SPARK-46925 URL: https://issues.apache.org/jira/browse/SPARK-46925 Project:

[jira] [Assigned] (SPARK-46924) Fix `Load New` button in `Master/HistoryServer` Log UI

2024-01-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46924: - Assignee: Dongjoon Hyun > Fix `Load New` button in `Master/HistoryServer` Log UI >

[jira] [Updated] (SPARK-46924) Fix `Load New` button in `Master/HistoryServer` Log UI

2024-01-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46924: -- Summary: Fix `Load New` button in `Master/HistoryServer` Log UI (was: Fix `Load New` button i

[jira] [Created] (SPARK-46924) Fix `Load New` button in `Master/HistoryServer` UI

2024-01-30 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-46924: - Summary: Fix `Load New` button in `Master/HistoryServer` UI Key: SPARK-46924 URL: https://issues.apache.org/jira/browse/SPARK-46924 Project: Spark Issue Ty

[jira] [Updated] (SPARK-46923) Limit width of config tables in documentation and style them consistently

2024-01-30 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-46923: - Summary: Limit width of config tables in documentation and style them consistently (was

[jira] [Updated] (SPARK-46830) Introducing collation concept into Spark

2024-01-30 Thread Aleksandar Tomic (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksandar Tomic updated SPARK-46830: - Description: This feature will introduce collation support to the Spark engine. This mea

[jira] [Updated] (SPARK-46911) Add deleteIfExists operator to StatefulProcessorHandle

2024-01-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46911: --- Labels: pull-request-available (was: ) > Add deleteIfExists operator to StatefulProcessorHa

[jira] [Updated] (SPARK-46798) Kafka custom partition location assignment in Spark Structured Streaming (rack awareness)

2024-01-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46798: --- Labels: pull-request-available (was: ) > Kafka custom partition location assignment in Spar

[jira] [Created] (SPARK-46923) Style config tables in documentation consistently

2024-01-30 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-46923: Summary: Style config tables in documentation consistently Key: SPARK-46923 URL: https://issues.apache.org/jira/browse/SPARK-46923 Project: Spark Iss

[jira] [Resolved] (SPARK-46921) Move `ProblemFilters` that do not belong to defaultExcludes to `v40excludes`.

2024-01-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46921. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44952 [https://

[jira] [Updated] (SPARK-46473) Reuse `getPartitionedFile` method

2024-01-30 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-46473: - Priority: Trivial (was: Minor) > Reuse `getPartitionedFile` method > --

[jira] [Created] (SPARK-46922) better handling for runtime user errors

2024-01-30 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-46922: --- Summary: better handling for runtime user errors Key: SPARK-46922 URL: https://issues.apache.org/jira/browse/SPARK-46922 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-46747) Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1

2024-01-30 Thread Bala Bellam (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17812345#comment-17812345 ] Bala Bellam commented on SPARK-46747: - Thanks [~yao] . So, does this get released to

[jira] [Updated] (SPARK-46919) Upgrade `grpcio*` to 1.60.0 and `grpc-java` to 1.61.0

2024-01-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46919: --- Labels: pull-request-available (was: ) > Upgrade `grpcio*` to 1.60.0 and `grpc-java` to 1.6

[jira] [Commented] (SPARK-46747) Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1

2024-01-30 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17812341#comment-17812341 ] Kent Yao commented on SPARK-46747: -- according to the release policies of Spark,patches

[jira] [Updated] (SPARK-46747) Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1

2024-01-30 Thread Bala Bellam (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bala Bellam updated SPARK-46747: Fix Version/s: (was: 2.3.2) > Too Many Shared Locks due to PostgresDialect.getTableExistsQuery

[jira] [Updated] (SPARK-46747) Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1

2024-01-30 Thread Bala Bellam (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bala Bellam updated SPARK-46747: Fix Version/s: 2.3.2 > Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1

[jira] [Commented] (SPARK-46747) Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1

2024-01-30 Thread Bala Bellam (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17812302#comment-17812302 ] Bala Bellam commented on SPARK-46747: - Thank you very much [~yao] . Sure, I can prov

[jira] [Updated] (SPARK-46747) Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1

2024-01-30 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-46747: - Target Version/s: (was: 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, 2.2.2, 2.2.3, 2

[jira] [Updated] (SPARK-46747) Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1

2024-01-30 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-46747: - Priority: Major (was: Critical) > Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LI

[jira] [Resolved] (SPARK-46747) Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1

2024-01-30 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-46747. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44948 [https://github.com

[jira] [Assigned] (SPARK-46747) Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1

2024-01-30 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-46747: Assignee: Kent Yao > Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1 >

[jira] [Resolved] (SPARK-46918) Replace self-defined variables with Hadoop ContainerExitStatus

2024-01-30 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-46918. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44950 [https://github.com

[jira] [Assigned] (SPARK-46918) Replace self-defined variables with Hadoop ContainerExitStatus

2024-01-30 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-46918: Assignee: Cheng Pan > Replace self-defined variables with Hadoop ContainerExitStatus > --

[jira] [Created] (SPARK-46919) Upgrade `grpcio*` to 1.60.0 and `grpc-java` to 1.61.0

2024-01-30 Thread Yang Jie (Jira)
Yang Jie created SPARK-46919: Summary: Upgrade `grpcio*` to 1.60.0 and `grpc-java` to 1.61.0 Key: SPARK-46919 URL: https://issues.apache.org/jira/browse/SPARK-46919 Project: Spark Issue Type: Imp

[jira] [Updated] (SPARK-46917) [SPARK-46753][PYTHON][TESTS] Fix pypy3 python test

2024-01-30 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-46917: Summary: [SPARK-46753][PYTHON][TESTS] Fix pypy3 python test (was: Improve merge_spark_pr.py) > [

[jira] [Updated] (SPARK-46918) Replace self-defined variables with Hadoop ContainerExitStatus

2024-01-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46918: --- Labels: pull-request-available (was: ) > Replace self-defined variables with Hadoop Contain

[jira] [Created] (SPARK-46918) Replace self-defined variables with Hadoop ContainerExitStatus

2024-01-30 Thread Cheng Pan (Jira)
Cheng Pan created SPARK-46918: - Summary: Replace self-defined variables with Hadoop ContainerExitStatus Key: SPARK-46918 URL: https://issues.apache.org/jira/browse/SPARK-46918 Project: Spark Iss

[jira] [Assigned] (SPARK-46890) CSV fails on a column with default and without enforcing schema

2024-01-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46890: -- Assignee: (was: Apache Spark) > CSV fails on a column with default and without en

[jira] [Assigned] (SPARK-46890) CSV fails on a column with default and without enforcing schema

2024-01-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46890: -- Assignee: Apache Spark > CSV fails on a column with default and without enforcing sch

[jira] [Updated] (SPARK-46851) Remove buf version information from the doc contributing.rst

2024-01-30 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-46851: Summary: Remove buf version information from the doc contributing.rst (was: Upgrade `buf` to v1.2