[jira] [Resolved] (SPARK-46753) Fix `pypy3` python test

[ https://issues.apache.org/jira/browse/SPARK-46753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-46753. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44778

[jira] [Assigned] (SPARK-46753) Fix `pypy3` python test

[ https://issues.apache.org/jira/browse/SPARK-46753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-46753: Assignee: BingKun Pan > Fix `pypy3` python test > --- > >

[jira] [Assigned] (SPARK-46926) Add `convert_dtypes`, `infer_objects` and `set_axis` in fallback list

[ https://issues.apache.org/jira/browse/SPARK-46926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-46926: - Assignee: Ruifeng Zheng > Add `convert_dtypes`, `infer_objects` and `set_axis` in

[jira] [Resolved] (SPARK-46926) Add `convert_dtypes`, `infer_objects` and `set_axis` in fallback list

[ https://issues.apache.org/jira/browse/SPARK-46926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-46926. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44965

[jira] [Created] (SPARK-46931) Implement {Frame, Series}.to_hdf

Ruifeng Zheng created SPARK-46931: - Summary: Implement {Frame, Series}.to_hdf Key: SPARK-46931 URL: https://issues.apache.org/jira/browse/SPARK-46931 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-46926) Add `convert_dtypes`, `infer_objects` and `set_axis` in fallback list

[ https://issues.apache.org/jira/browse/SPARK-46926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-46926: -- Summary: Add `convert_dtypes`, `infer_objects` and `set_axis` in fallback list (was: Support

[jira] [Updated] (SPARK-46926) Support `convert_dtypes`, `infer_objects` and `set_axis` in fallback mode

[ https://issues.apache.org/jira/browse/SPARK-46926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-46926: -- Summary: Support `convert_dtypes`, `infer_objects` and `set_axis` in fallback mode (was:

[jira] [Updated] (SPARK-46930) Add support for a custom prefix for fields of Avro union type when enableStableIdentifiersForUnionType is enabled

[ https://issues.apache.org/jira/browse/SPARK-46930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-46930: - Description: {{enableStableIdentifiersForUnionType}} allows to enable stable identifiers in

[jira] [Created] (SPARK-46930) Add support for a custom prefix for fields of Avro union type when enableStableIdentifiersForUnionType is enabled

Ivan Sadikov created SPARK-46930: Summary: Add support for a custom prefix for fields of Avro union type when enableStableIdentifiersForUnionType is enabled Key: SPARK-46930 URL:

[jira] [Updated] (SPARK-46929) Use ThreadUtils.shutdown to close thread pools

[ https://issues.apache.org/jira/browse/SPARK-46929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaan Geng updated SPARK-46929: --- Component/s: Connect Spark Core SS (was: SQL)

[jira] [Updated] (SPARK-46747) Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1

[ https://issues.apache.org/jira/browse/SPARK-46747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-46747: - Fix Version/s: 3.5.1 3.4.3 > Too Many Shared Locks due to

[jira] [Resolved] (SPARK-46924) Fix `Load New` button in `Master/HistoryServer` Log UI

[ https://issues.apache.org/jira/browse/SPARK-46924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-46924. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44957

[jira] [Resolved] (SPARK-46927) Make `assertDataFrameEqual` work properly without PyArrow

[ https://issues.apache.org/jira/browse/SPARK-46927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-46927. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44960

[jira] [Created] (SPARK-46929) Use ThreadUtils.shutdown to close thread pools

Jiaan Geng created SPARK-46929: -- Summary: Use ThreadUtils.shutdown to close thread pools Key: SPARK-46929 URL: https://issues.apache.org/jira/browse/SPARK-46929 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-46927) Make `assertDataFrameEqual` work properly without PyArrow

[ https://issues.apache.org/jira/browse/SPARK-46927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-46927: Assignee: Haejoon Lee > Make `assertDataFrameEqual` work properly without PyArrow >

[jira] [Created] (SPARK-46927) Make `assertDataFrameEqual` work properly without PyArrow

Haejoon Lee created SPARK-46927: --- Summary: Make `assertDataFrameEqual` work properly without PyArrow Key: SPARK-46927 URL: https://issues.apache.org/jira/browse/SPARK-46927 Project: Spark

[jira] [Created] (SPARK-46928) Support ListState in Arbitrary State API v2

Bhuwan Sahni created SPARK-46928: Summary: Support ListState in Arbitrary State API v2 Key: SPARK-46928 URL: https://issues.apache.org/jira/browse/SPARK-46928 Project: Spark Issue Type: New

[jira] [Created] (SPARK-46926) Support `convert_dtypes` and `infer_objects` in fallback mode

Ruifeng Zheng created SPARK-46926: - Summary: Support `convert_dtypes` and `infer_objects` in fallback mode Key: SPARK-46926 URL: https://issues.apache.org/jira/browse/SPARK-46926 Project: Spark

[jira] [Assigned] (SPARK-46925) Add a warning that instructs to install memory_profiler for memory profiling

[ https://issues.apache.org/jira/browse/SPARK-46925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-46925: Assignee: Xinrong Meng > Add a warning that instructs to install memory_profiler for

[jira] [Resolved] (SPARK-46925) Add a warning that instructs to install memory_profiler for memory profiling

[ https://issues.apache.org/jira/browse/SPARK-46925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-46925. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44958

[jira] [Resolved] (SPARK-46192) failed to insert the table using the default value of union

[ https://issues.apache.org/jira/browse/SPARK-46192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zengxl resolved SPARK-46192. Resolution: Fixed > failed to insert the table using the default value of union >

[jira] [Commented] (SPARK-46192) failed to insert the table using the default value of union

[ https://issues.apache.org/jira/browse/SPARK-46192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17812518#comment-17812518 ] zengxl commented on SPARK-46192: This patch solves all of these problems

[jira] [Resolved] (SPARK-46923) Limit width of config tables in documentation and style them consistently

[ https://issues.apache.org/jira/browse/SPARK-46923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-46923. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44955

[jira] [Assigned] (SPARK-46923) Limit width of config tables in documentation and style them consistently

[ https://issues.apache.org/jira/browse/SPARK-46923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-46923: Assignee: Nicholas Chammas > Limit width of config tables in documentation and style

[jira] [Created] (SPARK-46925) Add a warning that instructs to install memory_profiler for memory profiling

Xinrong Meng created SPARK-46925: Summary: Add a warning that instructs to install memory_profiler for memory profiling Key: SPARK-46925 URL: https://issues.apache.org/jira/browse/SPARK-46925

[jira] [Assigned] (SPARK-46924) Fix `Load New` button in `Master/HistoryServer` Log UI

[ https://issues.apache.org/jira/browse/SPARK-46924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46924: - Assignee: Dongjoon Hyun > Fix `Load New` button in `Master/HistoryServer` Log UI >

[jira] [Updated] (SPARK-46924) Fix `Load New` button in `Master/HistoryServer` Log UI

[ https://issues.apache.org/jira/browse/SPARK-46924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46924: -- Summary: Fix `Load New` button in `Master/HistoryServer` Log UI (was: Fix `Load New` button

[jira] [Created] (SPARK-46924) Fix `Load New` button in `Master/HistoryServer` UI

Dongjoon Hyun created SPARK-46924: - Summary: Fix `Load New` button in `Master/HistoryServer` UI Key: SPARK-46924 URL: https://issues.apache.org/jira/browse/SPARK-46924 Project: Spark Issue

[jira] [Updated] (SPARK-46923) Limit width of config tables in documentation and style them consistently

[ https://issues.apache.org/jira/browse/SPARK-46923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-46923: - Summary: Limit width of config tables in documentation and style them consistently

[jira] [Updated] (SPARK-46830) Introducing collation concept into Spark

[ https://issues.apache.org/jira/browse/SPARK-46830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksandar Tomic updated SPARK-46830: - Description: This feature will introduce collation support to the Spark engine. This

[jira] [Updated] (SPARK-46911) Add deleteIfExists operator to StatefulProcessorHandle

[ https://issues.apache.org/jira/browse/SPARK-46911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46911: --- Labels: pull-request-available (was: ) > Add deleteIfExists operator to

[jira] [Updated] (SPARK-46798) Kafka custom partition location assignment in Spark Structured Streaming (rack awareness)

[ https://issues.apache.org/jira/browse/SPARK-46798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46798: --- Labels: pull-request-available (was: ) > Kafka custom partition location assignment in

[jira] [Created] (SPARK-46923) Style config tables in documentation consistently

Nicholas Chammas created SPARK-46923: Summary: Style config tables in documentation consistently Key: SPARK-46923 URL: https://issues.apache.org/jira/browse/SPARK-46923 Project: Spark

[jira] [Resolved] (SPARK-46921) Move `ProblemFilters` that do not belong to defaultExcludes to `v40excludes`.

[ https://issues.apache.org/jira/browse/SPARK-46921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46921. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44952

[jira] [Updated] (SPARK-46473) Reuse `getPartitionedFile` method

[ https://issues.apache.org/jira/browse/SPARK-46473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-46473: - Priority: Trivial (was: Minor) > Reuse `getPartitionedFile` method >

[jira] [Created] (SPARK-46922) better handling for runtime user errors

Wenchen Fan created SPARK-46922: --- Summary: better handling for runtime user errors Key: SPARK-46922 URL: https://issues.apache.org/jira/browse/SPARK-46922 Project: Spark Issue Type:

[jira] [Commented] (SPARK-46747) Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1

[ https://issues.apache.org/jira/browse/SPARK-46747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17812345#comment-17812345 ] Bala Bellam commented on SPARK-46747: - Thanks [~yao] . So, does this get released to 3.3 & higher? 

[jira] [Updated] (SPARK-46919) Upgrade `grpcio*` to 1.60.0 and `grpc-java` to 1.61.0

[ https://issues.apache.org/jira/browse/SPARK-46919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46919: --- Labels: pull-request-available (was: ) > Upgrade `grpcio*` to 1.60.0 and `grpc-java` to

[jira] [Commented] (SPARK-46747) Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1

[ https://issues.apache.org/jira/browse/SPARK-46747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17812341#comment-17812341 ] Kent Yao commented on SPARK-46747: -- according to the release policies of Spark，patches never get merged

[jira] [Updated] (SPARK-46747) Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1

[ https://issues.apache.org/jira/browse/SPARK-46747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bala Bellam updated SPARK-46747: Fix Version/s: (was: 2.3.2) > Too Many Shared Locks due to

[jira] [Updated] (SPARK-46747) Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1

[ https://issues.apache.org/jira/browse/SPARK-46747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bala Bellam updated SPARK-46747: Fix Version/s: 2.3.2 > Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1

[jira] [Commented] (SPARK-46747) Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1

[ https://issues.apache.org/jira/browse/SPARK-46747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17812302#comment-17812302 ] Bala Bellam commented on SPARK-46747: - Thank you very much [~yao] . Sure, I can provide those number

[jira] [Updated] (SPARK-46747) Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1

[ https://issues.apache.org/jira/browse/SPARK-46747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-46747: - Target Version/s: (was: 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.2.0, 2.2.1, 2.2.2, 2.2.3,

[jira] [Updated] (SPARK-46747) Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1

[ https://issues.apache.org/jira/browse/SPARK-46747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-46747: - Priority: Major (was: Critical) > Too Many Shared Locks due to PostgresDialect.getTableExistsQuery -

[jira] [Resolved] (SPARK-46747) Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1

[ https://issues.apache.org/jira/browse/SPARK-46747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-46747. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44948

[jira] [Assigned] (SPARK-46747) Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1

[ https://issues.apache.org/jira/browse/SPARK-46747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-46747: Assignee: Kent Yao > Too Many Shared Locks due to PostgresDialect.getTableExistsQuery - LIMIT 1

[jira] [Resolved] (SPARK-46918) Replace self-defined variables with Hadoop ContainerExitStatus

[ https://issues.apache.org/jira/browse/SPARK-46918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-46918. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44950

[jira] [Assigned] (SPARK-46918) Replace self-defined variables with Hadoop ContainerExitStatus

[ https://issues.apache.org/jira/browse/SPARK-46918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-46918: Assignee: Cheng Pan > Replace self-defined variables with Hadoop ContainerExitStatus >

[jira] [Created] (SPARK-46919) Upgrade `grpcio*` to 1.60.0 and `grpc-java` to 1.61.0

Yang Jie created SPARK-46919: Summary: Upgrade `grpcio*` to 1.60.0 and `grpc-java` to 1.61.0 Key: SPARK-46919 URL: https://issues.apache.org/jira/browse/SPARK-46919 Project: Spark Issue Type:

[jira] [Updated] (SPARK-46917) [SPARK-46753][PYTHON][TESTS] Fix pypy3 python test

[ https://issues.apache.org/jira/browse/SPARK-46917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-46917: Summary: [SPARK-46753][PYTHON][TESTS] Fix pypy3 python test (was: Improve merge_spark_pr.py) >

[jira] [Updated] (SPARK-46918) Replace self-defined variables with Hadoop ContainerExitStatus

[ https://issues.apache.org/jira/browse/SPARK-46918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46918: --- Labels: pull-request-available (was: ) > Replace self-defined variables with Hadoop

[jira] [Created] (SPARK-46918) Replace self-defined variables with Hadoop ContainerExitStatus

Cheng Pan created SPARK-46918: - Summary: Replace self-defined variables with Hadoop ContainerExitStatus Key: SPARK-46918 URL: https://issues.apache.org/jira/browse/SPARK-46918 Project: Spark

[jira] [Assigned] (SPARK-46890) CSV fails on a column with default and without enforcing schema

[ https://issues.apache.org/jira/browse/SPARK-46890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46890: -- Assignee: (was: Apache Spark) > CSV fails on a column with default and without

[jira] [Assigned] (SPARK-46890) CSV fails on a column with default and without enforcing schema

[ https://issues.apache.org/jira/browse/SPARK-46890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46890: -- Assignee: Apache Spark > CSV fails on a column with default and without enforcing

[jira] [Updated] (SPARK-46851) Remove buf version information from the doc contributing.rst

[ https://issues.apache.org/jira/browse/SPARK-46851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-46851: Summary: Remove buf version information from the doc contributing.rst (was: Upgrade `buf` to