[jira] [Updated] (SPARK-46522) Block Python data source registration with name conflicts

2023-12-26 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46522: --- Labels: pull-request-available (was: ) > Block Python data source registration with name

[jira] [Created] (SPARK-46522) Block Python data source registration with name conflicts

2023-12-26 Thread Allison Wang (Jira)
Allison Wang created SPARK-46522: Summary: Block Python data source registration with name conflicts Key: SPARK-46522 URL: https://issues.apache.org/jira/browse/SPARK-46522 Project: Spark

[jira] [Updated] (SPARK-46521) Refine docstring of `array_compact/array_distinct/array_remove`

2023-12-26 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46521: --- Labels: pull-request-available (was: ) > Refine docstring of

[jira] [Resolved] (SPARK-46517) Reorganize `IndexingTest`

2023-12-26 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-46517. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44502

[jira] [Updated] (SPARK-46516) autoBroadcastJoinThreshold compared to plan.statistics not a table size

2023-12-26 Thread Guram Savinov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guram Savinov updated SPARK-46516: -- Description: >From the docs: spark.sql.autoBroadcastJoinThreshold - Configures the maximum

[jira] [Created] (SPARK-46521) Refine docstring of `array_compact/array_distinct/array_remove`

2023-12-26 Thread Yang Jie (Jira)
Yang Jie created SPARK-46521: Summary: Refine docstring of `array_compact/array_distinct/array_remove` Key: SPARK-46521 URL: https://issues.apache.org/jira/browse/SPARK-46521 Project: Spark

[jira] [Updated] (SPARK-46520) Support overwrite mode for Python data source write

2023-12-26 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46520: --- Labels: pull-request-available (was: ) > Support overwrite mode for Python data source

[jira] [Updated] (SPARK-45917) Statically register Python Data Source

2023-12-26 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45917: --- Labels: pull-request-available (was: ) > Statically register Python Data Source >

[jira] [Commented] (SPARK-38388) Repartition + Stage retries could lead to incorrect data

2023-12-26 Thread Wei Lu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17800682#comment-17800682 ] Wei Lu commented on SPARK-38388: We had the same problem(using Spark 3.2.1),is there any plan to fix the

[jira] [Created] (SPARK-46520) Support overwrite mode for Python data source write

2023-12-26 Thread Allison Wang (Jira)
Allison Wang created SPARK-46520: Summary: Support overwrite mode for Python data source write Key: SPARK-46520 URL: https://issues.apache.org/jira/browse/SPARK-46520 Project: Spark Issue

[jira] [Updated] (SPARK-46519) Clear unused error classes from error-classes.json file

2023-12-26 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46519: --- Labels: pull-request-available (was: ) > Clear unused error classes from

[jira] [Created] (SPARK-46519) Clear unused error classes from error-classes.json file

2023-12-26 Thread BingKun Pan (Jira)
BingKun Pan created SPARK-46519: --- Summary: Clear unused error classes from error-classes.json file Key: SPARK-46519 URL: https://issues.apache.org/jira/browse/SPARK-46519 Project: Spark Issue

[jira] [Updated] (SPARK-43338) Support modify the SESSION_CATALOG_NAME value

2023-12-26 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] melin updated SPARK-43338: -- Description: {code:java} private[sql] object CatalogManager { val SESSION_CATALOG_NAME: String =

[jira] [Updated] (SPARK-43338) Support modify the SESSION_CATALOG_NAME value

2023-12-26 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] melin updated SPARK-43338: -- Attachment: image-2023-12-27-09-55-55-693.png > Support modify the SESSION_CATALOG_NAME value >

[jira] [Updated] (SPARK-46518) Support for copy from write compatible postgresql databases (pg, redshift, snowflake, gauss)

2023-12-26 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] melin updated SPARK-46518: -- Description: Now many databases are compatible with pg syntax and support copy from syntax. The copy form

[jira] [Updated] (SPARK-46518) Support for copy from write compatible postgresql databases (pg, redshift, snowflake, gauss)

2023-12-26 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] melin updated SPARK-46518: -- Attachment: image-2023-12-27-09-44-19-292.png > Support for copy from write compatible postgresql databases

[jira] [Created] (SPARK-46518) Support for copy from write compatible postgresql databases (pg, redshift, snowflake, gauss)

2023-12-26 Thread melin (Jira)
melin created SPARK-46518: - Summary: Support for copy from write compatible postgresql databases (pg, redshift, snowflake, gauss) Key: SPARK-46518 URL: https://issues.apache.org/jira/browse/SPARK-46518

[jira] [Resolved] (SPARK-46508) Upgrade Jackson to 2.16.1

2023-12-26 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46508. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44494

[jira] [Updated] (SPARK-46517) Reorganize `IndexingTest`

2023-12-26 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46517: --- Labels: pull-request-available (was: ) > Reorganize `IndexingTest` >

[jira] [Created] (SPARK-46517) Reorganize `IndexingTest`

2023-12-26 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-46517: - Summary: Reorganize `IndexingTest` Key: SPARK-46517 URL: https://issues.apache.org/jira/browse/SPARK-46517 Project: Spark Issue Type: Sub-task

[jira] [Resolved] (SPARK-46513) Move `BasicIndexingTests` to `pyspark.pandas.tests.indexes.*`

2023-12-26 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-46513. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44499

[jira] [Assigned] (SPARK-46513) Move `BasicIndexingTests` to `pyspark.pandas.tests.indexes.*`

2023-12-26 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-46513: Assignee: Ruifeng Zheng > Move `BasicIndexingTests` to `pyspark.pandas.tests.indexes.*`

[jira] [Updated] (SPARK-46516) autoBroadcastJoinThreshold compared to plan.statistics not a table size

2023-12-26 Thread Guram Savinov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guram Savinov updated SPARK-46516: -- Description: >From the docs: spark.sql.autoBroadcastJoinThreshold - Configures the maximum

[jira] [Updated] (SPARK-46516) autoBroadcastJoinThreshold compared to plan.statistics not a table size

2023-12-26 Thread Guram Savinov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guram Savinov updated SPARK-46516: -- Description: >From the docs: spark.sql.autoBroadcastJoinThreshold - Configures the maximum

[jira] [Updated] (SPARK-46516) autoBroadcastJoinThreshold compared to plan.statistics not a table size

2023-12-26 Thread Guram Savinov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guram Savinov updated SPARK-46516: -- Description: >From the docs: spark.sql.autoBroadcastJoinThreshold - Configures the maximum

[jira] [Updated] (SPARK-46516) autoBroadcastJoinThreshold compared to plan.statistics not a table size

2023-12-26 Thread Guram Savinov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guram Savinov updated SPARK-46516: -- Description: >From the docs: spark.sql.autoBroadcastJoinThreshold - Configures the maximum

[jira] [Updated] (SPARK-46516) autoBroadcastJoinThreshold compared to plan.statistics not a table size

2023-12-26 Thread Guram Savinov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guram Savinov updated SPARK-46516: -- Description: >From the docs: spark.sql.autoBroadcastJoinThreshold - Configures the maximum

[jira] [Updated] (SPARK-46516) autoBroadcastJoinThreshold compared to plan.statistics not a table size

2023-12-26 Thread Guram Savinov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guram Savinov updated SPARK-46516: -- Issue Type: Bug (was: Documentation) > autoBroadcastJoinThreshold compared to

[jira] [Created] (SPARK-46516) autoBroadcastJoinThreshold compared to plan.statistics not a table size

2023-12-26 Thread Guram Savinov (Jira)
Guram Savinov created SPARK-46516: - Summary: autoBroadcastJoinThreshold compared to plan.statistics not a table size Key: SPARK-46516 URL: https://issues.apache.org/jira/browse/SPARK-46516 Project:

[jira] [Assigned] (SPARK-46506) Refine docstring of `array_intersect/array_union/array_except`

2023-12-26 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-46506: Assignee: Yang Jie > Refine docstring of `array_intersect/array_union/array_except` >

[jira] [Resolved] (SPARK-46506) Refine docstring of `array_intersect/array_union/array_except`

2023-12-26 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-46506. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44490

[jira] [Commented] (SPARK-46192) failed to insert the table using the default value of union

2023-12-26 Thread zengxl (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17800471#comment-17800471 ] zengxl commented on SPARK-46192: {code:java} create table test_spark_3(k string default null,v int

[jira] [Updated] (SPARK-46514) Fix HiveMetastoreLazyInitializationSuite

2023-12-26 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46514: --- Labels: pull-request-available (was: ) > Fix HiveMetastoreLazyInitializationSuite >

[jira] [Updated] (SPARK-46513) Move `BasicIndexingTests` to `pyspark.pandas.tests.indexes.*`

2023-12-26 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46513: --- Labels: pull-request-available (was: ) > Move `BasicIndexingTests` to

[jira] [Updated] (SPARK-46512) Optimize shuffle reading when both sort and combine are used.

2023-12-26 Thread Chenyu Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chenyu Zheng updated SPARK-46512: - Description: After the shuffle reader obtains the block, it will first perform a combine

[jira] [Created] (SPARK-46514) Fix HiveMetastoreLazyInitializationSuite

2023-12-26 Thread Kent Yao (Jira)
Kent Yao created SPARK-46514: Summary: Fix HiveMetastoreLazyInitializationSuite Key: SPARK-46514 URL: https://issues.apache.org/jira/browse/SPARK-46514 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-46513) Move `BasicIndexingTests` to `pyspark.pandas.tests.indexes.*`

2023-12-26 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-46513: - Summary: Move `BasicIndexingTests` to `pyspark.pandas.tests.indexes.*` Key: SPARK-46513 URL: https://issues.apache.org/jira/browse/SPARK-46513 Project: Spark

[jira] [Assigned] (SPARK-46510) Spark shell log filter should be applied to all AbstractAppender

2023-12-26 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46510: -- Assignee: (was: Apache Spark) > Spark shell log filter should be applied to all

[jira] [Assigned] (SPARK-46510) Spark shell log filter should be applied to all AbstractAppender

2023-12-26 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46510: -- Assignee: Apache Spark > Spark shell log filter should be applied to all

[jira] [Assigned] (SPARK-46510) Spark shell log filter should be applied to all AbstractAppender

2023-12-26 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46510: -- Assignee: Apache Spark > Spark shell log filter should be applied to all

[jira] [Assigned] (SPARK-46510) Spark shell log filter should be applied to all AbstractAppender

2023-12-26 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46510: -- Assignee: (was: Apache Spark) > Spark shell log filter should be applied to all

[jira] [Updated] (SPARK-46460) The filter of partition including cast function may lead the partition pruning to disable

2023-12-26 Thread Zhou Tong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhou Tong updated SPARK-46460: -- Attachment: SPARK-46460.patch > The filter of partition including cast function may lead the

[jira] [Resolved] (SPARK-46511) Optimize spark jdbc write speed with Multi-Row Inserts

2023-12-26 Thread melin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] melin resolved SPARK-46511. --- Resolution: Fixed > Optimize spark jdbc write speed with Multi-Row Inserts >

[jira] [Updated] (SPARK-46498) Remove `shuffleServiceEnabled` from `o.a.spark.util.Utils#getConfiguredLocalDirs`

2023-12-26 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46498: -- Summary: Remove `shuffleServiceEnabled` from `o.a.spark.util.Utils#getConfiguredLocalDirs`

[jira] [Assigned] (SPARK-46498) Remove an unused local variables from `o.a.spark.util.Utils#getConfiguredLocalDirs`

2023-12-26 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46498: - Assignee: Yang Jie > Remove an unused local variables from >

[jira] [Resolved] (SPARK-46498) Remove an unused local variables from `o.a.spark.util.Utils#getConfiguredLocalDirs`

2023-12-26 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46498. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44475

[jira] [Resolved] (SPARK-46371) Clean up outdated items in `.rat-excludes`

2023-12-26 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46371. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44293

[jira] [Assigned] (SPARK-46371) Clean up outdated items in `.rat-excludes`

2023-12-26 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46371: - Assignee: BingKun Pan > Clean up outdated items in `.rat-excludes` >

[jira] [Updated] (SPARK-45914) Support `commit` and `abort` API for Python data source write

2023-12-26 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45914: --- Labels: pull-request-available (was: ) > Support `commit` and `abort` API for Python data