[jira] [Assigned] (SPARK-46734) Combine pip installations for lint and doc respectively

2024-01-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46734: - Assignee: Ruifeng Zheng > Combine pip installations for lint and doc respectively >

[jira] [Resolved] (SPARK-46734) Combine pip installations for lint and doc respectively

2024-01-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46734. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44754

[jira] [Updated] (SPARK-46732) Propagate JobArtifactSet to broadcast execution thread

2024-01-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-46732: - Fix Version/s: 4.0.0 3.5.1 > Propagate JobArtifactSet to broadcast execution

[jira] [Created] (SPARK-46743) Count bug introduced for scalar subquery when using TEMPORARY VIEW, as compared to using table

2024-01-16 Thread Andy Lam (Jira)
Andy Lam created SPARK-46743: Summary: Count bug introduced for scalar subquery when using TEMPORARY VIEW, as compared to using table Key: SPARK-46743 URL: https://issues.apache.org/jira/browse/SPARK-46743

[jira] [Assigned] (SPARK-46742) Add ORC compression tests for `hive` module OrcFileFormat

2024-01-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46742: - Assignee: Dongjoon Hyun > Add ORC compression tests for `hive` module OrcFileFormat >

[jira] [Resolved] (SPARK-46742) Add ORC compression tests for `hive` module OrcFileFormat

2024-01-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46742. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44765

[jira] [Assigned] (SPARK-46737) Use the default ORC compression in OrcReadBenchmark

2024-01-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46737: - Assignee: Dongjoon Hyun > Use the default ORC compression in OrcReadBenchmark >

[jira] [Resolved] (SPARK-46737) Use the default ORC compression in OrcReadBenchmark

2024-01-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46737. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44761

[jira] [Created] (SPARK-46742) Add ORC compression tests for `hive` module OrcFileFormat

2024-01-16 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-46742: - Summary: Add ORC compression tests for `hive` module OrcFileFormat Key: SPARK-46742 URL: https://issues.apache.org/jira/browse/SPARK-46742 Project: Spark

[jira] [Updated] (SPARK-46742) Add ORC compression tests for `hive` module OrcFileFormat

2024-01-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46742: --- Labels: pull-request-available (was: ) > Add ORC compression tests for `hive` module

[jira] [Updated] (SPARK-46741) CacheTable AsSelect should inherit from CTEInChildren to make sure it can be matched

2024-01-16 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-46741: -- Description: Current code since CaheTableAsSelelct not inherit CETInChildren,  still return

[jira] [Updated] (SPARK-46741) CacheTable AsSelect should inherit from CTEInChildren to make sure it can be matched

2024-01-16 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-46741: -- Attachment: image-2024-01-17-11-48-28-867.png > CacheTable AsSelect should inherit from CTEInChildren

[jira] [Created] (SPARK-46741) CacheTable AsSelect should inherit from CTEInChildren to make sure it can be matched

2024-01-16 Thread angerszhu (Jira)
angerszhu created SPARK-46741: - Summary: CacheTable AsSelect should inherit from CTEInChildren to make sure it can be matched Key: SPARK-46741 URL: https://issues.apache.org/jira/browse/SPARK-46741

[jira] [Updated] (SPARK-46740) Only convert to ParquetFileScan for normal Hive Parquet table

2024-01-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46740: --- Labels: pull-request-available (was: ) > Only convert to ParquetFileScan for normal Hive

[jira] [Created] (SPARK-46740) Only convert to ParquetFileScan for normal Hive Parquet table

2024-01-16 Thread Rui Wang (Jira)
Rui Wang created SPARK-46740: Summary: Only convert to ParquetFileScan for normal Hive Parquet table Key: SPARK-46740 URL: https://issues.apache.org/jira/browse/SPARK-46740 Project: Spark Issue

[jira] [Assigned] (SPARK-46612) Clickhouse's JDBC throws `java.lang.IllegalArgumentException: Unknown data type: string` when write array string with Apache Spark scala

2024-01-16 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-46612: Assignee: Nguyen Phan Huy > Clickhouse's JDBC throws `java.lang.IllegalArgumentException:

[jira] [Resolved] (SPARK-46612) Clickhouse's JDBC throws `java.lang.IllegalArgumentException: Unknown data type: string` when write array string with Apache Spark scala

2024-01-16 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-46612. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44459

[jira] [Commented] (SPARK-46738) `Cast` of pyspark displayed different results between Regular Spark and Spark Connect

2024-01-16 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807499#comment-17807499 ] BingKun Pan commented on SPARK-46738: - I work on it. > `Cast` of pyspark displayed different

[jira] [Updated] (SPARK-46738) `Cast` of pyspark displayed different results between Regular Spark and Spark Connect

2024-01-16 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-46738: Description: The following doctest will throw an error in the tests of the pyspark-connect

[jira] [Created] (SPARK-46739) Add an error class for unsupported method calls

2024-01-16 Thread Max Gekk (Jira)
Max Gekk created SPARK-46739: Summary: Add an error class for unsupported method calls Key: SPARK-46739 URL: https://issues.apache.org/jira/browse/SPARK-46739 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-46738) `Cast` of pyspark displayed different results between Regular Spark and Spark Connect

2024-01-16 Thread BingKun Pan (Jira)
BingKun Pan created SPARK-46738: --- Summary: `Cast` of pyspark displayed different results between Regular Spark and Spark Connect Key: SPARK-46738 URL: https://issues.apache.org/jira/browse/SPARK-46738

[jira] [Updated] (SPARK-46737) Use the default ORC compression in OrcReadBenchmark

2024-01-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46737: --- Labels: pull-request-available (was: ) > Use the default ORC compression in

[jira] [Created] (SPARK-46737) Use the default ORC compression in OrcReadBenchmark

2024-01-16 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-46737: - Summary: Use the default ORC compression in OrcReadBenchmark Key: SPARK-46737 URL: https://issues.apache.org/jira/browse/SPARK-46737 Project: Spark Issue

[jira] [Assigned] (SPARK-46732) Propagate JobArtifactSet to broadcast execution thread

2024-01-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-46732: Assignee: xie shuiahu > Propagate JobArtifactSet to broadcast execution thread >

[jira] [Resolved] (SPARK-46732) Propagate JobArtifactSet to broadcast execution thread

2024-01-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-46732. -- Resolution: Fixed Fixed in https://github.com/apache/spark/pull/44753 > Propagate

[jira] [Assigned] (SPARK-46730) Refine docstring of `str_to_map/map_filter/map_zip_with`

2024-01-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-46730: Assignee: Yang Jie > Refine docstring of `str_to_map/map_filter/map_zip_with` >

[jira] [Resolved] (SPARK-46730) Refine docstring of `str_to_map/map_filter/map_zip_with`

2024-01-16 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-46730. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44747

[jira] [Resolved] (SPARK-46735) `pyspark.sql.tests.test_group` should skip Pandas/PyArrow tests if not available

2024-01-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46735. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44759

[jira] [Assigned] (SPARK-46735) `pyspark.sql.tests.test_group` should skip Pandas/PyArrow tests if not available

2024-01-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46735: - Assignee: Dongjoon Hyun > `pyspark.sql.tests.test_group` should skip Pandas/PyArrow

[jira] [Created] (SPARK-46736) Retain empty protobuf message in schema for rpotobuf connector

2024-01-16 Thread Chaoqin Li (Jira)
Chaoqin Li created SPARK-46736: -- Summary: Retain empty protobuf message in schema for rpotobuf connector Key: SPARK-46736 URL: https://issues.apache.org/jira/browse/SPARK-46736 Project: Spark

[jira] [Updated] (SPARK-46735) pyspark.sql.tests.test_group should skip Pandas/PyArrow tests if not available

2024-01-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46735: -- Summary: pyspark.sql.tests.test_group should skip Pandas/PyArrow tests if not available

[jira] [Updated] (SPARK-46735) `pyspark.sql.tests.test_group` should skip Pandas/PyArrow tests if not available

2024-01-16 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46735: -- Summary: `pyspark.sql.tests.test_group` should skip Pandas/PyArrow tests if not available

[jira] [Updated] (SPARK-46735) `pyspark.sql.tests.test_group` should skip Pandas tests if not available

2024-01-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46735: --- Labels: pull-request-available (was: ) > `pyspark.sql.tests.test_group` should skip Pandas

[jira] [Created] (SPARK-46735) `pyspark.sql.tests.test_group` should skip Pandas tests if not available

2024-01-16 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-46735: - Summary: `pyspark.sql.tests.test_group` should skip Pandas tests if not available Key: SPARK-46735 URL: https://issues.apache.org/jira/browse/SPARK-46735 Project:

[jira] [Resolved] (SPARK-46727) Port classifyException() in JDBC dialects on error classes

2024-01-16 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-46727. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44739

[jira] [Updated] (SPARK-43919) Extract JSON functionality out of Row

2024-01-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-43919: --- Labels: pull-request-available (was: ) > Extract JSON functionality out of Row >

[jira] [Updated] (SPARK-46733) Simplify the ContextCleaner|BlockManager by the exit operation only depend on interrupt thread

2024-01-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46733: --- Labels: pull-request-available (was: ) > Simplify the ContextCleaner|BlockManager by the

[jira] [Updated] (SPARK-46659) Add customizable TaskScheduling param, to avoid randomly choosing executor for tasks, and downscale on low micro-batches activity

2024-01-16 Thread Arnaud Nauwynck (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arnaud Nauwynck updated SPARK-46659: Description: When using dynamicAllocation (but not spark.decommission.enabled=true) with

[jira] [Updated] (SPARK-46734) Combine pip installations for lint and doc respectively

2024-01-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46734: --- Labels: pull-request-available (was: ) > Combine pip installations for lint and doc

[jira] [Created] (SPARK-46734) Combine pip installations for lint and doc respectively

2024-01-16 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-46734: - Summary: Combine pip installations for lint and doc respectively Key: SPARK-46734 URL: https://issues.apache.org/jira/browse/SPARK-46734 Project: Spark

[jira] [Commented] (SPARK-33545) Support Fallback Storage during Worker decommission

2024-01-16 Thread mahesh kumar behera (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807166#comment-17807166 ] mahesh kumar behera commented on SPARK-33545: - [~dongjoon]  As per this PR, the shuffle

[jira] [Resolved] (SPARK-46729) Withdraw the recommendation of using Concurrent Mark Sweep (CMS) Garbage Collector

2024-01-16 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-46729. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44746

[jira] [Assigned] (SPARK-46729) Withdraw the recommendation of using Concurrent Mark Sweep (CMS) Garbage Collector

2024-01-16 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-46729: Assignee: Kent Yao > Withdraw the recommendation of using Concurrent Mark Sweep (CMS) Garbage >

[jira] [Created] (SPARK-46733) Simplify the ContextCleaner|BlockManager by the exit operation only depend on interrupt thread

2024-01-16 Thread Yang Jie (Jira)
Yang Jie created SPARK-46733: Summary: Simplify the ContextCleaner|BlockManager by the exit operation only depend on interrupt thread Key: SPARK-46733 URL: https://issues.apache.org/jira/browse/SPARK-46733

[jira] [Updated] (SPARK-46727) Port classifyException() in JDBC dialects on error classes

2024-01-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46727: --- Labels: pull-request-available (was: ) > Port classifyException() in JDBC dialects on

[jira] [Updated] (SPARK-46732) Propagate JobArtifactSet to broadcast execution thread

2024-01-16 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46732: --- Labels: pull-request-available (was: ) > Propagate JobArtifactSet to broadcast execution

[jira] [Created] (SPARK-46732) Propagate ArtifactSet to broadcast execution thread

2024-01-16 Thread xie shuiahu (Jira)
xie shuiahu created SPARK-46732: --- Summary: Propagate ArtifactSet to broadcast execution thread Key: SPARK-46732 URL: https://issues.apache.org/jira/browse/SPARK-46732 Project: Spark Issue

[jira] [Updated] (SPARK-46732) Propagate JobArtifactSet to broadcast execution thread

2024-01-16 Thread xie shuiahu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xie shuiahu updated SPARK-46732: Summary: Propagate JobArtifactSet to broadcast execution thread (was: Propagate ArtifactSet to