[jira] [Assigned] (SPARK-47155) Fix incorrect error class in create_data_source.py

[ https://issues.apache.org/jira/browse/SPARK-47155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-47155: Assignee: Chaoqin Li > Fix incorrect error class in create_data_source.py >

[jira] [Resolved] (SPARK-47155) Fix incorrect error class in create_data_source.py

[ https://issues.apache.org/jira/browse/SPARK-47155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-47155. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45306

[jira] [Commented] (SPARK-42694) Data duplication and loss occur after executing 'insert overwrite...' in Spark 3.1.1

[ https://issues.apache.org/jira/browse/SPARK-42694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824279#comment-17824279 ] liukai commented on SPARK-42694: I also encountered this situation in spark3.1.1. Setting the parameter

[jira] [Updated] (SPARK-47315) tempView created in createTempView UT is not cleaned up

[ https://issues.apache.org/jira/browse/SPARK-47315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47315: --- Labels: pull-request-available (was: ) > tempView created in createTempView UT is not

[jira] [Resolved] (SPARK-47305) PruneFilters incorrectly tags isStreaming flag when replacing child of Filter with LocalRelation

[ https://issues.apache.org/jira/browse/SPARK-47305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-47305. -- Fix Version/s: 3.4.3 3.5.2 4.0.0 Resolution:

[jira] [Assigned] (SPARK-47305) PruneFilters incorrectly tags isStreaming flag when replacing child of Filter with LocalRelation

[ https://issues.apache.org/jira/browse/SPARK-47305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-47305: Assignee: Jungtaek Lim > PruneFilters incorrectly tags isStreaming flag when replacing

[jira] [Commented] (SPARK-47315) tempView created in createTempView UT is not cleaned up

[ https://issues.apache.org/jira/browse/SPARK-47315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824259#comment-17824259 ] Zhen Wang commented on SPARK-47315: --- https://github.com/apache/spark/pull/45417 > tempView created in

[jira] [Updated] (SPARK-47315) tempView created in createTempView UT is not cleaned up

[ https://issues.apache.org/jira/browse/SPARK-47315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhen Wang updated SPARK-47315: -- Description: tempView created in createTempView UT (which is in DatasetSuite.scala) is not cleaned up

[jira] [Created] (SPARK-47315) tempView created in createTempView UT is not cleaned up

Zhen Wang created SPARK-47315: - Summary: tempView created in createTempView UT is not cleaned up Key: SPARK-47315 URL: https://issues.apache.org/jira/browse/SPARK-47315 Project: Spark Issue

[jira] [Updated] (SPARK-43124) Dataset.show should not trigger job execution on CommandResults

[ https://issues.apache.org/jira/browse/SPARK-43124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-43124: --- Labels: pull-request-available (was: ) > Dataset.show should not trigger job execution on

[jira] [Resolved] (SPARK-47238) Reduce executor memory usage by making generated code in WSCG a broadcast variable

[ https://issues.apache.org/jira/browse/SPARK-47238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-47238. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45348

[jira] [Resolved] (SPARK-47311) Suppress Python exceptions where PySpark is not in the Python path

[ https://issues.apache.org/jira/browse/SPARK-47311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-47311. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45414

[jira] [Assigned] (SPARK-47311) Suppress Python exceptions where PySpark is not in the Python path

[ https://issues.apache.org/jira/browse/SPARK-47311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-47311: Assignee: Hyukjin Kwon > Suppress Python exceptions where PySpark is not in the Python

[jira] [Assigned] (SPARK-46988) proto message abbreviation should support map fields

[ https://issues.apache.org/jira/browse/SPARK-46988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-46988: - Assignee: Ruifeng Zheng > proto message abbreviation should support map fields >

[jira] [Resolved] (SPARK-46988) proto message abbreviation should support map fields

[ https://issues.apache.org/jira/browse/SPARK-46988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-46988. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45413

[jira] [Updated] (SPARK-47314) [DOC] Correct the ExternalSorter#writePartitionedMapOutput method comment

[ https://issues.apache.org/jira/browse/SPARK-47314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47314: --- Labels: pull-request-available (was: ) > [DOC] Correct the

[jira] [Created] (SPARK-47314) [DOC] Correct the ExternalSorter#writePartitionedMapOutput method comment

Binjie Yang created SPARK-47314: --- Summary: [DOC] Correct the ExternalSorter#writePartitionedMapOutput method comment Key: SPARK-47314 URL: https://issues.apache.org/jira/browse/SPARK-47314 Project:

[jira] [Created] (SPARK-47313) scala.MatchError should be treated as internal error

Wenchen Fan created SPARK-47313: --- Summary: scala.MatchError should be treated as internal error Key: SPARK-47313 URL: https://issues.apache.org/jira/browse/SPARK-47313 Project: Spark Issue

[jira] [Updated] (SPARK-47311) Suppress Python exceptions where PySpark is not in the Python path

[ https://issues.apache.org/jira/browse/SPARK-47311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47311: --- Labels: pull-request-available (was: ) > Suppress Python exceptions where PySpark is not

[jira] [Updated] (SPARK-47311) Suppress Python exceptions where PySpark is not in the Python path

[ https://issues.apache.org/jira/browse/SPARK-47311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-47311: - Description: {code} scala> sql("create table t(i int) using json") 24/03/06 16:09:44 WARN

[jira] [Updated] (SPARK-47311) Suppress Python exceptions where PySpark is not in the Python path

[ https://issues.apache.org/jira/browse/SPARK-47311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-47311: - Summary: Suppress Python exceptions where PySpark is not in the Python path (was: Surpress

[jira] [Updated] (SPARK-47311) Surpress Python exceptions where PySpark is not in the Python path

[ https://issues.apache.org/jira/browse/SPARK-47311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-47311: - Priority: Minor (was: Major) > Surpress Python exceptions where PySpark is not in the Python

[jira] [Created] (SPARK-47311) Surpress Python exceptions where PySpark is not in the Python path

Hyukjin Kwon created SPARK-47311: Summary: Surpress Python exceptions where PySpark is not in the Python path Key: SPARK-47311 URL: https://issues.apache.org/jira/browse/SPARK-47311 Project: Spark

[jira] [Updated] (SPARK-46988) proto message abbreviation should support map fields

[ https://issues.apache.org/jira/browse/SPARK-46988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46988: --- Labels: pull-request-available (was: ) > proto message abbreviation should support map

[jira] [Assigned] (SPARK-47271) Explain importance of statistics on SQL performance tuning page

[ https://issues.apache.org/jira/browse/SPARK-47271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-47271: Assignee: Nicholas Chammas > Explain importance of statistics on SQL performance tuning

[jira] [Resolved] (SPARK-47271) Explain importance of statistics on SQL performance tuning page

[ https://issues.apache.org/jira/browse/SPARK-47271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-47271. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45374

[jira] [Assigned] (SPARK-47303) Restructuring MasterSuite

[ https://issues.apache.org/jira/browse/SPARK-47303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-47303: Assignee: Hyukjin Kwon > Restructuring MasterSuite > - > >

[jira] [Resolved] (SPARK-47303) Restructuring MasterSuite

[ https://issues.apache.org/jira/browse/SPARK-47303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-47303. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45366

[jira] [Updated] (SPARK-47309) [XML] Add schema inference unit tests

[ https://issues.apache.org/jira/browse/SPARK-47309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47309: --- Labels: pull-request-available (was: ) > [XML] Add schema inference unit tests >

[jira] [Created] (SPARK-47309) [XML] Add schema inference unit tests

Shujing Yang created SPARK-47309: Summary: [XML] Add schema inference unit tests Key: SPARK-47309 URL: https://issues.apache.org/jira/browse/SPARK-47309 Project: Spark Issue Type:

[jira] [Created] (SPARK-47308) LATERAL regresses correlation name resolution

Serge Rielau created SPARK-47308: Summary: LATERAL regresses correlation name resolution Key: SPARK-47308 URL: https://issues.apache.org/jira/browse/SPARK-47308 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-39771) If spark.default.parallelism is unset, RDD defaultPartitioner may pick a value that is too large to successfully run

[ https://issues.apache.org/jira/browse/SPARK-39771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanjian Li resolved SPARK-39771. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45266

[jira] [Commented] (SPARK-47307) Spark 3.3 produces invalid base64

[ https://issues.apache.org/jira/browse/SPARK-47307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824115#comment-17824115 ] Willi Raschkowski commented on SPARK-47307: --- The behavior change is as follows: * Spark 3.2,

[jira] [Updated] (SPARK-47307) Spark 3.3 breaks base64

[ https://issues.apache.org/jira/browse/SPARK-47307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Willi Raschkowski updated SPARK-47307: -- Description: SPARK-37820 was introduced in Spark 3.3 and breaks behavior of

[jira] [Commented] (SPARK-39441) Speed up DeduplicateRelations

[ https://issues.apache.org/jira/browse/SPARK-39441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824102#comment-17824102 ] Asif commented on SPARK-39441: -- this issue should be resolved by the PR for ticket

[jira] [Comment Edited] (SPARK-39441) Speed up DeduplicateRelations

[ https://issues.apache.org/jira/browse/SPARK-39441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824102#comment-17824102 ] Asif edited comment on SPARK-39441 at 3/6/24 5:33 PM: -- this issue should be

[jira] [Created] (SPARK-47307) Spark 3.3 breaks base64

Willi Raschkowski created SPARK-47307: - Summary: Spark 3.3 breaks base64 Key: SPARK-47307 URL: https://issues.apache.org/jira/browse/SPARK-47307 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-47254) Assign names to the error classes _LEGACY_ERROR_TEMP_325[1-9]

[ https://issues.apache.org/jira/browse/SPARK-47254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824092#comment-17824092 ] Stefan Bukorovic commented on SPARK-47254: -- I want to work on this issue. Raised a PR for

[jira] [Updated] (SPARK-47254) Assign names to the error classes _LEGACY_ERROR_TEMP_325[1-9]

[ https://issues.apache.org/jira/browse/SPARK-47254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47254: --- Labels: pull-request-available starter (was: starter) > Assign names to the error classes

[jira] [Resolved] (SPARK-47304) Split DataFrameSuite

[ https://issues.apache.org/jira/browse/SPARK-47304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-47304. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45392

[jira] [Created] (SPARK-47306) Explicit repartitioning is ignored when reading bucketed table but with bucketing disabled

Arttu Voutilainen created SPARK-47306: - Summary: Explicit repartitioning is ignored when reading bucketed table but with bucketing disabled Key: SPARK-47306 URL:

[jira] [Updated] (SPARK-47305) PruneFilters incorrectly tags isStreaming flag when replacing child of Filter with LocalRelation

[ https://issues.apache.org/jira/browse/SPARK-47305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47305: --- Labels: pull-request-available (was: ) > PruneFilters incorrectly tags isStreaming flag

[jira] [Created] (SPARK-47305) PruneFilters incorrectly tags isStreaming flag when replacing child of Filter with LocalRelation

Jungtaek Lim created SPARK-47305: Summary: PruneFilters incorrectly tags isStreaming flag when replacing child of Filter with LocalRelation Key: SPARK-47305 URL: https://issues.apache.org/jira/browse/SPARK-47305

[jira] [Updated] (SPARK-47304) Split DataFrameSuite

[ https://issues.apache.org/jira/browse/SPARK-47304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47304: --- Labels: pull-request-available (was: ) > Split DataFrameSuite > > >

[jira] [Updated] (SPARK-47302) Collation name should be identifier

[ https://issues.apache.org/jira/browse/SPARK-47302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47302: --- Labels: pull-request-available (was: ) > Collation name should be identifier >

[jira] [Created] (SPARK-47304) Split DataFrameSuite

Max Gekk created SPARK-47304: Summary: Split DataFrameSuite Key: SPARK-47304 URL: https://issues.apache.org/jira/browse/SPARK-47304 Project: Spark Issue Type: Test Components: SQL

[jira] [Updated] (SPARK-47303) Restructuring MasterSuite

[ https://issues.apache.org/jira/browse/SPARK-47303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47303: --- Labels: pull-request-available (was: ) > Restructuring MasterSuite >

[jira] [Created] (SPARK-47303) Restructuring MasterSuite

Hyukjin Kwon created SPARK-47303: Summary: Restructuring MasterSuite Key: SPARK-47303 URL: https://issues.apache.org/jira/browse/SPARK-47303 Project: Spark Issue Type: Test

[jira] [Created] (SPARK-47302) Collation name should be identifier

Aleksandar Tomic created SPARK-47302: Summary: Collation name should be identifier Key: SPARK-47302 URL: https://issues.apache.org/jira/browse/SPARK-47302 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-47301) Fix flaky ParquetIOSuite

[ https://issues.apache.org/jira/browse/SPARK-47301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47301: -- Assignee: (was: Apache Spark) > Fix flaky ParquetIOSuite >

[jira] [Assigned] (SPARK-47301) Fix flaky ParquetIOSuite

[ https://issues.apache.org/jira/browse/SPARK-47301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47301: -- Assignee: Apache Spark > Fix flaky ParquetIOSuite > > >

[jira] [Assigned] (SPARK-47301) Fix flaky ParquetIOSuite

[ https://issues.apache.org/jira/browse/SPARK-47301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47301: -- Assignee: (was: Apache Spark) > Fix flaky ParquetIOSuite >

[jira] [Assigned] (SPARK-47301) Fix flaky ParquetIOSuite

[ https://issues.apache.org/jira/browse/SPARK-47301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47301: -- Assignee: Apache Spark > Fix flaky ParquetIOSuite > > >

[jira] [Updated] (SPARK-47301) Fix flaky ParquetIOSuite

[ https://issues.apache.org/jira/browse/SPARK-47301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47301: --- Labels: pull-request-available (was: ) > Fix flaky ParquetIOSuite >

[jira] [Updated] (SPARK-47301) Fix flaky ParquetIOSuite

[ https://issues.apache.org/jira/browse/SPARK-47301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-47301: Component/s: SQL (was: Spark Core) > Fix flaky ParquetIOSuite >

[jira] [Created] (SPARK-47301) Fix flaky ParquetIOSuite

BingKun Pan created SPARK-47301: --- Summary: Fix flaky ParquetIOSuite Key: SPARK-47301 URL: https://issues.apache.org/jira/browse/SPARK-47301 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-47295) startswith, endswith (non-binary collations)

[ https://issues.apache.org/jira/browse/SPARK-47295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uroš Bojanić updated SPARK-47295: - Description: Implement *startsWith* and *endsWith* built-in string Spark functions using

[jira] [Updated] (SPARK-47295) startswith, endswith (non-binary collations)

[ https://issues.apache.org/jira/browse/SPARK-47295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uroš Bojanić updated SPARK-47295: - Description: Implement *startsWith* and *endsWith* built-in string Spark functions using

[jira] [Updated] (SPARK-47295) startswith, endswith (non-binary collations)

[ https://issues.apache.org/jira/browse/SPARK-47295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uroš Bojanić updated SPARK-47295: - Description: Implement *startsWith* and *endsWith* built-in string Spark functions using

[jira] [Updated] (SPARK-47295) startswith, endswith (non-binary collations)

[ https://issues.apache.org/jira/browse/SPARK-47295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uroš Bojanić updated SPARK-47295: - Description: Implement *startsWith* and *endsWith* built-in string Spark functions using

[jira] [Updated] (SPARK-47300) quoteIfNeeded should quote identifier starts with digits

[ https://issues.apache.org/jira/browse/SPARK-47300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47300: --- Labels: pull-request-available (was: ) > quoteIfNeeded should quote identifier starts with

[jira] [Created] (SPARK-47300) quoteIfNeeded should quote identifier starts with digits

Wenchen Fan created SPARK-47300: --- Summary: quoteIfNeeded should quote identifier starts with digits Key: SPARK-47300 URL: https://issues.apache.org/jira/browse/SPARK-47300 Project: Spark Issue