[jira] [Resolved] (SPARK-51742) Upgrade snowflake-jdbc to 3.23.2

2025-04-08 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-51742. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50535 [https://github.com

[jira] [Created] (SPARK-51744) Columns with spaces cannot be aliased or used in expressions when quotedRegexColumnNames=true

2025-04-08 Thread Jarno Rajala (Jira)
Jarno Rajala created SPARK-51744: Summary: Columns with spaces cannot be aliased or used in expressions when quotedRegexColumnNames=true Key: SPARK-51744 URL: https://issues.apache.org/jira/browse/SPARK-51744

[jira] [Assigned] (SPARK-51742) Upgrade snowflake-jdbc to 3.23.2

2025-04-08 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-51742: Assignee: Yang Jie > Upgrade snowflake-jdbc to 3.23.2 > - > >

[jira] [Updated] (SPARK-51744) Columns with spaces cannot be aliased or used in expressions when quotedRegexColumnNames=true

2025-04-08 Thread Jarno Rajala (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarno Rajala updated SPARK-51744: - Description: When {{{}spark.sql.parser.quotedRegexColumnNames=true{}}}, the following query fai

[jira] [Updated] (SPARK-51728) Add support for SELECT * EXCEPT

2025-04-08 Thread German Schiavon Matteo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] German Schiavon Matteo updated SPARK-51728: --- Description: This change introduces support for the SELECT * EXCEPT(col1, co

[jira] [Updated] (SPARK-51711) Propagates the active remote spark session to new threads to fix parallelFitTasks

2025-04-08 Thread Xi Lyu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xi Lyu updated SPARK-51711: --- Parent: SPARK-51236 Issue Type: Sub-task (was: Bug) > Propagates the active remote spark session to

[jira] [Resolved] (SPARK-51721) Change default value of ANALYZER_SINGLE_PASS_RESOLVER_RELATION_BRIDGING_ENABLED flag

2025-04-08 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-51721. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50516 [https://github.com

[jira] [Assigned] (SPARK-51721) Change default value of ANALYZER_SINGLE_PASS_RESOLVER_RELATION_BRIDGING_ENABLED flag

2025-04-08 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-51721: Assignee: Mihailo Aleksic > Change default value of > ANALYZER_SINGLE_PASS_RESOLVER_RELATION_BRI

[jira] [Resolved] (SPARK-46171) Remove `HiveContext` from Apache Spark 4.0

2025-04-08 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-46171. -- Resolution: Won't Fix > Remove `HiveContext` from Apache Spark 4.0 > -

[jira] [Assigned] (SPARK-51611) Implement new single-pass Analyzer functionality

2025-04-08 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-51611: --- Assignee: Vladimir Golubev > Implement new single-pass Analyzer functionality > ---

[jira] [Resolved] (SPARK-51724) RocksDB StateStore's lineage manager should be synchronized

2025-04-08 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51724. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 50520 [https://gi

[jira] [Assigned] (SPARK-51724) RocksDB StateStore's lineage manager should be synchronized

2025-04-08 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51724: Assignee: Siying Dong > RocksDB StateStore's lineage manager should be synchronized > ---

[jira] [Created] (SPARK-51745) Enforce State Machine for RocksDBStateStore

2025-04-08 Thread Eric Marnadi (Jira)
Eric Marnadi created SPARK-51745: Summary: Enforce State Machine for RocksDBStateStore Key: SPARK-51745 URL: https://issues.apache.org/jira/browse/SPARK-51745 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-51658) SPIP: Add geospatial types in Spark

2025-04-08 Thread Menelaos Karavelas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Menelaos Karavelas updated SPARK-51658: --- Description: *Q1. What are you trying to do? Articulate your objectives using absolu

[jira] [Resolved] (SPARK-51711) Propagates the active remote spark session to new threads to fix parallelFitTasks

2025-04-08 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-51711. --- Fix Version/s: (was: 4.0.0) Resolution: Fixed Issue resolved by pull request 5053

[jira] [Updated] (SPARK-51744) Columns with spaces cannot be aliased or used in expressions when quotedRegexColumnNames=true

2025-04-08 Thread Jarno Rajala (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarno Rajala updated SPARK-51744: - External issue URL: https://github.com/apache/spark/pull/33950 > Columns with spaces cannot be a

[jira] [Updated] (SPARK-51744) Columns with spaces cannot be aliased or used in expressions when quotedRegexColumnNames=true

2025-04-08 Thread Jarno Rajala (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarno Rajala updated SPARK-51744: - External issue URL: (was: https://github.com/apache/spark/pull/33950) > Columns with spaces ca

[jira] [Updated] (SPARK-51744) Columns with spaces cannot be aliased or used in expressions when quotedRegexColumnNames=true

2025-04-08 Thread Jarno Rajala (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarno Rajala updated SPARK-51744: - Description: When {{{}spark.sql.parser.quotedRegexColumnNames=true{}}}, the following query fai

[jira] [Resolved] (SPARK-43092) Cleanup unsuppoerted function `dropDuplicatesWithinWatermark` from `Dataset`

2025-04-08 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-43092. -- Resolution: Won't Fix > Cleanup unsuppoerted function `dropDuplicatesWithinWatermark` from `Dataset` >

[jira] [Created] (SPARK-51746) Data dropped when aggregate on CSV corrupt_record column

2025-04-08 Thread Ted Chester Jenks (Jira)
Ted Chester Jenks created SPARK-51746: - Summary: Data dropped when aggregate on CSV corrupt_record column Key: SPARK-51746 URL: https://issues.apache.org/jira/browse/SPARK-51746 Project: Spark

[jira] [Resolved] (SPARK-51743) Add `describe_(database|table)`, `show_(database|table)`, `explain` sql test and answer files

2025-04-08 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-51743. --- Fix Version/s: connect-swift-0.1.0 Resolution: Fixed Issue resolved by pull request 4

[jira] [Updated] (SPARK-51747) Data source cached plan should account for options

2025-04-08 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-51747: --- Description: Before this, DataSourceStrategy caches the first query plan and does not respect optio

[jira] [Updated] (SPARK-51747) Data source cached plan should account for options

2025-04-08 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-51747: --- Description: Before this, DataSourceStrategy does not take into account options.   Before: ``` sp

[jira] [Updated] (SPARK-51747) Data source cached plan should account for options

2025-04-08 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-51747: --- Description: Before this, DataSourceStrategy caches the first query plan and does not respect optio

[jira] [Updated] (SPARK-51747) Data source cached plan should respect options

2025-04-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-51747: --- Labels: pull-request-available (was: ) > Data source cached plan should respect options > -

[jira] [Updated] (SPARK-51747) Data source cached plan should respect options

2025-04-08 Thread Amanda Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amanda Liu updated SPARK-51747: --- Summary: Data source cached plan should respect options (was: Data source cached plan should accoun

[jira] [Updated] (SPARK-51549) Bump Parquet 1.15.1

2025-04-08 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-51549: Fix Version/s: 3.5.6 > Bump Parquet 1.15.1 > --- > > Key: SPARK-51

[jira] [Resolved] (SPARK-51732) Apply `rpad` on attributes with same `ExprId` if they need to be deduplicated

2025-04-08 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-51732. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 50527 [https://gith

[jira] [Assigned] (SPARK-51732) Apply `rpad` on attributes with same `ExprId` if they need to be deduplicated

2025-04-08 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-51732: --- Assignee: Mihailo Timotic > Apply `rpad` on attributes with same `ExprId` if they need to b

[jira] [Resolved] (SPARK-51714) Add Failure Ingestion test to test state store checkpoint format V2

2025-04-08 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51714. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50508 [https://gi

[jira] [Resolved] (SPARK-51512) Filter out null MapStatus when cleaning up shuffle data with ExternalShuffleService

2025-04-08 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-51512. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 50277 [https://github.com

[jira] [Updated] (SPARK-49862) Add BASIC_PIP_PKGS and CONNECT_PIP_PKGS to Python 3.13 image when it supports Python 3.13

2025-04-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49862: --- Labels: pull-request-available (was: ) > Add BASIC_PIP_PKGS and CONNECT_PIP_PKGS to Python

[jira] [Created] (SPARK-51749) Add `MacOS` integration test with Apache Spark 4.0.0 RC3

2025-04-08 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-51749: - Summary: Add `MacOS` integration test with Apache Spark 4.0.0 RC3 Key: SPARK-51749 URL: https://issues.apache.org/jira/browse/SPARK-51749 Project: Spark Is

[jira] [Updated] (SPARK-51732) Apply `rpad` on attributes with same `ExprId` if they need to be deduplicated

2025-04-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-51732: --- Labels: pull-request-available (was: ) > Apply `rpad` on attributes with same `ExprId` if t

[jira] [Resolved] (SPARK-51611) Implement new single-pass Analyzer functionality

2025-04-08 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-51611. - Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50406 [https://gith

[jira] [Commented] (SPARK-51553) Modify EXTRACT to support TIME data type

2025-04-08 Thread Vinod KC (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17941908#comment-17941908 ] Vinod KC commented on SPARK-51553: -- Im working on this > Modify EXTRACT to support TIM

[jira] [Commented] (SPARK-51564) TIME literals in the 12hr clock format

2025-04-08 Thread Sakthi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17941988#comment-17941988 ] Sakthi commented on SPARK-51564: Interested in working on this task. > TIME literals in

[jira] [Resolved] (SPARK-51695) Introduce Parser Changes for Table Constraints (CHECK, UNIQUE, PK, FK)

2025-04-08 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-51695. Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50496 [https:

[jira] [Updated] (SPARK-51748) UnsafeKVExternalSorter uses size() instead of length()

2025-04-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-51748: --- Labels: pull-request-available (was: ) > UnsafeKVExternalSorter uses size() instead of leng

[jira] [Updated] (SPARK-51738) IN subquery with struct of mismatching field names fails with PLAN_VALIDATION_FAILED_RULE_IN_BATCH

2025-04-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-51738: --- Labels: pull-request-available (was: ) > IN subquery with struct of mismatching field names

[jira] [Updated] (SPARK-51633) The sequential testing of ExecutorSuite and ExternalShuffleServiceSuite will cause ExternalShuffleServiceSuite test fail on macOs

2025-04-08 Thread Igor Dvorzhak (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Dvorzhak updated SPARK-51633: -- Affects Version/s: 4.0.0 > The sequential testing of ExecutorSuite and ExternalShuffleServiceS

[jira] [Updated] (SPARK-51749) Add `MacOS` integration test with Apache Spark 4.0.0 RC3

2025-04-08 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-51749: --- Labels: pull-request-available (was: ) > Add `MacOS` integration test with Apache Spark 4.0

[jira] [Resolved] (SPARK-51296) Support collecting corrupt data in singleVariantColumn mode

2025-04-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-51296. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50517 [https://gi

[jira] [Assigned] (SPARK-51296) Support collecting corrupt data in singleVariantColumn mode

2025-04-08 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-51296: Assignee: Chenhao Li > Support collecting corrupt data in singleVariantColumn mode >

[jira] [Resolved] (SPARK-51749) Add `MacOS` integration test with Apache Spark 4.0.0 RC3

2025-04-08 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-51749. --- Fix Version/s: connect-swift-0.1.0 Resolution: Fixed Issue resolved by pull request 4

[jira] [Assigned] (SPARK-51733) make-distribution.sh supports sbt

2025-04-08 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-51733: Assignee: Cheng Pan > make-distribution.sh supports sbt > - > >

[jira] [Resolved] (SPARK-51733) make-distribution.sh supports sbt

2025-04-08 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-51733. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50529 [https://github.com

[jira] [Created] (SPARK-51748) UnsafeKVExternalSorter uses size() instead of length()

2025-04-08 Thread Harsh Motwani (Jira)
Harsh Motwani created SPARK-51748: - Summary: UnsafeKVExternalSorter uses size() instead of length() Key: SPARK-51748 URL: https://issues.apache.org/jira/browse/SPARK-51748 Project: Spark Issu

[jira] [Resolved] (SPARK-49862) Add BASIC_PIP_PKGS and CONNECT_PIP_PKGS to Python 3.13 image when it supports Python 3.13

2025-04-08 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-49862. --- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50539 [https://

[jira] [Created] (SPARK-51740) get_json_object path parsing ignores leading spaces

2025-04-08 Thread Harsh Motwani (Jira)
Harsh Motwani created SPARK-51740: - Summary: get_json_object path parsing ignores leading spaces Key: SPARK-51740 URL: https://issues.apache.org/jira/browse/SPARK-51740 Project: Spark Issue T

[jira] [Resolved] (SPARK-51717) Possible SST mismatch error for the second snapshot created for a new query

2025-04-08 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-51717. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 50512 [https://gi

[jira] [Assigned] (SPARK-51717) Possible SST mismatch error for the second snapshot created for a new query

2025-04-08 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-51717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-51717: Assignee: B. Micheal Okutubo > Possible SST mismatch error for the second snapshot create