[jira] [Updated] (SPARK-45878) ConcurrentModificationException in CliSuite

2023-11-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45878: --- Labels: pull-request-available (was: ) > ConcurrentModificationException in CliSuite >

[jira] [Created] (SPARK-45878) ConcurrentModificationException in CliSuite

2023-11-09 Thread Kent Yao (Jira)
Kent Yao created SPARK-45878: Summary: ConcurrentModificationException in CliSuite Key: SPARK-45878 URL: https://issues.apache.org/jira/browse/SPARK-45878 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-45877) ExecutorFailureTracker support for standalone mode

2023-11-09 Thread Kent Yao (Jira)
Kent Yao created SPARK-45877: Summary: ExecutorFailureTracker support for standalone mode Key: SPARK-45877 URL: https://issues.apache.org/jira/browse/SPARK-45877 Project: Spark Issue Type:

[jira] [Created] (SPARK-45876) Filters are not pushed down across lateral view

2023-11-09 Thread Alexander Petrossian (PAF) (Jira)
Alexander Petrossian (PAF) created SPARK-45876: -- Summary: Filters are not pushed down across lateral view Key: SPARK-45876 URL: https://issues.apache.org/jira/browse/SPARK-45876 Project:

[jira] [Updated] (SPARK-45875) Remove `MissingStageTableRowData` from `core` module

2023-11-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45875: --- Labels: pull-request-available (was: ) > Remove `MissingStageTableRowData` from `core`

[jira] [Created] (SPARK-45875) Remove `MissingStageTableRowData` from `core` module

2023-11-09 Thread Yang Jie (Jira)
Yang Jie created SPARK-45875: Summary: Remove `MissingStageTableRowData` from `core` module Key: SPARK-45875 URL: https://issues.apache.org/jira/browse/SPARK-45875 Project: Spark Issue Type:

[jira] [Updated] (SPARK-45874) Remove Java version check from `IsolatedClientLoader`

2023-11-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45874: --- Labels: pull-request-available (was: ) > Remove Java version check from

[jira] [Created] (SPARK-45874) Remove Java version check from `IsolatedClientLoader`

2023-11-09 Thread Yang Jie (Jira)
Yang Jie created SPARK-45874: Summary: Remove Java version check from `IsolatedClientLoader` Key: SPARK-45874 URL: https://issues.apache.org/jira/browse/SPARK-45874 Project: Spark Issue Type:

[jira] [Updated] (SPARK-45847) CliSuite flakiness due to non-sequential guarantee for stdout

2023-11-09 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-45847: - Fix Version/s: 3.4.2 > CliSuite flakiness due to non-sequential guarantee for stdout >

[jira] [Updated] (SPARK-45873) Make ExecutorFailureTracker more tolerant when app remains sufficient resources

2023-11-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45873: --- Labels: pull-request-available (was: ) > Make ExecutorFailureTracker more tolerant when

[jira] [Created] (SPARK-45873) Make ExecutorFailureTracker more tolerant when app remains sufficient resources

2023-11-09 Thread Kent Yao (Jira)
Kent Yao created SPARK-45873: Summary: Make ExecutorFailureTracker more tolerant when app remains sufficient resources Key: SPARK-45873 URL: https://issues.apache.org/jira/browse/SPARK-45873 Project:

[jira] [Created] (SPARK-45872) Update plugin for SBOM generation to 2.7.10

2023-11-09 Thread Vinod Anandan (Jira)
Vinod Anandan created SPARK-45872: - Summary: Update plugin for SBOM generation to 2.7.10 Key: SPARK-45872 URL: https://issues.apache.org/jira/browse/SPARK-45872 Project: Spark Issue Type:

[jira] [Updated] (SPARK-45871) Change `.toBuffer.toSeq` to `.toSeq`

2023-11-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45871: --- Labels: pull-request-available (was: ) > Change `.toBuffer.toSeq` to `.toSeq` >

[jira] [Created] (SPARK-45871) Change `.toBuffer.toSeq` to `.toSeq`

2023-11-09 Thread Yang Jie (Jira)
Yang Jie created SPARK-45871: Summary: Change `.toBuffer.toSeq` to `.toSeq` Key: SPARK-45871 URL: https://issues.apache.org/jira/browse/SPARK-45871 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-45814) ArrowConverters.createEmptyArrowBatch may cause memory leak

2023-11-09 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-45814: - Fix Version/s: 4.0.0 3.5.1 > ArrowConverters.createEmptyArrowBatch may cause memory

[jira] [Resolved] (SPARK-45814) ArrowConverters.createEmptyArrowBatch may cause memory leak

2023-11-09 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-45814. -- Fix Version/s: 3.4.2 Resolution: Fixed Issue resolved by pull request 43728

[jira] [Updated] (SPARK-43305) Add Java17 dockerfiles for 3.5.0

2023-11-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-43305: --- Labels: pull-request-available (was: ) > Add Java17 dockerfiles for 3.5.0 >

[jira] [Commented] (SPARK-43305) Add Java17 dockerfiles for 3.5.0

2023-11-09 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784669#comment-17784669 ] Yikun Jiang commented on SPARK-43305: - Resolved by https://github.com/apache/spark-docker/pull/56 >

[jira] [Updated] (SPARK-43305) Add Java17 dockerfiles for 3.5.0

2023-11-09 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-43305: Summary: Add Java17 dockerfiles for 3.5.0 (was: Add Java17 dockerfiles for 3.4.0) > Add Java17

[jira] [Updated] (SPARK-45373) Minimizing calls to HiveMetaStore layer for getting partitions, when tables are repeated

2023-11-09 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45373: - Shepherd: (was: Peter Toth) > Minimizing calls to HiveMetaStore layer for getting partitions, when tables >

[jira] [Updated] (SPARK-33152) SPIP: Constraint Propagation code causes OOM issues or increasing compilation time to hours

2023-11-09 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-33152: - Affects Version/s: 3.5.0 (was: 2.4.0) (was: 3.0.1)

[jira] [Updated] (SPARK-45373) Minimizing calls to HiveMetaStore layer for getting partitions, when tables are repeated

2023-11-09 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45373: - Affects Version/s: 3.5.0 (was: 4.0.0) > Minimizing calls to HiveMetaStore layer for

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-11-09 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Affects Version/s: 3.5.0 (was: 3.5.1) > SPIP: Improving performance of

[jira] [Resolved] (SPARK-45850) Upgrade oracle jdbc driver to 23.3.0.23.09

2023-11-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45850. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43662

[jira] [Updated] (SPARK-45869) Revisit and Improve Spark Standalone Cluster

2023-11-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45869: -- Labels: releasenotes (was: ) > Revisit and Improve Spark Standalone Cluster >

[jira] [Updated] (SPARK-45756) Support `spark.master.useAppNameAsAppId.enabled`

2023-11-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45756: -- Parent: SPARK-45869 Issue Type: Sub-task (was: Improvement) > Support

[jira] [Updated] (SPARK-45756) Support `spark.master.useAppNameAsAppId.enabled`

2023-11-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45756: -- Labels: pull-request-available (was: pull-request-available releasenotes) > Support

[jira] [Updated] (SPARK-45869) Revisit and Improve Spark Standalone Cluster

2023-11-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45869: -- Summary: Revisit and Improve Spark Standalone Cluster (was: Support

[jira] [Updated] (SPARK-45756) Support `spark.master.useAppNameAsAppId.enabled`

2023-11-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45756: -- Summary: Support `spark.master.useAppNameAsAppId.enabled` (was: Revisit and Improve Spark

[jira] [Updated] (SPARK-45869) Support `spark.master.useAppNameAsAppId.enabled`

2023-11-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45869: -- Labels: (was: pull-request-available) > Support `spark.master.useAppNameAsAppId.enabled` >

[jira] (SPARK-45869) Support `spark.master.useAppNameAsAppId.enabled`

2023-11-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45869 ] Dongjoon Hyun deleted comment on SPARK-45869: --- was (Author: dongjoon): This is resolved via [https://github.com/apache/spark/pull/43743] > Support `spark.master.useAppNameAsAppId.enabled`

[jira] [Updated] (SPARK-45869) Support `spark.master.useAppNameAsAppId.enabled`

2023-11-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45869: -- Parent: (was: SPARK-45756) Issue Type: Improvement (was: Sub-task) > Support

[jira] [Updated] (SPARK-45850) Upgrade oracle jdbc driver to 23.3.0.23.09

2023-11-09 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-45850: Component/s: Tests > Upgrade oracle jdbc driver to 23.3.0.23.09 >

[jira] [Updated] (SPARK-45850) Upgrade oracle jdbc driver to 23.3.0.23.09

2023-11-09 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-45850: Summary: Upgrade oracle jdbc driver to 23.3.0.23.09 (was: Attempt to stabilize

[jira] [Resolved] (SPARK-45869) Support `spark.master.useAppNameAsAppId.enabled`

2023-11-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45869. --- Fix Version/s: 4.0.0 Resolution: Fixed This is resolved via

[jira] [Updated] (SPARK-45869) Support `spark.master.useAppNameAsAppId.enabled`

2023-11-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45869: --- Labels: pull-request-available (was: ) > Support `spark.master.useAppNameAsAppId.enabled`

[jira] [Assigned] (SPARK-45869) Support `spark.master.useAppNameAsAppId.enabled`

2023-11-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45869: - Assignee: Dongjoon Hyun > Support `spark.master.useAppNameAsAppId.enabled` >

[jira] [Assigned] (SPARK-45798) Assert server-side session ID in Spark Connect

2023-11-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-45798: Assignee: Martin Grund > Assert server-side session ID in Spark Connect >

[jira] [Resolved] (SPARK-45798) Assert server-side session ID in Spark Connect

2023-11-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-45798. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43664

[jira] [Updated] (SPARK-45756) Revisit and Improve Spark Standalone Cluster

2023-11-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45756: --- Labels: pull-request-available releasenotes (was: releasenotes) > Revisit and Improve

[jira] [Created] (SPARK-45869) Support `spark.master.useAppNameAsAppId.enabled`

2023-11-09 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-45869: - Summary: Support `spark.master.useAppNameAsAppId.enabled` Key: SPARK-45869 URL: https://issues.apache.org/jira/browse/SPARK-45869 Project: Spark Issue

[jira] [Updated] (SPARK-45600) Make Python data source registration session level

2023-11-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45600: --- Labels: pull-request-available (was: ) > Make Python data source registration session

[jira] [Updated] (SPARK-45600) Make Python data source registration session level

2023-11-09 Thread Allison Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated SPARK-45600: - Description: Currently, registered data sources are stored in `sharedState` and can be accessed

[jira] [Updated] (SPARK-45600) Make data source registration session level

2023-11-09 Thread Allison Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated SPARK-45600: - Summary: Make data source registration session level (was: Separate the Python data source

[jira] [Updated] (SPARK-45600) Make Python data source registration session level

2023-11-09 Thread Allison Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated SPARK-45600: - Summary: Make Python data source registration session level (was: Make data source

[jira] [Updated] (SPARK-43905) Consolidate BlockId parsing and creation

2023-11-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-43905: --- Labels: pull-request-available (was: ) > Consolidate BlockId parsing and creation >

[jira] [Updated] (SPARK-44179) When a task failed and the inferred task for that task is still executing, the number of dynamically scheduled executors will be calculated incorrectly

2023-11-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-44179: --- Labels: pull-request-available (was: ) > When a task failed and the inferred task for that

[jira] [Updated] (SPARK-44595) Make the user session cache number and cache time be configurable in spark connect service

2023-11-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-44595: --- Labels: pull-request-available (was: ) > Make the user session cache number and cache time

[jira] [Resolved] (SPARK-45731) Update partition statistics with ANALYZE TABLE command

2023-11-09 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved SPARK-45731. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43629

[jira] [Assigned] (SPARK-45731) Update partition statistics with ANALYZE TABLE command

2023-11-09 Thread Chao Sun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun reassigned SPARK-45731: Assignee: Chao Sun > Update partition statistics with ANALYZE TABLE command >

[jira] [Resolved] (SPARK-45867) Support `spark.worker.idPattern`

2023-11-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45867. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43740

[jira] [Assigned] (SPARK-45867) Support `spark.worker.idPattern`

2023-11-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45867: - Assignee: Dongjoon Hyun > Support `spark.worker.idPattern` >

[jira] [Updated] (SPARK-45868) Make spark.table use the same parser with vanilla spark

2023-11-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45868: --- Labels: pull-request-available (was: ) > Make spark.table use the same parser with vanilla

[jira] [Created] (SPARK-45868) Make spark.table use the same parser with vanilla spark

2023-11-09 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-45868: - Summary: Make spark.table use the same parser with vanilla spark Key: SPARK-45868 URL: https://issues.apache.org/jira/browse/SPARK-45868 Project: Spark

[jira] [Updated] (SPARK-45867) Support `spark.worker.idPattern`

2023-11-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45867: --- Labels: pull-request-available (was: ) > Support `spark.worker.idPattern` >

[jira] [Created] (SPARK-45867) Support `spark.worker.idPattern`

2023-11-09 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-45867: - Summary: Support `spark.worker.idPattern` Key: SPARK-45867 URL: https://issues.apache.org/jira/browse/SPARK-45867 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-45866) Reuse of exchange in AQE does not happen when run time filters are pushed down to the underlying Scan ( like iceberg )

2023-11-09 Thread Asif (Jira)
Asif created SPARK-45866: Summary: Reuse of exchange in AQE does not happen when run time filters are pushed down to the underlying Scan ( like iceberg ) Key: SPARK-45866 URL:

[jira] [Created] (SPARK-45865) Add user guide for window operations

2023-11-09 Thread Allison Wang (Jira)
Allison Wang created SPARK-45865: Summary: Add user guide for window operations Key: SPARK-45865 URL: https://issues.apache.org/jira/browse/SPARK-45865 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-45864) Add user guide for groupby and aggregate

2023-11-09 Thread Allison Wang (Jira)
Allison Wang created SPARK-45864: Summary: Add user guide for groupby and aggregate Key: SPARK-45864 URL: https://issues.apache.org/jira/browse/SPARK-45864 Project: Spark Issue Type:

[jira] [Created] (SPARK-45863) Add user guide for column selections

2023-11-09 Thread Allison Wang (Jira)
Allison Wang created SPARK-45863: Summary: Add user guide for column selections Key: SPARK-45863 URL: https://issues.apache.org/jira/browse/SPARK-45863 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-45862) Add user guide for basic dataframe operations

2023-11-09 Thread Allison Wang (Jira)
Allison Wang created SPARK-45862: Summary: Add user guide for basic dataframe operations Key: SPARK-45862 URL: https://issues.apache.org/jira/browse/SPARK-45862 Project: Spark Issue Type:

[jira] [Created] (SPARK-45861) Add user guide for dataframe creation

2023-11-09 Thread Allison Wang (Jira)
Allison Wang created SPARK-45861: Summary: Add user guide for dataframe creation Key: SPARK-45861 URL: https://issues.apache.org/jira/browse/SPARK-45861 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-45592) AQE and InMemoryTableScanExec correctness bug

2023-11-09 Thread Emil Ejbyfeldt (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Emil Ejbyfeldt updated SPARK-45592: --- Description: The following query should return 100 {code:java} import

[jira] [Created] (SPARK-45860) ClassCastException with SerializedLambda in Spark Cluster Mode

2023-11-09 Thread Abhilash (Jira)
Abhilash created SPARK-45860: Summary: ClassCastException with SerializedLambda in Spark Cluster Mode Key: SPARK-45860 URL: https://issues.apache.org/jira/browse/SPARK-45860 Project: Spark

[jira] [Updated] (SPARK-45859) Make UDF objects in ml.functions lazy

2023-11-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45859: --- Labels: pull-request-available (was: ) > Make UDF objects in ml.functions lazy >

[jira] [Created] (SPARK-45859) Make UDF objects in ml.functions lazy

2023-11-09 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-45859: - Summary: Make UDF objects in ml.functions lazy Key: SPARK-45859 URL: https://issues.apache.org/jira/browse/SPARK-45859 Project: Spark Issue Type:

[jira] [Commented] (SPARK-45658) Canonicalization of DynamicPruningSubquery is broken

2023-11-09 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784567#comment-17784567 ] Asif commented on SPARK-45658: -- I also think that during canonicalization of DynamicPruningSubquery, the

[jira] [Updated] (SPARK-44609) ExecutorPodsAllocator doesn't create new executors if no pod snapshot captured pod creation

2023-11-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-44609: --- Labels: pull-request-available (was: ) > ExecutorPodsAllocator doesn't create new

[jira] [Assigned] (SPARK-45592) AQE and InMemoryTableScanExec correctness bug

2023-11-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45592: - Assignee: Emil Ejbyfeldt (was: Apache Spark) > AQE and InMemoryTableScanExec

[jira] [Updated] (SPARK-45592) AQE and InMemoryTableScanExec correctness bug

2023-11-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45592: -- Target Version/s: 3.4.2 > AQE and InMemoryTableScanExec correctness bug >

[jira] [Updated] (SPARK-45592) AQE and InMemoryTableScanExec correctness bug

2023-11-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45592: -- Affects Version/s: 3.4.1 > AQE and InMemoryTableScanExec correctness bug >

[jira] [Created] (SPARK-45858) Consistent FetchFailed/NoSuchFileExceptions when decommissioning is enabled

2023-11-09 Thread Alibi Yeslambek (Jira)
Alibi Yeslambek created SPARK-45858: --- Summary: Consistent FetchFailed/NoSuchFileExceptions when decommissioning is enabled Key: SPARK-45858 URL: https://issues.apache.org/jira/browse/SPARK-45858

[jira] [Updated] (SPARK-45658) Canonicalization of DynamicPruningSubquery is broken

2023-11-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45658: --- Labels: pull-request-available (was: ) > Canonicalization of DynamicPruningSubquery is

[jira] [Updated] (SPARK-45857) Enforce the error classes in sub-classes of AnalysisException

2023-11-09 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-45857: - Description: Make the error class in sub-classes of AnalysisException mandatory to enforce callers to

[jira] [Created] (SPARK-45857) Enforce the error classes in sub-classes of AnalysisException

2023-11-09 Thread Max Gekk (Jira)
Max Gekk created SPARK-45857: Summary: Enforce the error classes in sub-classes of AnalysisException Key: SPARK-45857 URL: https://issues.apache.org/jira/browse/SPARK-45857 Project: Spark Issue

[jira] [Updated] (SPARK-45857) Enforce the error classes in sub-classes of AnalysisException

2023-11-09 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-45857: - Description: Make the error class in sub-classes of ParseException mandatory to enforce callers to

[jira] [Resolved] (SPARK-45855) Unable to set compression codec for Hive CTAS

2023-11-09 Thread Tim Robertson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Robertson resolved SPARK-45855. --- Resolution: Fixed I found this is fixed in 3.5.0 and I strongly suspect is caused by the

[jira] [Commented] (SPARK-45855) Unable to set compression codec for Hive CTAS

2023-11-09 Thread Tim Robertson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784533#comment-17784533 ] Tim Robertson commented on SPARK-45855: --- I suspect it is this

[jira] [Updated] (SPARK-45855) Unable to set compression codec for Hive CTAS

2023-11-09 Thread Tim Robertson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Robertson updated SPARK-45855: -- Fix Version/s: 3.5.0 > Unable to set compression codec for Hive CTAS >

[jira] [Commented] (SPARK-45855) Unable to set compression codec for Hive CTAS

2023-11-09 Thread Tim Robertson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784530#comment-17784530 ] Tim Robertson commented on SPARK-45855: --- This also seems to fail with 3.4.1 but seems to be fixed

[jira] [Updated] (SPARK-45856) Move ArtifactManager from Spark Connect into SparkSession (sql/core)

2023-11-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45856: --- Labels: pull-request-available (was: ) > Move ArtifactManager from Spark Connect into

[jira] [Updated] (SPARK-45855) Unable to set compression codec for Hive CTAS

2023-11-09 Thread Tim Robertson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Robertson updated SPARK-45855: -- Summary: Unable to set compression codec for Hive CTAS (was: Unable to set codec for Hive

[jira] [Updated] (SPARK-45855) Unable to set codec for Hive CTAS

2023-11-09 Thread Tim Robertson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Robertson updated SPARK-45855: -- Description: Hi, We've discovered code that worked in Spark 3.3.0 doesn't in 3.4.0. I can't

[jira] [Created] (SPARK-45855) Unable to set codec for Hive CTAS

2023-11-09 Thread Tim Robertson (Jira)
Tim Robertson created SPARK-45855: - Summary: Unable to set codec for Hive CTAS Key: SPARK-45855 URL: https://issues.apache.org/jira/browse/SPARK-45855 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-45849) Remove uneccessary toSeq when encoding Set to catalyst

2023-11-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45849: --- Labels: pull-request-available (was: ) > Remove uneccessary toSeq when encoding Set to

[jira] [Created] (SPARK-45854) spark.catalog.listTables fails with ParseException after upgrading to Spark 3.4.1 from 3.3.1

2023-11-09 Thread Andrej Zachar (Jira)
Andrej Zachar created SPARK-45854: - Summary: spark.catalog.listTables fails with ParseException after upgrading to Spark 3.4.1 from 3.3.1 Key: SPARK-45854 URL: https://issues.apache.org/jira/browse/SPARK-45854

[jira] [Created] (SPARK-45853) Add Iceberg and Hudi to third party projects

2023-11-09 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-45853: --- Summary: Add Iceberg and Hudi to third party projects Key: SPARK-45853 URL: https://issues.apache.org/jira/browse/SPARK-45853 Project: Spark Issue Type:

[jira] [Updated] (SPARK-45850) Attempt to stabilize `OracleIntegrationSuite ` by upgrading the oracle jdbc driver version

2023-11-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45850: --- Labels: pull-request-available (was: ) > Attempt to stabilize `OracleIntegrationSuite ` by

[jira] [Updated] (SPARK-45852) Gracefully deal with recursion exception during Spark Connect logging

2023-11-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45852: --- Labels: pull-request-available (was: ) > Gracefully deal with recursion exception during

[jira] [Created] (SPARK-45852) Gracefully deal with recursion exception during Spark Connect logging

2023-11-09 Thread Martin Grund (Jira)
Martin Grund created SPARK-45852: Summary: Gracefully deal with recursion exception during Spark Connect logging Key: SPARK-45852 URL: https://issues.apache.org/jira/browse/SPARK-45852 Project: Spark

[jira] [Assigned] (SPARK-45815) Provide an interface for Streaming sources to add _metadata columns

2023-11-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-45815: --- Assignee: Yaohua Zhao > Provide an interface for Streaming sources to add _metadata

[jira] [Resolved] (SPARK-45815) Provide an interface for Streaming sources to add _metadata columns

2023-11-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-45815. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43692

[jira] [Resolved] (SPARK-44886) Introduce CLUSTER BY SQL clause to CREATE/REPLACE TABLE

2023-11-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44886. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42577

[jira] [Assigned] (SPARK-44886) Introduce CLUSTER BY SQL clause to CREATE/REPLACE TABLE

2023-11-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44886: --- Assignee: Terry Kim > Introduce CLUSTER BY SQL clause to CREATE/REPLACE TABLE >

[jira] [Created] (SPARK-45851) (Scala) Support different retry policies for connect client

2023-11-09 Thread Alice Sayutina (Jira)
Alice Sayutina created SPARK-45851: -- Summary: (Scala) Support different retry policies for connect client Key: SPARK-45851 URL: https://issues.apache.org/jira/browse/SPARK-45851 Project: Spark

[jira] [Updated] (SPARK-45733) (Python) Support different retry policies for connect client

2023-11-09 Thread Alice Sayutina (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alice Sayutina updated SPARK-45733: --- Summary: (Python) Support different retry policies for connect client (was: Classify

[jira] [Commented] (SPARK-45850) Attempt to stabilize `OracleIntegrationSuite ` by upgrading the oracle jdbc driver version

2023-11-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784335#comment-17784335 ] ASF GitHub Bot commented on SPARK-45850: User 'panbingkun' has created a pull request for this

[jira] [Assigned] (SPARK-45850) Attempt to stabilize `OracleIntegrationSuite ` by upgrading the oracle jdbc driver version

2023-11-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45850: -- Assignee: Apache Spark > Attempt to stabilize `OracleIntegrationSuite ` by upgrading

[jira] [Assigned] (SPARK-45850) Attempt to stabilize `OracleIntegrationSuite ` by upgrading the oracle jdbc driver version

2023-11-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-45850: -- Assignee: (was: Apache Spark) > Attempt to stabilize `OracleIntegrationSuite `

[jira] [Commented] (SPARK-45282) Join loses records for cached datasets

2023-11-09 Thread Emil Ejbyfeldt (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784320#comment-17784320 ] Emil Ejbyfeldt commented on SPARK-45282: Created this

  1   2   >