[jira] [Resolved] (SPARK-47289) Allow extensions to log extended information in explain plan

2024-04-04 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-47289. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45488

[jira] [Assigned] (SPARK-47289) Allow extensions to log extended information in explain plan

2024-04-04 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-47289: --- Assignee: Parth Chandra > Allow extensions to log extended information in explain plan >

[jira] [Updated] (SPARK-47718) .sql() does not recognize watermark defined upstream

2024-04-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47718: --- Labels: pull-request-available (was: ) > .sql() does not recognize watermark defined

[jira] [Updated] (SPARK-47735) Make pyspark.testing.connectutils compatible with pyspark-connect

2024-04-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47735: --- Labels: pull-request-available (was: ) > Make pyspark.testing.connectutils compatible with

[jira] [Created] (SPARK-47735) Make pyspark.testing.connectutils compatible with pyspark-connect

2024-04-04 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-47735: Summary: Make pyspark.testing.connectutils compatible with pyspark-connect Key: SPARK-47735 URL: https://issues.apache.org/jira/browse/SPARK-47735 Project: Spark

[jira] [Resolved] (SPARK-47734) Fix flaky pyspark.sql.dataframe.DataFrame.writeStream doctest by stopping streaming query

2024-04-04 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-47734. -- Fix Version/s: 4.0.0 3.5.2 Resolution: Fixed Fixed in

[jira] [Updated] (SPARK-47733) Add operational metrics for TWS operators

2024-04-04 Thread Jing Zhan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhan updated SPARK-47733: -- Description: Add metrics to improve observability for the newly added operator TransformWithState

[jira] [Updated] (SPARK-47734) Fix flaky pyspark.sql.dataframe.DataFrame.writeStream doctest by stopping streaming query

2024-04-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47734: --- Labels: pull-request-available (was: ) > Fix flaky

[jira] [Created] (SPARK-47734) Fix flaky pyspark.sql.dataframe.DataFrame.writeStream doctest by stopping streaming query

2024-04-04 Thread Josh Rosen (Jira)
Josh Rosen created SPARK-47734: -- Summary: Fix flaky pyspark.sql.dataframe.DataFrame.writeStream doctest by stopping streaming query Key: SPARK-47734 URL: https://issues.apache.org/jira/browse/SPARK-47734

[jira] [Updated] (SPARK-47592) Connector module: Migrate logError with variables to structured logging framework

2024-04-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47592: --- Labels: pull-request-available (was: ) > Connector module: Migrate logError with variables

[jira] [Resolved] (SPARK-47598) MLLib: Migrate logError with variables to structured logging framework

2024-04-04 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-47598. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45837

[jira] [Updated] (SPARK-26875) Add an option on FileStreamSource to include modified files

2024-04-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-26875: --- Labels: pull-request-available (was: ) > Add an option on FileStreamSource to include

[jira] [Updated] (SPARK-47731) Fix the 2b+ rows in a single rowgroup for row_index in Parquet reader

2024-04-04 Thread Thang Long Vu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thang Long Vu updated SPARK-47731: -- Description: Parquet reader in Spark has a bug where a file containing 2b+ rows in a single

[jira] [Created] (SPARK-47731) Fix the 2b+ rows in a single rowgroup for row_index in Parquet reader

2024-04-04 Thread Thang Long Vu (Jira)
Thang Long Vu created SPARK-47731: - Summary: Fix the 2b+ rows in a single rowgroup for row_index in Parquet reader Key: SPARK-47731 URL: https://issues.apache.org/jira/browse/SPARK-47731 Project:

[jira] [Created] (SPARK-47730) Support APP_ID and EXECUTOR_ID placeholder in labels

2024-04-04 Thread Xi Chen (Jira)
Xi Chen created SPARK-47730: --- Summary: Support APP_ID and EXECUTOR_ID placeholder in labels Key: SPARK-47730 URL: https://issues.apache.org/jira/browse/SPARK-47730 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-47728) Document G1 Concurrent GC metrics

2024-04-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47728. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45874

[jira] [Assigned] (SPARK-47728) Document G1 Concurrent GC metrics

2024-04-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47728: - Assignee: Luca Canali > Document G1 Concurrent GC metrics >

[jira] [Resolved] (SPARK-47729) Get the proper default port for pyspark-connect testcases

2024-04-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47729. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45875

[jira] [Resolved] (SPARK-47565) PySpark workers dying in daemon mode idle queue fail query

2024-04-04 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-47565. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45635

[jira] [Assigned] (SPARK-47565) PySpark workers dying in daemon mode idle queue fail query

2024-04-04 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-47565: Assignee: Nikita Awasthi > PySpark workers dying in daemon mode idle queue fail query >

[jira] [Assigned] (SPARK-47694) Make max message size configurable on client side

2024-04-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47694: -- Assignee: Martin Grund (was: Apache Spark) > Make max message size configurable on

[jira] [Assigned] (SPARK-47694) Make max message size configurable on client side

2024-04-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47694: -- Assignee: Apache Spark (was: Martin Grund) > Make max message size configurable on

[jira] [Assigned] (SPARK-47359) StringTranslate (all collations)

2024-04-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47359: -- Assignee: (was: Apache Spark) > StringTranslate (all collations) >

[jira] [Assigned] (SPARK-47359) StringTranslate (all collations)

2024-04-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47359: -- Assignee: Apache Spark > StringTranslate (all collations) >

[jira] [Assigned] (SPARK-47359) StringTranslate (all collations)

2024-04-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47359: -- Assignee: (was: Apache Spark) > StringTranslate (all collations) >

[jira] [Assigned] (SPARK-47567) StringLocate

2024-04-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47567: -- Assignee: Apache Spark > StringLocate > > > Key:

[jira] [Assigned] (SPARK-47567) StringLocate

2024-04-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47567: -- Assignee: Apache Spark > StringLocate > > > Key:

[jira] [Assigned] (SPARK-47567) StringLocate

2024-04-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47567: -- Assignee: (was: Apache Spark) > StringLocate > > >

[jira] [Assigned] (SPARK-47359) StringTranslate (all collations)

2024-04-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47359: -- Assignee: Apache Spark > StringTranslate (all collations) >

[jira] [Assigned] (SPARK-47567) StringLocate

2024-04-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47567: -- Assignee: (was: Apache Spark) > StringLocate > > >

[jira] [Updated] (SPARK-47593) Connector module: Migrate logWarn with variables to structured logging framework

2024-04-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47593: --- Labels: pull-request-available (was: ) > Connector module: Migrate logWarn with variables

[jira] [Updated] (SPARK-47586) Hive module: Migrate logError with variables to structured logging framework

2024-04-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47586: --- Labels: pull-request-available (was: ) > Hive module: Migrate logError with variables to

[jira] [Updated] (SPARK-47727) Make SparkConf to root level to for both SparkSession and SparkContext

2024-04-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47727: --- Labels: pull-request-available (was: ) > Make SparkConf to root level to for both

[jira] [Updated] (SPARK-47728) Document G1 Concurrent GC metrics

2024-04-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47728: --- Labels: pull-request-available (was: ) > Document G1 Concurrent GC metrics >

[jira] [Created] (SPARK-47728) Document G1 Concurrent GC metrics

2024-04-04 Thread Luca Canali (Jira)
Luca Canali created SPARK-47728: --- Summary: Document G1 Concurrent GC metrics Key: SPARK-47728 URL: https://issues.apache.org/jira/browse/SPARK-47728 Project: Spark Issue Type: Documentation

[jira] [Created] (SPARK-47727) Make SparkConf to root level to for both SparkSession and SparkContext

2024-04-04 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-47727: Summary: Make SparkConf to root level to for both SparkSession and SparkContext Key: SPARK-47727 URL: https://issues.apache.org/jira/browse/SPARK-47727 Project:

[jira] [Updated] (SPARK-47726) Document push-based shuffle metrics

2024-04-04 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47726: --- Labels: pull-request-available (was: ) > Document push-based shuffle metrics >

[jira] [Created] (SPARK-47726) Document push-based shuffle metrics

2024-04-04 Thread Luca Canali (Jira)
Luca Canali created SPARK-47726: --- Summary: Document push-based shuffle metrics Key: SPARK-47726 URL: https://issues.apache.org/jira/browse/SPARK-47726 Project: Spark Issue Type: Documentation