[jira] [Updated] (SPARK-47248) contains (non-binary collations)

[ https://issues.apache.org/jira/browse/SPARK-47248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47248: --- Labels: pull-request-available (was: ) > contains (non-binary collations) >

[jira] [Assigned] (SPARK-47146) Possible thread leak when doing sort merge join

[ https://issues.apache.org/jira/browse/SPARK-47146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-47146: --- Assignee: JacobZheng > Possible thread leak when doing sort merge join >

[jira] [Resolved] (SPARK-47146) Possible thread leak when doing sort merge join

[ https://issues.apache.org/jira/browse/SPARK-47146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-47146. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request

[jira] [Updated] (SPARK-47279) spark driver process hangs due to "unable to create new native thread"

[ https://issues.apache.org/jira/browse/SPARK-47279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] TianyiMa updated SPARK-47279: - Description: we encounter that spark driver hangs for about 11 hours,  and finall killed by user. In

[jira] [Updated] (SPARK-47279) spark driver process hangs due to "unable to create new native thread"

[ https://issues.apache.org/jira/browse/SPARK-47279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] TianyiMa updated SPARK-47279: - Description: we encounter that spark driver hangs for about 11 hours,  and finall killed by user. In

[jira] [Updated] (SPARK-47279) spark driver process hangs due to "unable to create new native thread"

[ https://issues.apache.org/jira/browse/SPARK-47279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] TianyiMa updated SPARK-47279: - Attachment: executor_4.png > spark driver process hangs due to "unable to create new native thread" >

[jira] [Updated] (SPARK-47279) spark driver process hangs due to "unable to create new native thread"

[ https://issues.apache.org/jira/browse/SPARK-47279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] TianyiMa updated SPARK-47279: - Description: we encounter that spark driver hangs for about 11 hours,  and finall killed by user. In

[jira] [Updated] (SPARK-47279) spark driver process hangs due to "unable to create new native thread"

[ https://issues.apache.org/jira/browse/SPARK-47279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] TianyiMa updated SPARK-47279: - Description: we encounter that spark driver hangs for about 11 hours,  and finall killed by user. In

[jira] [Updated] (SPARK-47279) spark driver process hangs due to "unable to create new native thread"

[ https://issues.apache.org/jira/browse/SPARK-47279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] TianyiMa updated SPARK-47279: - Attachment: driver_submit_task.png > spark driver process hangs due to "unable to create new native

[jira] [Updated] (SPARK-47278) Upgrade rocksdbjni to 8.11.3

[ https://issues.apache.org/jira/browse/SPARK-47278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47278: --- Labels: pull-request-available (was: ) > Upgrade rocksdbjni to 8.11.3 >

[jira] [Created] (SPARK-47279) spark driver process hangs due to "unable to create new native thread"

TianyiMa created SPARK-47279: Summary: spark driver process hangs due to "unable to create new native thread" Key: SPARK-47279 URL: https://issues.apache.org/jira/browse/SPARK-47279 Project: Spark

[jira] [Created] (SPARK-47278) Upgrade rocksdbjni to 8.11.3

Yang Jie created SPARK-47278: Summary: Upgrade rocksdbjni to 8.11.3 Key: SPARK-47278 URL: https://issues.apache.org/jira/browse/SPARK-47278 Project: Spark Issue Type: Sub-task

[jira] [Resolved] (SPARK-47177) Cached SQL plan do not display final AQE plan in explain string

[ https://issues.apache.org/jira/browse/SPARK-47177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You resolved SPARK-47177. --- Fix Version/s: 3.5.2 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-47177) Cached SQL plan do not display final AQE plan in explain string

[ https://issues.apache.org/jira/browse/SPARK-47177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You reassigned SPARK-47177: - Assignee: XiDuo You > Cached SQL plan do not display final AQE plan in explain string >

[jira] [Updated] (SPARK-47277) PySpark util function assertDataFrameEqual should not support streaming DF

[ https://issues.apache.org/jira/browse/SPARK-47277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47277: --- Labels: pull-request-available (was: ) > PySpark util function assertDataFrameEqual should

[jira] [Created] (SPARK-47277) PySpark util function assertDataFrameEqual should not support streaming DF

Wei Liu created SPARK-47277: --- Summary: PySpark util function assertDataFrameEqual should not support streaming DF Key: SPARK-47277 URL: https://issues.apache.org/jira/browse/SPARK-47277 Project: Spark

[jira] [Updated] (SPARK-47272) MapState Implementation for State V2

[ https://issues.apache.org/jira/browse/SPARK-47272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47272: --- Labels: pull-request-available (was: ) > MapState Implementation for State V2 >

[jira] [Created] (SPARK-47276) Introduce `spark.profile.clear` for SparkSession-based profiling

Xinrong Meng created SPARK-47276: Summary: Introduce `spark.profile.clear` for SparkSession-based profiling Key: SPARK-47276 URL: https://issues.apache.org/jira/browse/SPARK-47276 Project: Spark

[jira] [Created] (SPARK-47275) XML: Change to not support DROPMALFORMED parse mode

Yousof Hosny created SPARK-47275: Summary: XML: Change to not support DROPMALFORMED parse mode Key: SPARK-47275 URL: https://issues.apache.org/jira/browse/SPARK-47275 Project: Spark Issue

[jira] [Resolved] (SPARK-47252) Clarify that pivot may trigger an eager computation

[ https://issues.apache.org/jira/browse/SPARK-47252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-47252. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45363

[jira] [Assigned] (SPARK-47252) Clarify that pivot may trigger an eager computation

[ https://issues.apache.org/jira/browse/SPARK-47252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-47252: Assignee: Nicholas Chammas > Clarify that pivot may trigger an eager computation >

[jira] [Updated] (SPARK-47274) Provide more useful context for PySpark DataFrame API errors

[ https://issues.apache.org/jira/browse/SPARK-47274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47274: --- Labels: pull-request-available (was: ) > Provide more useful context for PySpark DataFrame

[jira] [Created] (SPARK-47274) Provide more useful context for PySpark DataFrame API errors

Haejoon Lee created SPARK-47274: --- Summary: Provide more useful context for PySpark DataFrame API errors Key: SPARK-47274 URL: https://issues.apache.org/jira/browse/SPARK-47274 Project: Spark

[jira] [Updated] (SPARK-47273) Implement python stream writer interface

[ https://issues.apache.org/jira/browse/SPARK-47273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoqin Li updated SPARK-47273: --- Description: In order to support developing spark streaming sink in python, we need to implement

[jira] [Updated] (SPARK-47273) Implement python stream writer interface

[ https://issues.apache.org/jira/browse/SPARK-47273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoqin Li updated SPARK-47273: --- Description: In order to support developing spark streaming sink in python, we need to implement 

[jira] [Created] (SPARK-47273) Implement python stream writer interface

Chaoqin Li created SPARK-47273: -- Summary: Implement python stream writer interface Key: SPARK-47273 URL: https://issues.apache.org/jira/browse/SPARK-47273 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-36691) PythonRunner failed should pass error message to ApplicationMaster too

[ https://issues.apache.org/jira/browse/SPARK-36691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-36691: --- Labels: pull-request-available (was: ) > PythonRunner failed should pass error message to

[jira] [Commented] (SPARK-46778) get_json_object flattens wildcard queries that match a single value

[ https://issues.apache.org/jira/browse/SPARK-46778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823371#comment-17823371 ] Pablo Langa Blanco commented on SPARK-46778: I was looking at it and I found a comment in

[jira] [Commented] (SPARK-33152) SPIP: Constraint Propagation code causes OOM issues or increasing compilation time to hours

[ https://issues.apache.org/jira/browse/SPARK-33152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823344#comment-17823344 ] Asif commented on SPARK-33152: -- [~tedjenks]  The issue has always been there  because of the way constraint

[jira] [Updated] (SPARK-44746) Improve the documentation for TABLE input arguments for UDTFs

[ https://issues.apache.org/jira/browse/SPARK-44746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-44746: --- Labels: pull-request-available (was: ) > Improve the documentation for TABLE input

[jira] [Resolved] (SPARK-47032) Create API for 'analyze' method to send input column(s) to output table unchanged

[ https://issues.apache.org/jira/browse/SPARK-47032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel resolved SPARK-47032. Resolution: Won't Fix After investigating this, we found that any performance benefits from skipping

[jira] [Updated] (SPARK-47272) MapState Implementation for State V2

[ https://issues.apache.org/jira/browse/SPARK-47272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhan updated SPARK-47272: -- Epic Link: SPARK-46815 Description: This task adds changes for MapState implementation in State

[jira] [Updated] (SPARK-47242) Bump ap-loader 3.0(v8) to support for async-profiler 3.0

[ https://issues.apache.org/jira/browse/SPARK-47242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47242: -- Parent: SPARK-47046 Issue Type: Sub-task (was: Improvement) > Bump ap-loader 3.0(v8)

[jira] [Resolved] (SPARK-47242) Bump ap-loader 3.0(v8) to support for async-profiler 3.0

[ https://issues.apache.org/jira/browse/SPARK-47242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47242. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45351

[jira] [Assigned] (SPARK-47242) Bump ap-loader 3.0(v8) to support for async-profiler 3.0

[ https://issues.apache.org/jira/browse/SPARK-47242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47242: - Assignee: Nicholas Jiang > Bump ap-loader 3.0(v8) to support for async-profiler 3.0 >

[jira] [Commented] (SPARK-33152) SPIP: Constraint Propagation code causes OOM issues or increasing compilation time to hours

[ https://issues.apache.org/jira/browse/SPARK-33152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823314#comment-17823314 ] Ted Chester Jenks commented on SPARK-33152: --- [~cloud_fan] [~ashahid7] We have seen a huge

[jira] [Updated] (SPARK-47269) Upgrade jetty to 11.0.20

[ https://issues.apache.org/jira/browse/SPARK-47269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47269: -- Parent: SPARK-47046 Issue Type: Sub-task (was: Improvement) > Upgrade jetty to

[jira] [Assigned] (SPARK-47269) Upgrade jetty to 11.0.20

[ https://issues.apache.org/jira/browse/SPARK-47269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-47269: - Assignee: Yang Jie > Upgrade jetty to 11.0.20 > > >

[jira] [Resolved] (SPARK-47269) Upgrade jetty to 11.0.20

[ https://issues.apache.org/jira/browse/SPARK-47269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47269. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45371

[jira] [Assigned] (SPARK-46961) Adding processorHandle as a Context Variable

[ https://issues.apache.org/jira/browse/SPARK-46961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-46961: Assignee: Eric Marnadi > Adding processorHandle as a Context Variable >

[jira] [Resolved] (SPARK-46961) Adding processorHandle as a Context Variable

[ https://issues.apache.org/jira/browse/SPARK-46961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-46961. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45359

[jira] [Updated] (SPARK-47271) Explain importance of statistics on SQL performance tuning page

[ https://issues.apache.org/jira/browse/SPARK-47271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47271: --- Labels: pull-request-available (was: ) > Explain importance of statistics on SQL

[jira] [Assigned] (SPARK-47268) Repartition by column should respect collation

[ https://issues.apache.org/jira/browse/SPARK-47268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-47268: --- Assignee: Aleksandar Tomic > Repartition by column should respect collation >

[jira] [Resolved] (SPARK-47268) Repartition by column should respect collation

[ https://issues.apache.org/jira/browse/SPARK-47268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-47268. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45370

[jira] [Created] (SPARK-47271) Explain importance of statistics on SQL performance tuning page

Nicholas Chammas created SPARK-47271: Summary: Explain importance of statistics on SQL performance tuning page Key: SPARK-47271 URL: https://issues.apache.org/jira/browse/SPARK-47271 Project:

[jira] [Resolved] (SPARK-47070) Subquery rewrite inside an aggregation makes an aggregation invalid

[ https://issues.apache.org/jira/browse/SPARK-47070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-47070. - Fix Version/s: 4.0.0 Resolution: Fixed > Subquery rewrite inside an aggregation makes an

[jira] [Resolved] (SPARK-47131) contains, startswith, endswith

[ https://issues.apache.org/jira/browse/SPARK-47131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-47131. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45216

[jira] [Assigned] (SPARK-47131) contains, startswith, endswith

[ https://issues.apache.org/jira/browse/SPARK-47131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-47131: --- Assignee: Uroš Bojanić > contains, startswith, endswith > -- >

[jira] [Updated] (SPARK-47270) Dataset.isEmpty should not trigger job execution on CommandResults

[ https://issues.apache.org/jira/browse/SPARK-47270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47270: --- Labels: pull-request-available (was: ) > Dataset.isEmpty should not trigger job execution

[jira] [Created] (SPARK-47270) Dataset.isEmpty should not trigger job execution on CommandResults

Zhen Wang created SPARK-47270: - Summary: Dataset.isEmpty should not trigger job execution on CommandResults Key: SPARK-47270 URL: https://issues.apache.org/jira/browse/SPARK-47270 Project: Spark

[jira] [Updated] (SPARK-47265) Use `createTable(..., schema: StructType, ...)` instead of `createTable(..., columns: Array[Column], ...)` in UT

[ https://issues.apache.org/jira/browse/SPARK-47265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-47265: Summary: Use `createTable(..., schema: StructType, ...)` instead of `createTable(..., columns:

[jira] [Updated] (SPARK-47269) Upgrade jetty to 11.0.20

[ https://issues.apache.org/jira/browse/SPARK-47269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47269: --- Labels: pull-request-available (was: ) > Upgrade jetty to 11.0.20 >

[jira] [Created] (SPARK-47269) Upgrade jetty to 11.0.20

Yang Jie created SPARK-47269: Summary: Upgrade jetty to 11.0.20 Key: SPARK-47269 URL: https://issues.apache.org/jira/browse/SPARK-47269 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-47268) Repartition by column should respect collation

[ https://issues.apache.org/jira/browse/SPARK-47268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47268: --- Labels: pull-request-available (was: ) > Repartition by column should respect collation >

[jira] [Resolved] (SPARK-43258) Assign a name to the error class _LEGACY_ERROR_TEMP_202[3,4,5]

[ https://issues.apache.org/jira/browse/SPARK-43258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-43258. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45288

[jira] [Assigned] (SPARK-43258) Assign a name to the error class _LEGACY_ERROR_TEMP_202[3,4,5]

[ https://issues.apache.org/jira/browse/SPARK-43258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-43258: Assignee: Deng Ziming > Assign a name to the error class _LEGACY_ERROR_TEMP_202[3,4,5] >

[jira] [Created] (SPARK-47268) Repartition by column should respect collation

Aleksandar Tomic created SPARK-47268: Summary: Repartition by column should respect collation Key: SPARK-47268 URL: https://issues.apache.org/jira/browse/SPARK-47268 Project: Spark Issue

[jira] [Created] (SPARK-47267) Hash functions should respect collation

Aleksandar Tomic created SPARK-47267: Summary: Hash functions should respect collation Key: SPARK-47267 URL: https://issues.apache.org/jira/browse/SPARK-47267 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-47131) contains, startswith, endswith

[ https://issues.apache.org/jira/browse/SPARK-47131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47131: -- Assignee: (was: Apache Spark) > contains, startswith, endswith >

[jira] [Assigned] (SPARK-47131) contains, startswith, endswith

[ https://issues.apache.org/jira/browse/SPARK-47131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47131: -- Assignee: Apache Spark > contains, startswith, endswith >

[jira] [Assigned] (SPARK-47131) contains, startswith, endswith

[ https://issues.apache.org/jira/browse/SPARK-47131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47131: -- Assignee: (was: Apache Spark) > contains, startswith, endswith >

[jira] [Assigned] (SPARK-47131) contains, startswith, endswith

[ https://issues.apache.org/jira/browse/SPARK-47131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47131: -- Assignee: Apache Spark > contains, startswith, endswith >

[jira] [Assigned] (SPARK-47070) Subquery rewrite inside an aggregation makes an aggregation invalid

[ https://issues.apache.org/jira/browse/SPARK-47070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47070: -- Assignee: (was: Apache Spark) > Subquery rewrite inside an aggregation makes an

[jira] [Assigned] (SPARK-47070) Subquery rewrite inside an aggregation makes an aggregation invalid

[ https://issues.apache.org/jira/browse/SPARK-47070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47070: -- Assignee: Apache Spark > Subquery rewrite inside an aggregation makes an aggregation

[jira] [Resolved] (SPARK-47266) Make `ProtoUtils.abbreviate` return the same type as the input

[ https://issues.apache.org/jira/browse/SPARK-47266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-47266. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45369

[jira] [Updated] (SPARK-47253) Allow LiveEventBus to stop without the completely draining of event queue

[ https://issues.apache.org/jira/browse/SPARK-47253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] TakawaAkirayo updated SPARK-47253: -- Affects Version/s: 4.0.0 (was: 3.5.0) > Allow LiveEventBus to stop

[jira] [Created] (SPARK-47266) Make `ProtoUtils.abbreviate` return the same type as the input

Ruifeng Zheng created SPARK-47266: - Summary: Make `ProtoUtils.abbreviate` return the same type as the input Key: SPARK-47266 URL: https://issues.apache.org/jira/browse/SPARK-47266 Project: Spark