[jira] [Updated] (SPARK-44772) Reading blocks from remote executors causes timeout issue

[ https://issues.apache.org/jira/browse/SPARK-44772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nebi mert aydin updated SPARK-44772: Component/s: Shuffle Spark Core > Reading blocks from remote executors c

[jira] [Updated] (SPARK-44772) Reading blocks from remote executors causes timeout issue

[ https://issues.apache.org/jira/browse/SPARK-44772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nebi mert aydin updated SPARK-44772: Description: I'm using EMR 6.5 with Spark 3.1.2 I'm shuffling 1.5 TiB of data with 3000 e

[jira] [Updated] (SPARK-44772) Reading blocks from remote executors causes timeout issue

[ https://issues.apache.org/jira/browse/SPARK-44772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nebi mert aydin updated SPARK-44772: Description: I'm using EMR 6.5 with Spark 3.1.2 I'm shuffling 1.5 TiB of data with 3000 e

[jira] [Created] (SPARK-44772) Reading blocks from remote executors causes timeout issue

nebi mert aydin created SPARK-44772: --- Summary: Reading blocks from remote executors causes timeout issue Key: SPARK-44772 URL: https://issues.apache.org/jira/browse/SPARK-44772 Project: Spark

[jira] [Commented] (SPARK-24578) Reading remote cache block behavior changes and causes timeout issue

[ https://issues.apache.org/jira/browse/SPARK-24578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17753038#comment-17753038 ] nebi mert aydin commented on SPARK-24578: - I still have this problem in Amazon E

[jira] [Commented] (SPARK-44719) NoClassDefFoundError when using Hive UDF

[ https://issues.apache.org/jira/browse/SPARK-44719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17753037#comment-17753037 ] Snoot.io commented on SPARK-44719: -- User 'wangyum' has created a pull request for this

[jira] [Commented] (SPARK-44461) Enable Process Isolation for streaming python worker

[ https://issues.apache.org/jira/browse/SPARK-44461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17753036#comment-17753036 ] Snoot.io commented on SPARK-44461: -- User 'WweiL' has created a pull request for this is

[jira] [Updated] (SPARK-44768) Improve WSCG handling of row buffer by accounting for executor memory . Exploding nested arrays can easily lead to out of memory errors.

[ https://issues.apache.org/jira/browse/SPARK-44768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franck Tago updated SPARK-44768: Description: The code sample below is to showcase the wholestagecodegen generated when exploding

[jira] [Commented] (SPARK-44768) Improve WSCG handling of row buffer by accounting for executor memory . Exploding nested arrays can easily lead to out of memory errors.

[ https://issues.apache.org/jira/browse/SPARK-44768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17753035#comment-17753035 ] Franck Tago commented on SPARK-44768: - !image-2023-08-10-20-32-55-684.png! > Improv

[jira] [Updated] (SPARK-44768) Improve WSCG handling of row buffer by accounting for executor memory . Exploding nested arrays can easily lead to out of memory errors.

[ https://issues.apache.org/jira/browse/SPARK-44768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franck Tago updated SPARK-44768: Attachment: image-2023-08-10-20-32-55-684.png > Improve WSCG handling of row buffer by accounting

[jira] [Commented] (SPARK-43194) PySpark 3.4.0 cannot convert timestamp-typed objects to pandas with pandas 2.0

[ https://issues.apache.org/jira/browse/SPARK-43194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17753034#comment-17753034 ] Berg Lloyd-Haig commented on SPARK-43194: - This is affecting us also with parque

[jira] [Resolved] (SPARK-44771) Remove 'sudo' in 'pip install' suggestions in the dev scripts

[ https://issues.apache.org/jira/browse/SPARK-44771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-44771. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42444 [https:

[jira] [Resolved] (SPARK-42132) DeduplicateRelations rule breaks plan when co-grouping the same DataFrame

[ https://issues.apache.org/jira/browse/SPARK-42132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Fan resolved SPARK-42132. - Fix Version/s: 3.5.0 Resolution: Fixed > DeduplicateRelations rule breaks plan when co-grouping t

[jira] [Created] (SPARK-44771) Remove 'sudo' in 'pip install' suggestions in the dev scripts

Gengliang Wang created SPARK-44771: -- Summary: Remove 'sudo' in 'pip install' suggestions in the dev scripts Key: SPARK-44771 URL: https://issues.apache.org/jira/browse/SPARK-44771 Project: Spark

[jira] [Resolved] (SPARK-44763) Fix a bug of promoting string as double in binary arithmetic with interval

[ https://issues.apache.org/jira/browse/SPARK-44763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-44763. Target Version/s: 4.0.0 Resolution: Fixed > Fix a bug of promoting string as dou

[jira] [Commented] (SPARK-44763) Fix a bug of promoting string as double in binary arithmetic with interval

[ https://issues.apache.org/jira/browse/SPARK-44763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17753020#comment-17753020 ] Gengliang Wang commented on SPARK-44763: Resolved in https://github.com/apache/s

[jira] [Resolved] (SPARK-43781) IllegalStateException when cogrouping two datasets derived from the same source

[ https://issues.apache.org/jira/browse/SPARK-43781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-43781. - Fix Version/s: 3.5.0 Assignee: Jia Fan Resolution: Fixed > IllegalStateException

[jira] [Reopened] (SPARK-44760) Index Out Of Bound for JIRA resolution in merge_spark_pr

[ https://issues.apache.org/jira/browse/SPARK-44760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-44760: -- Assignee: (was: Kent Yao) Reverted at https://github.com/apache/spark/commit/3164ff51dd

[jira] [Updated] (SPARK-44760) Index Out Of Bound for JIRA resolution in merge_spark_pr

[ https://issues.apache.org/jira/browse/SPARK-44760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44760: - Fix Version/s: (was: 4.0.0) > Index Out Of Bound for JIRA resolution in merge_spark_pr > ---

[jira] [Assigned] (SPARK-43872) Enable DataFramePlotMatplotlibTests for pandas 2.0.0.

[ https://issues.apache.org/jira/browse/SPARK-43872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43872: Assignee: Haejoon Lee > Enable DataFramePlotMatplotlibTests for pandas 2.0.0. > -

[jira] [Resolved] (SPARK-44705) Make PythonRunner single-threaded

[ https://issues.apache.org/jira/browse/SPARK-44705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44705. -- Assignee: Utkarsh Agarwal Resolution: Fixed Fixed in https://github.com/apache/spark/pul

[jira] [Assigned] (SPARK-44765) Make ReleaseExecute retry in ExecutePlanResponseReattachableIterator reuse common mechanism

[ https://issues.apache.org/jira/browse/SPARK-44765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-44765: Assignee: Juliusz Sompolski > Make ReleaseExecute retry in ExecutePlanResponseReattachabl

[jira] [Updated] (SPARK-43032) Add StreamingQueryManager API

[ https://issues.apache.org/jira/browse/SPARK-43032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-43032: - Fix Version/s: 3.5.0 4.0.0 > Add StreamingQueryManager API >

[jira] [Resolved] (SPARK-44766) Cache the pandas converter for Python UDTFs

[ https://issues.apache.org/jira/browse/SPARK-44766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-44766. -- Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-44766) Cache the pandas converter for Python UDTFs

[ https://issues.apache.org/jira/browse/SPARK-44766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-44766: Assignee: Allison Wang > Cache the pandas converter for Python UDTFs > --

[jira] [Updated] (SPARK-44768) Improve WSCG handling of row buffer by accounting for executor memory . Exploding nested arrays can easily lead to out of memory errors.

[ https://issues.apache.org/jira/browse/SPARK-44768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franck Tago updated SPARK-44768: Description: consider a scenario where you flatten  a nested array  // e.g you can use the follow

[jira] [Updated] (SPARK-44768) Improve WSCG handling of row buffer by accounting for executor memory . Exploding nested arrays can easily lead to out of memory errors.

[ https://issues.apache.org/jira/browse/SPARK-44768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franck Tago updated SPARK-44768: Attachment: spark-jira_wscg_code.txt > Improve WSCG handling of row buffer by accounting for execu

[jira] [Created] (SPARK-44770) Add a displayOrder variable to WebUITab to specify the order in which tabs appear

Jason Li created SPARK-44770: Summary: Add a displayOrder variable to WebUITab to specify the order in which tabs appear Key: SPARK-44770 URL: https://issues.apache.org/jira/browse/SPARK-44770 Project: Sp

[jira] [Created] (SPARK-44769) Add SQL statement to create an empty array with a type

Holden Karau created SPARK-44769: Summary: Add SQL statement to create an empty array with a type Key: SPARK-44769 URL: https://issues.apache.org/jira/browse/SPARK-44769 Project: Spark Issue

[jira] [Updated] (SPARK-44768) Improve WSCG handling of row buffer by accounting for executor memory . Exploding nested arrays can easily lead to out of memory errors.

[ https://issues.apache.org/jira/browse/SPARK-44768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franck Tago updated SPARK-44768: Summary: Improve WSCG handling of row buffer by accounting for executor memory . Exploding neste

[jira] [Created] (SPARK-44768) Improve WSCG handling of row buffer by accounting for executor memory

Franck Tago created SPARK-44768: --- Summary: Improve WSCG handling of row buffer by accounting for executor memory Key: SPARK-44768 URL: https://issues.apache.org/jira/browse/SPARK-44768 Project: Spark

[jira] [Commented] (SPARK-44767) Plugin API for PySpark and SparkR workers

[ https://issues.apache.org/jira/browse/SPARK-44767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17752926#comment-17752926 ] Willi Raschkowski commented on SPARK-44767: --- I put up a proposal implementatio

[jira] [Updated] (SPARK-44767) Plugin API for PySpark and SparkR workers

[ https://issues.apache.org/jira/browse/SPARK-44767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Willi Raschkowski updated SPARK-44767: -- Summary: Plugin API for PySpark and SparkR workers (was: Plugin API for PySpark and S

[jira] [Created] (SPARK-44767) Plugin API for PySpark and SparkR subprocesses

Willi Raschkowski created SPARK-44767: - Summary: Plugin API for PySpark and SparkR subprocesses Key: SPARK-44767 URL: https://issues.apache.org/jira/browse/SPARK-44767 Project: Spark Issu

[jira] [Updated] (SPARK-44767) Plugin API for PySpark and SparkR subprocesses

[ https://issues.apache.org/jira/browse/SPARK-44767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Willi Raschkowski updated SPARK-44767: -- Description: An API to customize Python and R workers allows for extensibility beyond

[jira] [Created] (SPARK-44766) Cache the pandas converter for Python UDTFs

Allison Wang created SPARK-44766: Summary: Cache the pandas converter for Python UDTFs Key: SPARK-44766 URL: https://issues.apache.org/jira/browse/SPARK-44766 Project: Spark Issue Type: Sub-t

[jira] [Created] (SPARK-44765) Make ReleaseExecute retry in ExecutePlanResponseReattachableIterator reuse common mechanism

Juliusz Sompolski created SPARK-44765: - Summary: Make ReleaseExecute retry in ExecutePlanResponseReattachableIterator reuse common mechanism Key: SPARK-44765 URL: https://issues.apache.org/jira/browse/SPARK-44

[jira] [Created] (SPARK-44764) Streaming process improvement

Wei Liu created SPARK-44764: --- Summary: Streaming process improvement Key: SPARK-44764 URL: https://issues.apache.org/jira/browse/SPARK-44764 Project: Spark Issue Type: New Feature Compone

[jira] [Updated] (SPARK-44759) Do not combine multiple Generate operators in the same WholeStageCodeGen node because it can easily cause OOM failures if arrays are relatively large

[ https://issues.apache.org/jira/browse/SPARK-44759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franck Tago updated SPARK-44759: Description: This is an issue since the WSCG  implementation of the generate node.  Because WSCG

[jira] [Created] (SPARK-44763) Fix a bug of promoting string as double in binary arithmetic with interval

Gengliang Wang created SPARK-44763: -- Summary: Fix a bug of promoting string as double in binary arithmetic with interval Key: SPARK-44763 URL: https://issues.apache.org/jira/browse/SPARK-44763 Proj

[jira] [Updated] (SPARK-44759) Do not combine multiple Generate operators in the same WholeStageCodeGen node because it can easily cause OOM failures if arrays are relatively large

[ https://issues.apache.org/jira/browse/SPARK-44759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franck Tago updated SPARK-44759: Summary: Do not combine multiple Generate operators in the same WholeStageCodeGen node because it

[jira] [Updated] (SPARK-44759) Do not combine multiple Generate nodes in the same WholeStageCodeGen node because it can easily cause OOM failures if arrays are relatively large

[ https://issues.apache.org/jira/browse/SPARK-44759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franck Tago updated SPARK-44759: Summary: Do not combine multiple Generate nodes in the same WholeStageCodeGen node because it can

[jira] [Updated] (SPARK-44759) Do not combine multiple Generate nodes in the same WholeStageCodeGen nodebecause it can easily cause OOM failures if arrays are relatively large

[ https://issues.apache.org/jira/browse/SPARK-44759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franck Tago updated SPARK-44759: Description: This is an issue since the WSCG  implementation of the generate node.  The generate

[jira] [Updated] (SPARK-44759) Do not combine multiple Generate nodes in the same WholeStageCodeGen nodebecause it can easily cause OOM failures if arrays are relatively large

[ https://issues.apache.org/jira/browse/SPARK-44759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franck Tago updated SPARK-44759: Affects Version/s: 3.4.1 3.4.0 3.3.2

[jira] [Updated] (SPARK-44759) Do not combine multiple Generate nodes in the same WholeStageCodeGen nodebecause it can easily cause OOM failures if arrays are relatively large

[ https://issues.apache.org/jira/browse/SPARK-44759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franck Tago updated SPARK-44759: Description: The generate node used to flatten array generally  produces an amount of output rows

[jira] [Updated] (SPARK-44759) Do not combine multiple Generate nodes in the same WholeStageCodeGen nodebecause it can easily cause OOM failures if arrays are relatively large

[ https://issues.apache.org/jira/browse/SPARK-44759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franck Tago updated SPARK-44759: Description: The generate node used to flatten array generally  produces an amount of output rows

[jira] [Commented] (SPARK-44759) Do not combine multiple Generate nodes in the same WholeStageCodeGen nodebecause it can easily cause OOM failures if arrays are relatively large

[ https://issues.apache.org/jira/browse/SPARK-44759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17752870#comment-17752870 ] Franck Tago commented on SPARK-44759: - Spark Dag for the use case . The failure is f

[jira] [Updated] (SPARK-44759) Do not combine multiple Generate nodes in the same WholeStageCodeGen nodebecause it can easily cause OOM failures if arrays are relatively large

[ https://issues.apache.org/jira/browse/SPARK-44759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franck Tago updated SPARK-44759: Attachment: image-2023-08-10-09-33-47-788.png > Do not combine multiple Generate nodes in the same

[jira] [Commented] (SPARK-44759) Do not combine multiple Generate nodes in the same WholeStageCodeGen nodebecause it can easily cause OOM failures if arrays are relatively large

[ https://issues.apache.org/jira/browse/SPARK-44759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17752868#comment-17752868 ] Franck Tago commented on SPARK-44759: - WSCG  generated code for second Generate node

[jira] [Updated] (SPARK-44759) Do not combine multiple Generate nodes in the same WholeStageCodeGen nodebecause it can easily cause OOM failures if arrays are relatively large

[ https://issues.apache.org/jira/browse/SPARK-44759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franck Tago updated SPARK-44759: Attachment: image-2023-08-10-09-32-46-163.png > Do not combine multiple Generate nodes in the same

[jira] [Updated] (SPARK-44759) Do not combine multiple Generate nodes in the same WholeStageCodeGen nodebecause it can easily cause OOM failures if arrays are relatively large

[ https://issues.apache.org/jira/browse/SPARK-44759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franck Tago updated SPARK-44759: Attachment: image-2023-08-10-09-29-24-804.png > Do not combine multiple Generate nodes in the same

[jira] [Commented] (SPARK-44759) Do not combine multiple Generate nodes in the same WholeStageCodeGen nodebecause it can easily cause OOM failures if arrays are relatively large

[ https://issues.apache.org/jira/browse/SPARK-44759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17752866#comment-17752866 ] Franck Tago commented on SPARK-44759: - WSCG  generated code for first Generate node 

[jira] [Comment Edited] (SPARK-44759) Do not combine multiple Generate nodes in the same WholeStageCodeGen nodebecause it can easily cause OOM failures if arrays are relatively large

[ https://issues.apache.org/jira/browse/SPARK-44759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17752864#comment-17752864 ] Franck Tago edited comment on SPARK-44759 at 8/10/23 4:28 PM:

[jira] [Updated] (SPARK-44759) Do not combine multiple Generate nodes in the same WholeStageCodeGen nodebecause it can easily cause OOM failures if arrays are relatively large

[ https://issues.apache.org/jira/browse/SPARK-44759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franck Tago updated SPARK-44759: Attachment: image-2023-08-10-09-27-24-124.png > Do not combine multiple Generate nodes in the same

[jira] [Commented] (SPARK-44759) Do not combine multiple Generate nodes in the same WholeStageCodeGen nodebecause it can easily cause OOM failures if arrays are relatively large

[ https://issues.apache.org/jira/browse/SPARK-44759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17752864#comment-17752864 ] Franck Tago commented on SPARK-44759: - !image-2023-08-10-09-27-24-124.png! > Do not

[jira] [Updated] (SPARK-44759) Do not combine multiple Generate nodes in the same WholeStageCodeGen nodebecause it can easily cause OOM failures if arrays are relatively large

[ https://issues.apache.org/jira/browse/SPARK-44759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franck Tago updated SPARK-44759: Summary: Do not combine multiple Generate nodes in the same WholeStageCodeGen nodebecause it can

[jira] [Updated] (SPARK-44759) Do not combine multiple Generate nodes in the same WholeStageCodeGen because it can easily cause OOM failures

[ https://issues.apache.org/jira/browse/SPARK-44759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franck Tago updated SPARK-44759: Description: The generate node used to flatten array generally  produces an amount of output rows

[jira] [Resolved] (SPARK-44760) Index Out Of Bound for JIRA resolution in merge_spark_pr

[ https://issues.apache.org/jira/browse/SPARK-44760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44760. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42429 [https://

[jira] [Created] (SPARK-44762) Add more documentation and examples for using job tags for interrupt

Juliusz Sompolski created SPARK-44762: - Summary: Add more documentation and examples for using job tags for interrupt Key: SPARK-44762 URL: https://issues.apache.org/jira/browse/SPARK-44762 Projec

[jira] [Updated] (SPARK-44741) Support regex-based MetricFilter in StatsdSink

[ https://issues.apache.org/jira/browse/SPARK-44741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44741: -- Affects Version/s: 4.0.0 (was: 3.4.1) > Support regex-based MetricF

[jira] [Updated] (SPARK-44741) Support regex-based MetricFilter in StatsdSink

[ https://issues.apache.org/jira/browse/SPARK-44741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44741: -- Summary: Support regex-based MetricFilter in StatsdSink (was: Spark StatsD metrics reported t

[jira] [Assigned] (SPARK-44741) Spark StatsD metrics reported to support metrics filter option

[ https://issues.apache.org/jira/browse/SPARK-44741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-44741: - Assignee: rameshkrishnan muthusamy > Spark StatsD metrics reported to support metrics f

[jira] [Resolved] (SPARK-44741) Spark StatsD metrics reported to support metrics filter option

[ https://issues.apache.org/jira/browse/SPARK-44741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44741. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42416 [https://

[jira] [Updated] (SPARK-44700) Rule OptimizeCsvJsonExprs should not be applied to expression like from_json(regexp_replace)

[ https://issues.apache.org/jira/browse/SPARK-44700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-44700: Fix Version/s: 3.3.0 > Rule OptimizeCsvJsonExprs should not be applied to expression like > from_

[jira] [Resolved] (SPARK-44700) Rule OptimizeCsvJsonExprs should not be applied to expression like from_json(regexp_replace)

[ https://issues.apache.org/jira/browse/SPARK-44700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-44700. - Resolution: Fixed Please upgrade Spark to the latest version to fix this issue. > Rule Optimize

[jira] [Updated] (SPARK-44700) Rule OptimizeCsvJsonExprs should not be applied to expression like from_json(regexp_replace)

[ https://issues.apache.org/jira/browse/SPARK-44700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-44700: Affects Version/s: 3.1.1 (was: 3.4.0) (was: 3

[jira] [Created] (SPARK-44761) Add DataStreamWriter.foreachBatch(org.apache.spark.api.java.function.VoidFunction2) signature

Herman van Hövell created SPARK-44761: - Summary: Add DataStreamWriter.foreachBatch(org.apache.spark.api.java.function.VoidFunction2) signature Key: SPARK-44761 URL: https://issues.apache.org/jira/browse/SPAR

[jira] [Commented] (SPARK-44756) Executor hangs when RetryingBlockTransferor fails to initiate retry

[ https://issues.apache.org/jira/browse/SPARK-44756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17752785#comment-17752785 ] GridGain Integration commented on SPARK-44756: -- User 'hdaikoku' has created

[jira] [Commented] (SPARK-44756) Executor hangs when RetryingBlockTransferor fails to initiate retry

[ https://issues.apache.org/jira/browse/SPARK-44756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17752781#comment-17752781 ] Harunobu Daikoku commented on SPARK-44756: -- PR raised: https://github.com/apach

[jira] [Updated] (SPARK-44756) Executor hangs when RetryingBlockTransferor fails to initiate retry

[ https://issues.apache.org/jira/browse/SPARK-44756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harunobu Daikoku updated SPARK-44756: - Component/s: Spark Core > Executor hangs when RetryingBlockTransferor fails to initiate

[jira] [Created] (SPARK-44760) Index Out Of Bound for JIRA resolution in merge_spark_pr

Kent Yao created SPARK-44760: Summary: Index Out Of Bound for JIRA resolution in merge_spark_pr Key: SPARK-44760 URL: https://issues.apache.org/jira/browse/SPARK-44760 Project: Spark Issue Type:

[jira] [Updated] (SPARK-44758) Support memory limit configurable

[ https://issues.apache.org/jira/browse/SPARK-44758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuml updated SPARK-44758: -- Description: Currently the memory request and limit are set by summing the values of spark.\{driver,executor}.

[jira] [Updated] (SPARK-44758) Support memory limit configurable

[ https://issues.apache.org/jira/browse/SPARK-44758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuml updated SPARK-44758: -- Summary: Support memory limit configurable (was: Support memory limit) > Support memory limit configurable >

[jira] [Updated] (SPARK-44759) Do not combine multiple Generate nodes in the same WholeStageCodeGen because it can easily cause OOM failures

[ https://issues.apache.org/jira/browse/SPARK-44759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franck Tago updated SPARK-44759: Attachment: (was: spark-verbosewithcodegenenabled) > Do not combine multiple Generate nodes in

[jira] [Updated] (SPARK-44759) Do not combine multiple Generate nodes in the same WholeStageCodeGen because it can easily cause OOM failures

[ https://issues.apache.org/jira/browse/SPARK-44759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franck Tago updated SPARK-44759: Attachment: wholestagecodegen_wc1_debug_wholecodegen_passed > Do not combine multiple Generate nod

[jira] [Updated] (SPARK-44759) Do not combine multiple Generate nodes in the same WholeStageCodeGen because it can easily cause OOM failures

[ https://issues.apache.org/jira/browse/SPARK-44759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Franck Tago updated SPARK-44759: Attachment: spark-verbosewithcodegenenabled > Do not combine multiple Generate nodes in the same W

[jira] [Created] (SPARK-44759) Do not combine multiple Generate nodes in the same WholeStageCodeGen because it can easily cause OOM failures

Franck Tago created SPARK-44759: --- Summary: Do not combine multiple Generate nodes in the same WholeStageCodeGen because it can easily cause OOM failures Key: SPARK-44759 URL: https://issues.apache.org/jira/browse/S

[jira] [Updated] (SPARK-44757) Vulnerabilities in Spark3.4

[ https://issues.apache.org/jira/browse/SPARK-44757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anand Balasubramaniam updated SPARK-44757: -- Priority: Major (was: Minor) > Vulnerabilities in Spark3.4 >

[jira] [Created] (SPARK-44758) Support memory limit

zhuml created SPARK-44758: - Summary: Support memory limit Key: SPARK-44758 URL: https://issues.apache.org/jira/browse/SPARK-44758 Project: Spark Issue Type: Improvement Components: Kubernet

[jira] [Created] (SPARK-44757) Vulnerabilities in Spark3.4

Anand Balasubramaniam created SPARK-44757: - Summary: Vulnerabilities in Spark3.4 Key: SPARK-44757 URL: https://issues.apache.org/jira/browse/SPARK-44757 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-43705) Enable TimedeltaIndexTests.test_properties for pandas 2.0.0.

[ https://issues.apache.org/jira/browse/SPARK-43705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43705: Assignee: Haejoon Lee > Enable TimedeltaIndexTests.test_properties for pandas 2.0.0. > --

[jira] [Resolved] (SPARK-43245) Fix DatetimeIndex.microsecond to return 'int32' instead of 'int64' type of Index.

[ https://issues.apache.org/jira/browse/SPARK-43245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43245. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42271 [https://gi

[jira] [Resolved] (SPARK-43705) Enable TimedeltaIndexTests.test_properties for pandas 2.0.0.

[ https://issues.apache.org/jira/browse/SPARK-43705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43705. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42271 [https://gi

[jira] [Assigned] (SPARK-43245) Fix DatetimeIndex.microsecond to return 'int32' instead of 'int64' type of Index.

[ https://issues.apache.org/jira/browse/SPARK-43245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-43245: Assignee: Haejoon Lee > Fix DatetimeIndex.microsecond to return 'int32' instead of 'int64

[jira] [Commented] (SPARK-44742) Add Spark version drop down to the PySpark doc site

[ https://issues.apache.org/jira/browse/SPARK-44742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17752689#comment-17752689 ] BingKun Pan commented on SPARK-44742: - I work on it. > Add Spark version drop down

[jira] [Updated] (SPARK-44756) Executor hangs when RetryingBlockTransferor fails to initiate retry

[ https://issues.apache.org/jira/browse/SPARK-44756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harunobu Daikoku updated SPARK-44756: - Description: We have been observing this issue several times in our production where som

[jira] [Updated] (SPARK-44756) Executor hangs when RetryingBlockTransferor fails to initiate retry

[ https://issues.apache.org/jira/browse/SPARK-44756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harunobu Daikoku updated SPARK-44756: - Description: We have been observing this issue several times in our production where som

[jira] [Updated] (SPARK-44756) Executor hangs when RetryingBlockTransferor fails to initiate retry

[ https://issues.apache.org/jira/browse/SPARK-44756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harunobu Daikoku updated SPARK-44756: - Summary: Executor hangs when RetryingBlockTransferor fails to initiate retry (was: Exec

[jira] [Created] (SPARK-44756) Executor hangs when RetryingBlockTransferor fails to submit retry request

Harunobu Daikoku created SPARK-44756: Summary: Executor hangs when RetryingBlockTransferor fails to submit retry request Key: SPARK-44756 URL: https://issues.apache.org/jira/browse/SPARK-44756 Pro

[jira] [Updated] (SPARK-44573) Couldn't submit Spark application to Kubenetes in versions v1.27.3

[ https://issues.apache.org/jira/browse/SPARK-44573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddaraju G C updated SPARK-44573: -- Priority: Blocker (was: Major) > Couldn't submit Spark application to Kubenetes in versions v