[jira] [Assigned] (SPARK-44909) Skip starting torch distributor log streaming server when it is not available

[ https://issues.apache.org/jira/browse/SPARK-44909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu reassigned SPARK-44909: -- Assignee: Weichen Xu > Skip starting torch distributor log streaming server when it is not av

[jira] [Commented] (SPARK-42905) pyspark.ml.stat.Correlation - Spearman Correlation method giving incorrect and inconsistent results for the same DataFrame if it has huge amount of Ties.

[ https://issues.apache.org/jira/browse/SPARK-42905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17757858#comment-17757858 ] zhangzhenhao commented on SPARK-42905: -- minimal reproducible example, the result is

[jira] [Resolved] (SPARK-44909) Skip starting torch distributor log streaming server when it is not available

[ https://issues.apache.org/jira/browse/SPARK-44909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu resolved SPARK-44909. Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by pull requ

[jira] [Comment Edited] (SPARK-42905) pyspark.ml.stat.Correlation - Spearman Correlation method giving incorrect and inconsistent results for the same DataFrame if it has huge amount of Ties.

[ https://issues.apache.org/jira/browse/SPARK-42905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17757858#comment-17757858 ] zhangzhenhao edited comment on SPARK-42905 at 8/23/23 7:35 AM: ---

[jira] [Updated] (SPARK-44871) Fix PERCENTILE_DISC behaviour

[ https://issues.apache.org/jira/browse/SPARK-44871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-44871: - Fix Version/s: 3.3.4 > Fix PERCENTILE_DISC behaviour > - > >

[jira] [Updated] (SPARK-44871) Fix PERCENTILE_DISC behaviour

[ https://issues.apache.org/jira/browse/SPARK-44871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-44871: - Fix Version/s: 4.0.0 > Fix PERCENTILE_DISC behaviour > - > >

[jira] [Created] (SPARK-44927) Replace the module alias 'sf' instead of 'F' in pyspark.sql import functions

Hyukjin Kwon created SPARK-44927: Summary: Replace the module alias 'sf' instead of 'F' in pyspark.sql import functions Key: SPARK-44927 URL: https://issues.apache.org/jira/browse/SPARK-44927 Project:

[jira] [Created] (SPARK-44928) Replace the module alias 'sf' instead of 'F' in pyspark.sql import functions

Hyukjin Kwon created SPARK-44928: Summary: Replace the module alias 'sf' instead of 'F' in pyspark.sql import functions Key: SPARK-44928 URL: https://issues.apache.org/jira/browse/SPARK-44928 Project:

[jira] [Created] (SPARK-44926) Replace the module alias 'sf' instead of 'F' in pyspark.sql import functions

Hyukjin Kwon created SPARK-44926: Summary: Replace the module alias 'sf' instead of 'F' in pyspark.sql import functions Key: SPARK-44926 URL: https://issues.apache.org/jira/browse/SPARK-44926 Project:

[jira] [Updated] (SPARK-44871) Fix PERCENTILE_DISC behaviour

[ https://issues.apache.org/jira/browse/SPARK-44871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-44871: --- Fix Version/s: 3.5.0 > Fix PERCENTILE_DISC behaviour > - > >

[jira] [Created] (SPARK-44929) Truncate log output for console appender in tests

Kent Yao created SPARK-44929: Summary: Truncate log output for console appender in tests Key: SPARK-44929 URL: https://issues.apache.org/jira/browse/SPARK-44929 Project: Spark Issue Type: Improve

[jira] [Deleted] (SPARK-44926) Replace the module alias 'sf' instead of 'F' in pyspark.sql import functions

[ https://issues.apache.org/jira/browse/SPARK-44926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon deleted SPARK-44926: - > Replace the module alias 'sf' instead of 'F' in pyspark.sql import functions > --

[jira] [Deleted] (SPARK-44927) Replace the module alias 'sf' instead of 'F' in pyspark.sql import functions

[ https://issues.apache.org/jira/browse/SPARK-44927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon deleted SPARK-44927: - > Replace the module alias 'sf' instead of 'F' in pyspark.sql import functions > --

[jira] [Updated] (SPARK-44928) Replace the module alias 'sf' instead of 'F' in pyspark.sql import functions

[ https://issues.apache.org/jira/browse/SPARK-44928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-44928: - Description: {code} from pyspark.sql import functions as F {code} isn’t very Pythonic - it does

[jira] [Created] (SPARK-44930) Deterministic ApplyFunctionExpression should be foldable

Xianyang Liu created SPARK-44930: Summary: Deterministic ApplyFunctionExpression should be foldable Key: SPARK-44930 URL: https://issues.apache.org/jira/browse/SPARK-44930 Project: Spark Issu

[jira] [Updated] (SPARK-44930) Deterministic ApplyFunctionExpression should be foldable

[ https://issues.apache.org/jira/browse/SPARK-44930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyang Liu updated SPARK-44930: - Description: Currently, ApplyFunctionExpression is unfoldable because inherits the default value

[jira] [Commented] (SPARK-44219) Add extra per-rule validation for optimization rewrites.

[ https://issues.apache.org/jira/browse/SPARK-44219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17757892#comment-17757892 ] ASF GitHub Bot commented on SPARK-44219: User 'YannisSismanis' has created a pul

[jira] [Updated] (SPARK-44930) Deterministic ApplyFunctionExpression should be foldable

[ https://issues.apache.org/jira/browse/SPARK-44930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyang Liu updated SPARK-44930: - Description: Currently, ApplyFunctionExpression is unfoldable because inherits the default value

[jira] [Commented] (SPARK-44930) Deterministic ApplyFunctionExpression should be foldable

[ https://issues.apache.org/jira/browse/SPARK-44930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17757893#comment-17757893 ] ASF GitHub Bot commented on SPARK-44930: User 'ConeyLiu' has created a pull requ

[jira] [Commented] (SPARK-44928) Replace the module alias 'sf' instead of 'F' in pyspark.sql import functions

[ https://issues.apache.org/jira/browse/SPARK-44928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17757894#comment-17757894 ] ASF GitHub Bot commented on SPARK-44928: User 'HyukjinKwon' has created a pull r

[jira] [Resolved] (SPARK-44923) Some directories should be cleared when regenerating files

[ https://issues.apache.org/jira/browse/SPARK-44923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-44923. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42622 [https://

[jira] [Assigned] (SPARK-44923) Some directories should be cleared when regenerating files

[ https://issues.apache.org/jira/browse/SPARK-44923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-44923: - Assignee: BingKun Pan > Some directories should be cleared when regenerating files > --

[jira] [Commented] (SPARK-44923) Some directories should be cleared when regenerating files

[ https://issues.apache.org/jira/browse/SPARK-44923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17757898#comment-17757898 ] ASF GitHub Bot commented on SPARK-44923: User 'panbingkun' has created a pull re

[jira] [Commented] (SPARK-44923) Some directories should be cleared when regenerating files

[ https://issues.apache.org/jira/browse/SPARK-44923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17757897#comment-17757897 ] ASF GitHub Bot commented on SPARK-44923: User 'panbingkun' has created a pull re

[jira] [Commented] (SPARK-44846) PushFoldableIntoBranches in complex grouping expressions may cause bindReference error

[ https://issues.apache.org/jira/browse/SPARK-44846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17757899#comment-17757899 ] ASF GitHub Bot commented on SPARK-44846: User 'zml1206' has created a pull reque

[jira] [Resolved] (SPARK-44908) Fix spark connect ML crossvalidator "foldCol" param

[ https://issues.apache.org/jira/browse/SPARK-44908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu resolved SPARK-44908. Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by pull requ

[jira] [Created] (SPARK-44931) Fix JSON Serailization for Spark Connect Event Listener

Martin Grund created SPARK-44931: Summary: Fix JSON Serailization for Spark Connect Event Listener Key: SPARK-44931 URL: https://issues.apache.org/jira/browse/SPARK-44931 Project: Spark Issue

[jira] [Resolved] (SPARK-44914) Upgrade Apache ivy to 2.5.2

[ https://issues.apache.org/jira/browse/SPARK-44914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-44914. -- Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-44914) Upgrade Apache ivy to 2.5.2

[ https://issues.apache.org/jira/browse/SPARK-44914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-44914: Assignee: Bjørn Jørgensen > Upgrade Apache ivy to 2.5.2 > > >

[jira] [Resolved] (SPARK-44549) Support correlated references under window functions

[ https://issues.apache.org/jira/browse/SPARK-44549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44549. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42383 [https://gith

[jira] [Assigned] (SPARK-44549) Support correlated references under window functions

[ https://issues.apache.org/jira/browse/SPARK-44549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44549: --- Assignee: Andrey Gubichev > Support correlated references under window functions >

[jira] [Commented] (SPARK-44931) Fix JSON Serailization for Spark Connect Event Listener

[ https://issues.apache.org/jira/browse/SPARK-44931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17758107#comment-17758107 ] GridGain Integration commented on SPARK-44931: -- User 'grundprinzip' has cre

[jira] [Updated] (SPARK-44900) Cached DataFrame keeps growing

[ https://issues.apache.org/jira/browse/SPARK-44900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Nalla updated SPARK-44900: Priority: Blocker (was: Critical) > Cached DataFrame keeps growing >

[jira] [Created] (SPARK-44932) Continuous Structured Streaming

Bryan Qiang created SPARK-44932: --- Summary: Continuous Structured Streaming Key: SPARK-44932 URL: https://issues.apache.org/jira/browse/SPARK-44932 Project: Spark Issue Type: Bug Comp

[jira] [Updated] (SPARK-44932) Continuous Structured Streaming not reporting streaming metrics

[ https://issues.apache.org/jira/browse/SPARK-44932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Qiang updated SPARK-44932: Summary: Continuous Structured Streaming not reporting streaming metrics (was: Continuous Structu

[jira] [Commented] (SPARK-23887) update query progress

[ https://issues.apache.org/jira/browse/SPARK-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17758145#comment-17758145 ] Bryan Qiang commented on SPARK-23887: - Hello folks, I'm wondering what's the final d

[jira] [Assigned] (SPARK-44861) [CONNECT] jsonignore SparkListenerConnectOperationStarted.planRequest

[ https://issues.apache.org/jira/browse/SPARK-44861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell reassigned SPARK-44861: - Assignee: Jean-Francois Desjeans Gauthier > [CONNECT] jsonignore SparkListenerC

[jira] [Resolved] (SPARK-44861) [CONNECT] jsonignore SparkListenerConnectOperationStarted.planRequest

[ https://issues.apache.org/jira/browse/SPARK-44861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-44861. --- Fix Version/s: 3.5.0 Resolution: Fixed > [CONNECT] jsonignore SparkListenerCo

[jira] [Resolved] (SPARK-44816) Cryptic error message when UDF associated class is not found

[ https://issues.apache.org/jira/browse/SPARK-44816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-44816. --- Fix Version/s: 3.5.0 Resolution: Fixed > Cryptic error message when UDF assoc

[jira] [Assigned] (SPARK-44816) Cryptic error message when UDF associated class is not found

[ https://issues.apache.org/jira/browse/SPARK-44816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell reassigned SPARK-44816: - Assignee: Niranjan Jayakar > Cryptic error message when UDF associated class is

[jira] [Commented] (SPARK-44900) Cached DataFrame keeps growing

[ https://issues.apache.org/jira/browse/SPARK-44900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17758161#comment-17758161 ] Varun Nalla commented on SPARK-44900: - [~yao] , is there a way I could prioritize th

[jira] [Commented] (SPARK-44884) Spark doesn't create SUCCESS file when external path is passed

[ https://issues.apache.org/jira/browse/SPARK-44884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17758167#comment-17758167 ] Steve Loughran commented on SPARK-44884: i'm not trying to replicate it; i have

[jira] [Assigned] (SPARK-44906) Move substituteAppNExecIds logic into kubernetesConf.annotations method

[ https://issues.apache.org/jira/browse/SPARK-44906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-44906: - Assignee: Binjie Yang > Move substituteAppNExecIds logic into kubernetesConf.annotation

[jira] [Resolved] (SPARK-44906) Move substituteAppNExecIds logic into kubernetesConf.annotations method

[ https://issues.apache.org/jira/browse/SPARK-44906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44906. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42600 [https://

[jira] [Updated] (SPARK-44906) Make Kubernetes[Driver|Executor]Conf.annotations substitute annotations instead of feature steps

[ https://issues.apache.org/jira/browse/SPARK-44906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44906: -- Priority: Minor (was: Major) > Make Kubernetes[Driver|Executor]Conf.annotations substitute an

[jira] [Updated] (SPARK-44906) Make Kubernetes[Driver|Executor]Conf.annotations substitute annotations instead of feature steps

[ https://issues.apache.org/jira/browse/SPARK-44906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44906: -- Affects Version/s: 4.0.0 (was: 3.4.1) > Make Kubernetes[Driver|Exec

[jira] [Updated] (SPARK-44906) Make Kubernetes[Driver|Executor]Conf.annotations substitute annotations instead of feature steps

[ https://issues.apache.org/jira/browse/SPARK-44906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44906: -- Summary: Make Kubernetes[Driver|Executor]Conf.annotations substitute annotations instead of fe

[jira] [Created] (SPARK-44933) Spark structured streaming performance regression in latency times reading/writing to kafka since 3.0.2

eddie baggott created SPARK-44933: - Summary: Spark structured streaming performance regression in latency times reading/writing to kafka since 3.0.2 Key: SPARK-44933 URL: https://issues.apache.org/jira/browse/SPAR

[jira] [Updated] (SPARK-44933) Spark structured streaming performance regression in latency times reading/writing to kafka since 3.0.2

[ https://issues.apache.org/jira/browse/SPARK-44933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] eddie baggott updated SPARK-44933: -- Description:  During a migration from spark 2.4.4 to spark 3.4.0 I have noticed slower latenc

[jira] [Created] (SPARK-44934) PushdownPredicatesAndPruneColumnsForCTEDef creates invalid plan when called over CTE with duplicate attributes

Wen Yuen Pang created SPARK-44934: - Summary: PushdownPredicatesAndPruneColumnsForCTEDef creates invalid plan when called over CTE with duplicate attributes Key: SPARK-44934 URL: https://issues.apache.org/jira/brow

[jira] [Updated] (SPARK-44934) PushdownPredicatesAndPruneColumnsForCTEDef creates invalid plan when called over CTE with duplicate attributes

[ https://issues.apache.org/jira/browse/SPARK-44934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wen Yuen Pang updated SPARK-44934: -- Description: When running the query {code:java} with cte as (  select c1, c1, c2, c3 from t wh

[jira] [Created] (SPARK-44935) Fix `RELEASE` file to have the correct information in Docker images

Dongjoon Hyun created SPARK-44935: - Summary: Fix `RELEASE` file to have the correct information in Docker images Key: SPARK-44935 URL: https://issues.apache.org/jira/browse/SPARK-44935 Project: Spark

[jira] [Created] (SPARK-44936) Simplify the log when Spark HybridStore hits the memory limit

Dongjoon Hyun created SPARK-44936: - Summary: Simplify the log when Spark HybridStore hits the memory limit Key: SPARK-44936 URL: https://issues.apache.org/jira/browse/SPARK-44936 Project: Spark

[jira] [Updated] (SPARK-44936) Simplify the log when Spark HybridStore hits the memory limit

[ https://issues.apache.org/jira/browse/SPARK-44936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44936: -- Description: *BEFORE* {code} 23/08/23 22:40:34 INFO FsHistoryProvider: Failed to create Hybrid

[jira] [Resolved] (SPARK-44935) Fix `RELEASE` file to have the correct information in Docker images

[ https://issues.apache.org/jira/browse/SPARK-44935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44935. --- Fix Version/s: 3.3.4 3.5.0 4.0.0 3.4

[jira] [Assigned] (SPARK-44935) Fix `RELEASE` file to have the correct information in Docker images

[ https://issues.apache.org/jira/browse/SPARK-44935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-44935: - Assignee: Dongjoon Hyun > Fix `RELEASE` file to have the correct information in Docker

[jira] [Created] (SPARK-44937) Add SSL/TLS support for RPC and Shuffle communications

Hasnain Lakhani created SPARK-44937: --- Summary: Add SSL/TLS support for RPC and Shuffle communications Key: SPARK-44937 URL: https://issues.apache.org/jira/browse/SPARK-44937 Project: Spark

[jira] [Commented] (SPARK-6373) Add SSL/TLS for the Netty based BlockTransferService

[ https://issues.apache.org/jira/browse/SPARK-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17758289#comment-17758289 ] Hasnain Lakhani commented on SPARK-6373: FYI: I filed https://issues.apache.org/j

[jira] [Resolved] (SPARK-44750) SparkSession.Builder should respect the options

[ https://issues.apache.org/jira/browse/SPARK-44750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-44750. --- Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by pul

[jira] [Resolved] (SPARK-42017) df["bad_key"] does not raise AnalysisException

[ https://issues.apache.org/jira/browse/SPARK-42017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-42017. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42608 [https://

[jira] [Assigned] (SPARK-44903) Refine docstring of `approx_count_distinct`

[ https://issues.apache.org/jira/browse/SPARK-44903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-44903: - Assignee: Yang Jie > Refine docstring of `approx_count_distinct` >

[jira] [Resolved] (SPARK-44903) Refine docstring of `approx_count_distinct`

[ https://issues.apache.org/jira/browse/SPARK-44903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-44903. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42596 [https://

[jira] [Created] (SPARK-44938) Change default value of spark.sql.maxSinglePartitionBytes to 128m

Cheng Pan created SPARK-44938: - Summary: Change default value of spark.sql.maxSinglePartitionBytes to 128m Key: SPARK-44938 URL: https://issues.apache.org/jira/browse/SPARK-44938 Project: Spark

[jira] [Commented] (SPARK-44820) Switch languages consistently across docs for all code snippets

[ https://issues.apache.org/jira/browse/SPARK-44820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17758314#comment-17758314 ] BingKun Pan commented on SPARK-44820: - Let me try to investigate it. > Switch langu

[jira] [Commented] (SPARK-44900) Cached DataFrame keeps growing

[ https://issues.apache.org/jira/browse/SPARK-44900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17758315#comment-17758315 ] Kent Yao commented on SPARK-44900: -- How about releasing the cached rdds if you never to

[jira] [Commented] (SPARK-44900) Cached DataFrame keeps growing

[ https://issues.apache.org/jira/browse/SPARK-44900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17758316#comment-17758316 ] Varun Nalla commented on SPARK-44900: - [~yao]  Thanks for the comment. However we ca

[jira] [Resolved] (SPARK-44936) Simplify the log when Spark HybridStore hits the memory limit

[ https://issues.apache.org/jira/browse/SPARK-44936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44936. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42638 [https://

[jira] [Assigned] (SPARK-44936) Simplify the log when Spark HybridStore hits the memory limit

[ https://issues.apache.org/jira/browse/SPARK-44936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-44936: - Assignee: Dongjoon Hyun > Simplify the log when Spark HybridStore hits the memory limit

[jira] [Created] (SPARK-44939) Support Java 21 in SparkR SystemRequirements

Dongjoon Hyun created SPARK-44939: - Summary: Support Java 21 in SparkR SystemRequirements Key: SPARK-44939 URL: https://issues.apache.org/jira/browse/SPARK-44939 Project: Spark Issue Type: Su

[jira] [Commented] (SPARK-44939) Support Java 21 in SparkR SystemRequirements

[ https://issues.apache.org/jira/browse/SPARK-44939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17758356#comment-17758356 ] Hudson commented on SPARK-44939: User 'dongjoon-hyun' has created a pull request for thi

[jira] [Resolved] (SPARK-44452) Ignore ArrowEncoderSuite for Java 21

[ https://issues.apache.org/jira/browse/SPARK-44452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-44452. -- Resolution: Won't Fix Arrow 13.0 has already been released, this fix is no longer needed. > Ignore Ar

[jira] [Assigned] (SPARK-44939) Support Java 21 in SparkR SystemRequirements

[ https://issues.apache.org/jira/browse/SPARK-44939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-44939: - Assignee: Dongjoon Hyun > Support Java 21 in SparkR SystemRequirements > --

[jira] [Resolved] (SPARK-44939) Support Java 21 in SparkR SystemRequirements

[ https://issues.apache.org/jira/browse/SPARK-44939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-44939. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42645 [https://

[jira] [Commented] (SPARK-44900) Cached DataFrame keeps growing

[ https://issues.apache.org/jira/browse/SPARK-44900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17758376#comment-17758376 ] Kent Yao commented on SPARK-44900: -- Please note that there is a storage limit in place.

[jira] [Resolved] (SPARK-44929) Truncate log output for console appender in tests

[ https://issues.apache.org/jira/browse/SPARK-44929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-44929. -- Fix Version/s: 3.4.2 3.5.0 4.0.0 Assignee: Kent Yao

[jira] [Updated] (SPARK-44929) Standardize log output for console appender in tests

[ https://issues.apache.org/jira/browse/SPARK-44929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-44929: - Summary: Standardize log output for console appender in tests (was: Truncate log output for console app

[jira] [Created] (SPARK-44940) Improve performance of JSON parsing when partial results are enabled

Ivan Sadikov created SPARK-44940: Summary: Improve performance of JSON parsing when partial results are enabled Key: SPARK-44940 URL: https://issues.apache.org/jira/browse/SPARK-44940 Project: Spark

[jira] [Updated] (SPARK-44940) Improve performance of JSON parsing when partial results are enabled

[ https://issues.apache.org/jira/browse/SPARK-44940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Sadikov updated SPARK-44940: - Description: Follow-up on https://issues.apache.org/jira/browse/SPARK-40646. I found that JSON

[jira] [Commented] (SPARK-44940) Improve performance of JSON parsing when partial results are enabled

[ https://issues.apache.org/jira/browse/SPARK-44940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17758394#comment-17758394 ] Ivan Sadikov commented on SPARK-44940: -- I have prototyped the fix and will open a P