[jira] [Created] (SPARK-45265) Support Hive 4.0 metastore

2023-09-21 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-45265: -- Summary: Support Hive 4.0 metastore Key: SPARK-45265 URL: https://issues.apache.org/jira/browse/SPARK-45265 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-44717) "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not help

2023-08-08 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751962#comment-17751962 ] Attila Zsolt Piros edited comment on SPARK-44717 at 8/8/23 8:56 AM:

[jira] [Commented] (SPARK-44717) "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not help

2023-08-08 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751962#comment-17751962 ] Attila Zsolt Piros commented on SPARK-44717: The TIMESTAMP_NTZ would work for sure. Here is

[jira] [Updated] (SPARK-44717) "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not help

2023-08-07 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-44717: --- Description: Use one of the existing test: - "11H" case of test_dataframe_resample

[jira] [Updated] (SPARK-44717) "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not help

2023-08-07 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-44717: --- Summary: "pyspark.pandas.resample" is incorrect when DST is overlapped and setting

[jira] [Commented] (SPARK-44717) "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not applied

2023-08-07 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751886#comment-17751886 ] Attila Zsolt Piros commented on SPARK-44717: cc [~itholic] [~gurwls223] >

[jira] [Updated] (SPARK-44717) "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not applied

2023-08-07 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-44717: --- Description: Use one of the existing test: - "11H" case of test_dataframe_resample

[jira] [Updated] (SPARK-44717) "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not applied

2023-08-07 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-44717: --- Description: Use one of the existing test: - "11H" case of test_dataframe_resample

[jira] [Created] (SPARK-44717) "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not applied

2023-08-07 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-44717: -- Summary: "pyspark.pandas.resample" is incorrect when DST is overlapped and setting "spark.sql.timestampType" to TIMESTAMP_NTZ does not applied Key: SPARK-44717 URL:

[jira] [Resolved] (SPARK-44198) Support propagation of the log level to the executors

2023-07-27 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros resolved SPARK-44198. Fix Version/s: 4.0.0 Assignee: Vinod KC Resolution: Fixed Issue

[jira] [Updated] (SPARK-44253) Potential memory leak when temp views created from DF created by structured streaming

2023-06-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-44253: --- Description: If the user registers a temporary view from a dataframe created by

[jira] [Updated] (SPARK-44253) Potential memory leak when temp views created from DF created by structured streaming

2023-06-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-44253: --- Description: If the user registers a temporary view from a dataframe created by

[jira] [Updated] (SPARK-44253) Potential memory leak when temp views created from DF created by structured streaming

2023-06-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-44253: --- Attachment: 2.png > Potential memory leak when temp views created from DF created

[jira] [Updated] (SPARK-44253) Potential memory leak when temp views created from DF created by structured streaming

2023-06-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-44253: --- Description: If the user registers a temporary view from a dataframe created by

[jira] [Created] (SPARK-44253) Potential memory leak when temp views created from DF created by structured streaming

2023-06-29 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-44253: -- Summary: Potential memory leak when temp views created from DF created by structured streaming Key: SPARK-44253 URL: https://issues.apache.org/jira/browse/SPARK-44253

[jira] [Updated] (SPARK-44253) Potential memory leak when temp views created from DF created by structured streaming

2023-06-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-44253: --- Attachment: 1.png > Potential memory leak when temp views created from DF created

[jira] [Comment Edited] (SPARK-42068) Implicit conversion is not working with parallelization in scala with java 11 and spark3

2023-01-31 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682768#comment-17682768 ] Attila Zsolt Piros edited comment on SPARK-42068 at 1/31/23 9:42 PM: -

[jira] [Comment Edited] (SPARK-42068) Implicit conversion is not working with parallelization in scala with java 11 and spark3

2023-01-31 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682768#comment-17682768 ] Attila Zsolt Piros edited comment on SPARK-42068 at 1/31/23 9:40 PM: -

[jira] [Comment Edited] (SPARK-42068) Implicit conversion is not working with parallelization in scala with java 11 and spark3

2023-01-31 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682768#comment-17682768 ] Attila Zsolt Piros edited comment on SPARK-42068 at 1/31/23 9:39 PM: -

[jira] [Resolved] (SPARK-42068) Implicit conversion is not working with parallelization in scala with java 11 and spark3

2023-01-31 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros resolved SPARK-42068. Resolution: Workaround Resolving this issue as a workaround exits by using a

[jira] [Closed] (SPARK-42068) Implicit conversion is not working with parallelization in scala with java 11 and spark3

2023-01-31 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros closed SPARK-42068. -- Assignee: Attila Zsolt Piros > Implicit conversion is not working with

[jira] [Comment Edited] (SPARK-42068) Implicit conversion is not working with parallelization in scala with java 11 and spark3

2023-01-31 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682756#comment-17682756 ] Attila Zsolt Piros edited comment on SPARK-42068 at 1/31/23 9:10 PM: -

[jira] [Comment Edited] (SPARK-42068) Implicit conversion is not working with parallelization in scala with java 11 and spark3

2023-01-31 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682756#comment-17682756 ] Attila Zsolt Piros edited comment on SPARK-42068 at 1/31/23 9:09 PM: -

[jira] [Comment Edited] (SPARK-42068) Implicit conversion is not working with parallelization in scala with java 11 and spark3

2023-01-31 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682756#comment-17682756 ] Attila Zsolt Piros edited comment on SPARK-42068 at 1/31/23 9:08 PM: -

[jira] [Commented] (SPARK-42068) Implicit conversion is not working with parallelization in scala with java 11 and spark3

2023-01-31 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682756#comment-17682756 ] Attila Zsolt Piros commented on SPARK-42068: This issue is because of JDK bugfix

[jira] [Updated] (SPARK-42219) Make "SPARK-34674: Close SparkContext after the Main method has finished" configurable

2023-01-27 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-42219: --- Description: We run into an error after an upgrade from Spark 3.1 to Spark 3.2

[jira] [Commented] (SPARK-42219) Make "SPARK-34674: Close SparkContext after the Main method has finished" configurable

2023-01-27 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681454#comment-17681454 ] Attila Zsolt Piros commented on SPARK-42219: I will open a PR regarding this > Make

[jira] [Updated] (SPARK-42219) Make "SPARK-34674: Close SparkContext after the Main method has finished" configurable

2023-01-27 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-42219: --- Description: We run into an error after an upgrade from Spark 3.1 to Spark 3.2

[jira] [Assigned] (SPARK-42219) Make "SPARK-34674: Close SparkContext after the Main method has finished" configurable

2023-01-27 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros reassigned SPARK-42219: -- Assignee: Attila Zsolt Piros > Make "SPARK-34674: Close SparkContext after

[jira] [Created] (SPARK-42219) Make "SPARK-34674: Close SparkContext after the Main method has finished" configurable

2023-01-27 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-42219: -- Summary: Make "SPARK-34674: Close SparkContext after the Main method has finished" configurable Key: SPARK-42219 URL:

[jira] [Assigned] (SPARK-35084) [k8s] On Spark 3, jars listed in spark.jars and spark.jars.packages are not added to sparkContext

2023-01-14 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros reassigned SPARK-35084: -- Assignee: Keunhyun Oh > [k8s] On Spark 3, jars listed in spark.jars and

[jira] [Assigned] (SPARK-35084) [k8s] On Spark 3, jars listed in spark.jars and spark.jars.packages are not added to sparkContext

2023-01-14 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros reassigned SPARK-35084: -- Assignee: (was: Attila Zsolt Piros) > [k8s] On Spark 3, jars listed in

[jira] [Resolved] (SPARK-35084) [k8s] On Spark 3, jars listed in spark.jars and spark.jars.packages are not added to sparkContext

2023-01-14 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros resolved SPARK-35084. Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-35084) [k8s] On Spark 3, jars listed in spark.jars and spark.jars.packages are not added to sparkContext

2023-01-14 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros reassigned SPARK-35084: -- Assignee: Attila Zsolt Piros > [k8s] On Spark 3, jars listed in spark.jars

[jira] [Commented] (SPARK-29683) Job failed due to executor failures all available nodes are blacklisted

2022-10-31 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626884#comment-17626884 ] Attila Zsolt Piros commented on SPARK-29683: [~srowen] I think we can close this as this

[jira] [Updated] (SPARK-40617) Assertion failed in ExecutorMetricsPoller "task count shouldn't below 0"

2022-10-04 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-40617: --- Fix Version/s: 3.2.0 > Assertion failed in ExecutorMetricsPoller "task count

[jira] [Resolved] (SPARK-40617) Assertion failed in ExecutorMetricsPoller "task count shouldn't below 0"

2022-10-03 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros resolved SPARK-40617. Fix Version/s: 3.4.0 3.3.1 Resolution: Fixed Issue

[jira] [Updated] (SPARK-40617) Assertion failed in ExecutorMetricsPoller "task count shouldn't below 0"

2022-09-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-40617: --- Description: Spurious failures because of the assert: {noformat} 22/09/29 09:46:24

[jira] [Commented] (SPARK-34779) ExecutorMetricsPoller should keep stage entry in stageTCMP until a heartbeat occurs

2022-09-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611353#comment-17611353 ] Attila Zsolt Piros commented on SPARK-34779: [~praetp] I managed to reproduce it in a unit

[jira] [Comment Edited] (SPARK-34779) ExecutorMetricsPoller should keep stage entry in stageTCMP until a heartbeat occurs

2022-09-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611256#comment-17611256 ] Attila Zsolt Piros edited comment on SPARK-34779 at 9/29/22 10:05 PM:

[jira] [Comment Edited] (SPARK-34779) ExecutorMetricsPoller should keep stage entry in stageTCMP until a heartbeat occurs

2022-09-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611256#comment-17611256 ] Attila Zsolt Piros edited comment on SPARK-34779 at 9/29/22 10:05 PM:

[jira] [Commented] (SPARK-34779) ExecutorMetricsPoller should keep stage entry in stageTCMP until a heartbeat occurs

2022-09-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611256#comment-17611256 ] Attila Zsolt Piros commented on SPARK-34779: [~praetp] I have a working theory but an

[jira] [Updated] (SPARK-40617) Assertion failed in ExecutorMetricsPoller "task count shouldn't below 0"

2022-09-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-40617: --- Description: {noformat} 22/09/29 09:46:24 ERROR SparkUncaughtExceptionHandler:

[jira] [Commented] (SPARK-34779) ExecutorMetricsPoller should keep stage entry in stageTCMP until a heartbeat occurs

2022-09-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611217#comment-17611217 ] Attila Zsolt Piros commented on SPARK-34779: [~praetp] I have opened a new Jira and soon PR:

[jira] [Updated] (SPARK-40617) Assertion failed in ExecutorMetricsPoller "task count shouldn't below 0"

2022-09-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-40617: --- Description: Spurious failures beacuse of the assert: {noformat} 22/09/29 09:46:24

[jira] [Updated] (SPARK-40617) Assertion failed in ExecutorMetricsPoller "task count shouldn't below 0"

2022-09-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-40617: --- Description: Spurious failures regarding the assert: {noformat} 22/09/29 09:46:24

[jira] [Updated] (SPARK-40617) Assertion failed in ExecutorMetricsPoller "task count shouldn't below 0"

2022-09-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-40617: --- Description: {noformat} 22/09/29 09:46:24 ERROR SparkUncaughtExceptionHandler:

[jira] [Assigned] (SPARK-40617) Assertion failed in ExecutorMetricsPoller "task count shouldn't below 0"

2022-09-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros reassigned SPARK-40617: -- Assignee: Attila Zsolt Piros > Assertion failed in ExecutorMetricsPoller

[jira] [Created] (SPARK-40617) Assertion failed in ExecutorMetricsPoller "task count shouldn't below 0"

2022-09-29 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-40617: -- Summary: Assertion failed in ExecutorMetricsPoller "task count shouldn't below 0" Key: SPARK-40617 URL: https://issues.apache.org/jira/browse/SPARK-40617

[jira] [Commented] (SPARK-34779) ExecutorMetricsPoller should keep stage entry in stageTCMP until a heartbeat occurs

2022-09-29 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611181#comment-17611181 ] Attila Zsolt Piros commented on SPARK-34779: [~praetp]

[jira] [Created] (SPARK-40612) On Kubernetes for long running app Spark using an invalid principal to renew the delegation token

2022-09-29 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-40612: -- Summary: On Kubernetes for long running app Spark using an invalid principal to renew the delegation token Key: SPARK-40612 URL:

[jira] [Created] (SPARK-40458) Bump Kubernetes Client Version to 6.1.1

2022-09-15 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-40458: -- Summary: Bump Kubernetes Client Version to 6.1.1 Key: SPARK-40458 URL: https://issues.apache.org/jira/browse/SPARK-40458 Project: Spark Issue

[jira] [Updated] (SPARK-40039) Introducing a streaming checkpoint file manager based on Hadoop's Abortable interface

2022-08-10 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-40039: --- Summary: Introducing a streaming checkpoint file manager based on Hadoop's

[jira] [Commented] (SPARK-40039) Introducing checkpoint file manager based on Hadoop's Abortable interface

2022-08-10 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578189#comment-17578189 ] Attila Zsolt Piros commented on SPARK-40039: I am working on this. > Introducing checkpoint

[jira] [Created] (SPARK-40039) Introducing checkpoint file manager based on Hadoop's Abortable interface

2022-08-10 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-40039: -- Summary: Introducing checkpoint file manager based on Hadoop's Abortable interface Key: SPARK-40039 URL: https://issues.apache.org/jira/browse/SPARK-40039

[jira] [Resolved] (SPARK-39473) NullPointerException in SparkRackResolver

2022-06-14 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros resolved SPARK-39473. Resolution: Invalid This is because of Scala 2.11 on latter version *asScala* is

[jira] [Created] (SPARK-39473) NullPointerException in SparkRackResolver

2022-06-14 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-39473: -- Summary: NullPointerException in SparkRackResolver Key: SPARK-39473 URL: https://issues.apache.org/jira/browse/SPARK-39473 Project: Spark Issue

[jira] [Created] (SPARK-39152) StreamCorruptedException cause job failure for disk persisted RDD

2022-05-11 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-39152: -- Summary: StreamCorruptedException cause job failure for disk persisted RDD Key: SPARK-39152 URL: https://issues.apache.org/jira/browse/SPARK-39152

[jira] [Commented] (SPARK-39152) StreamCorruptedException cause job failure for disk persisted RDD

2022-05-11 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17535044#comment-17535044 ] Attila Zsolt Piros commented on SPARK-39152: I am working on this >

[jira] [Commented] (SPARK-33206) Spark Shuffle Index Cache calculates memory usage wrong

2022-02-17 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17493745#comment-17493745 ] Attila Zsolt Piros commented on SPARK-33206: I am working on this soon a PR will be opened.

[jira] [Commented] (SPARK-34372) Speculation results in broken CSV files in Amazon S3

2022-01-25 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17482041#comment-17482041 ] Attila Zsolt Piros commented on SPARK-34372: hi [~daeheh]! Please look around here:

[jira] [Assigned] (SPARK-36967) Report accurate shuffle block size if its skewed

2022-01-13 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros reassigned SPARK-36967: -- Assignee: Wan Kun (was: Apache Spark) > Report accurate shuffle block size

[jira] [Resolved] (SPARK-36967) Report accurate shuffle block size if its skewed

2022-01-13 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros resolved SPARK-36967. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Created] (SPARK-37881) Cleanup ShuffleBlockResolver from polluted methods to create a developer API

2022-01-12 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-37881: -- Summary: Cleanup ShuffleBlockResolver from polluted methods to create a developer API Key: SPARK-37881 URL: https://issues.apache.org/jira/browse/SPARK-37881

[jira] [Resolved] (SPARK-36406) No longer do file truncate operation before delete a write failed file held by DiskBlockObjectWriter

2021-12-20 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros resolved SPARK-36406. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-36406) No longer do file truncate operation before delete a write failed file held by DiskBlockObjectWriter

2021-12-20 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros reassigned SPARK-36406: -- Assignee: Yang Jie > No longer do file truncate operation before delete a

[jira] [Updated] (SPARK-37373) Collect LocalSparkContext worker logs in case of test failure

2021-11-18 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-37373: --- Affects Version/s: 3.3.0 (was: 3.2.0) > Collect

[jira] [Created] (SPARK-37373) Collect LocalSparkContext worker logs in case of test failure

2021-11-18 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-37373: -- Summary: Collect LocalSparkContext worker logs in case of test failure Key: SPARK-37373 URL: https://issues.apache.org/jira/browse/SPARK-37373 Project:

[jira] [Resolved] (SPARK-35672) Spark fails to launch executors with very large user classpath lists on YARN

2021-11-16 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros resolved SPARK-35672. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Resolved] (SPARK-36965) Extend python test runner by logging out the temp output files

2021-10-18 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros resolved SPARK-36965. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Updated] (SPARK-36965) Extend python test runner by logging out the temp output files

2021-10-09 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-36965: --- Description: I was running a python test which was extremely slow and I was

[jira] [Updated] (SPARK-36965) Extend python test runner by logging out the temp output files

2021-10-09 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-36965: --- Description: I was running a python test which was extremely slow and I was

[jira] [Updated] (SPARK-36965) Extend python test runner by logging out the temp output files

2021-10-09 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-36965: --- Description: I was running a python test which was extremely slow and I was

[jira] [Updated] (SPARK-36965) Extend python test runner by logging out the temp output files

2021-10-09 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-36965: --- Priority: Minor (was: Major) > Extend python test runner by logging out the temp

[jira] [Created] (SPARK-36965) Extend python test runner by logging out the temp output files

2021-10-09 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-36965: -- Summary: Extend python test runner by logging out the temp output files Key: SPARK-36965 URL: https://issues.apache.org/jira/browse/SPARK-36965 Project:

[jira] [Commented] (SPARK-36719) Supporting Netty Logging at the network layer

2021-09-10 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17413167#comment-17413167 ] Attila Zsolt Piros commented on SPARK-36719: I am working on this and soon I will open a PR.

[jira] [Created] (SPARK-36719) Supporting Netty Logging at the network layer

2021-09-10 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-36719: -- Summary: Supporting Netty Logging at the network layer Key: SPARK-36719 URL: https://issues.apache.org/jira/browse/SPARK-36719 Project: Spark

[jira] [Updated] (SPARK-36358) Upgrade Kubernetes Client Version to 5.6.0

2021-07-30 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-36358: --- Description: This way [Retry HTTP operation in case IOException too (exponential

[jira] [Created] (SPARK-36358) Upgrade Kubernetes Client Version to 5.6.0

2021-07-30 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-36358: -- Summary: Upgrade Kubernetes Client Version to 5.6.0 Key: SPARK-36358 URL: https://issues.apache.org/jira/browse/SPARK-36358 Project: Spark Issue

[jira] [Updated] (SPARK-35334) Spark should be more resilient to intermittent K8s flakiness

2021-07-14 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-35334: --- Issue Type: New Feature (was: Bug) > Spark should be more resilient to

[jira] [Commented] (SPARK-35334) Spark should be more resilient to intermittent K8s flakiness

2021-07-14 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380390#comment-17380390 ] Attila Zsolt Piros commented on SPARK-35334: [~dongjoon] oh sorry this is feature I just

[jira] [Resolved] (SPARK-35334) Spark should be more resilient to intermittent K8s flakiness

2021-07-13 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros resolved SPARK-35334. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Updated] (SPARK-36052) Introduce pending pod limit for Spark on K8s

2021-07-08 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-36052: --- Description: Introduce pending POD limit for Spark on K8S as K8S scheduler could be

[jira] [Updated] (SPARK-36052) Introduce pending pod limit for Spark on K8s

2021-07-08 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-36052: --- Description: Introduce a new configuration to limit the number of pending PODs for

[jira] [Created] (SPARK-36052) Introduce pending pod limit for Spark on K8s

2021-07-08 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-36052: -- Summary: Introduce pending pod limit for Spark on K8s Key: SPARK-36052 URL: https://issues.apache.org/jira/browse/SPARK-36052 Project: Spark

[jira] [Updated] (SPARK-36026) Upgrade Kubernetes Client Version to 5.5.0

2021-07-06 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-36026: --- Summary: Upgrade Kubernetes Client Version to 5.5.0 (was: Bump Kubernetes Client

[jira] [Created] (SPARK-36026) Bump Kubernetes Client Version to 5.5.0

2021-07-06 Thread Attila Zsolt Piros (Jira)
Attila Zsolt Piros created SPARK-36026: -- Summary: Bump Kubernetes Client Version to 5.5.0 Key: SPARK-36026 URL: https://issues.apache.org/jira/browse/SPARK-36026 Project: Spark Issue

[jira] [Resolved] (SPARK-35543) Small memory leak in BlockManagerMasterEndpoint

2021-06-07 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros resolved SPARK-35543. Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

[jira] [Updated] (SPARK-35610) Memory leak in Spark interpreter

2021-06-02 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-35610: --- Description: I have identified this leak by running the Livy tests (I know it is

[jira] [Updated] (SPARK-35610) Memory leak in Spark interpreter

2021-06-02 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-35610: --- Description: I have identified this leak by running the Livy tests (I know it is

[jira] [Updated] (SPARK-35610) Memory leak in Spark interpreter

2021-06-02 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-35610: --- Description: I have identified this leak by running the Livy tests (I know it is

[jira] [Updated] (SPARK-35610) Memory leak in Spark interpreter

2021-06-02 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-35610: --- Description: I have identified this leak by running the Livy tests (I know it is

[jira] [Updated] (SPARK-35610) Memory leak in Spark interpreter

2021-06-02 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-35610: --- Description: I have identified this leak by running the Livy tests (I know it is

[jira] [Updated] (SPARK-35610) Memory leak in Spark interpreter

2021-06-02 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-35610: --- Description: I have identified this leak by running the Livy tests (I know it is

[jira] [Updated] (SPARK-35610) Memory leak in Spark interpreter

2021-06-02 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-35610: --- Description: I have identified this leak by running the Livy tests (I know it is

[jira] [Updated] (SPARK-35610) Memory leak in Spark interpreter

2021-06-02 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-35610: --- Description: I have identified this leak by running the Livy tests (I know it is

[jira] [Updated] (SPARK-35610) Memory leak in Spark interpreter

2021-06-02 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-35610: --- Description: I have identified this leak by running the Livy tests (I know it is

[jira] [Updated] (SPARK-35610) Memory leak in Spark interpreter

2021-06-02 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-35610: --- Description: I have identified this leak by running the Livy tests (I know it is

[jira] [Commented] (SPARK-35610) Memory leak in Spark interpreter

2021-06-02 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355614#comment-17355614 ] Attila Zsolt Piros commented on SPARK-35610: I am working on this, soon a PR will be opened.

[jira] [Assigned] (SPARK-35610) Memory leak in Spark interpreter

2021-06-02 Thread Attila Zsolt Piros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros reassigned SPARK-35610: -- Assignee: Attila Zsolt Piros > Memory leak in Spark interpreter >

  1   2   3   4   >