[jira] [Commented] (SPARK-35610) Memory leak in Spark interpreter

[ https://issues.apache.org/jira/browse/SPARK-35610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355614#comment-17355614 ] Attila Zsolt Piros commented on SPARK-35610: I am working on this, soon a PR will be opened.

[jira] [Assigned] (SPARK-35610) Memory leak in Spark interpreter

[ https://issues.apache.org/jira/browse/SPARK-35610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros reassigned SPARK-35610: -- Assignee: Attila Zsolt Piros > Memory leak in Spark interpreter >

[jira] [Created] (SPARK-35610) Memory leak in Spark interpreter

Attila Zsolt Piros created SPARK-35610: -- Summary: Memory leak in Spark interpreter Key: SPARK-35610 URL: https://issues.apache.org/jira/browse/SPARK-35610 Project: Spark Issue Type:

[jira] [Updated] (SPARK-35543) Small memory leak in BlockManagerMasterEndpoint

[ https://issues.apache.org/jira/browse/SPARK-35543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-35543: --- Description: It is regarding _blockStatusByShuffleService_ when all the blocks are

[jira] [Assigned] (SPARK-35543) Small memory leak in BlockManagerMasterEndpoint

[ https://issues.apache.org/jira/browse/SPARK-35543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros reassigned SPARK-35543: -- Assignee: Attila Zsolt Piros > Small memory leak in

[jira] [Created] (SPARK-35543) Small memory leak in BlockManagerMasterEndpoint

Attila Zsolt Piros created SPARK-35543: -- Summary: Small memory leak in BlockManagerMasterEndpoint Key: SPARK-35543 URL: https://issues.apache.org/jira/browse/SPARK-35543 Project: Spark

[jira] [Commented] (SPARK-34738) Upgrade Minikube and kubernetes and move to docker 'virtualization' layer

[ https://issues.apache.org/jira/browse/SPARK-34738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17346572#comment-17346572 ] Attila Zsolt Piros commented on SPARK-34738: Thanks [~shaneknapp]! I opened a separate

[jira] [Comment Edited] (SPARK-34738) Upgrade Minikube and kubernetes and move to docker 'virtualization' layer

[ https://issues.apache.org/jira/browse/SPARK-34738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17346572#comment-17346572 ] Attila Zsolt Piros edited comment on SPARK-34738 at 5/18/21, 3:56 AM:

[jira] [Created] (SPARK-35430) Investigate the failure of "PVs with local storage" integration test on Docker driver

Attila Zsolt Piros created SPARK-35430: -- Summary: Investigate the failure of "PVs with local storage" integration test on Docker driver Key: SPARK-35430 URL: https://issues.apache.org/jira/browse/SPARK-35430

[jira] [Commented] (SPARK-34738) Upgrade Minikube and kubernetes and move to docker 'virtualization' layer

[ https://issues.apache.org/jira/browse/SPARK-34738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17342040#comment-17342040 ] Attila Zsolt Piros commented on SPARK-34738: [~shaneknapp] I am sorry to hear about the

[jira] [Commented] (SPARK-34738) Upgrade Minikube and kubernetes and move to docker 'virtualization' layer

[ https://issues.apache.org/jira/browse/SPARK-34738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17342018#comment-17342018 ] Attila Zsolt Piros commented on SPARK-34738: [~shaneknapp] I have merged my commit this

[jira] [Resolved] (SPARK-34736) Kubernetes and Minikube version upgrade for integration tests

[ https://issues.apache.org/jira/browse/SPARK-34736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros resolved SPARK-34736. Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-34736) Kubernetes and Minikube version upgrade for integration tests

[ https://issues.apache.org/jira/browse/SPARK-34736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros reassigned SPARK-34736: -- Assignee: Attila Zsolt Piros > Kubernetes and Minikube version upgrade for

[jira] [Updated] (SPARK-35334) Spark should be more resilient to intermittent K8s flakiness

[ https://issues.apache.org/jira/browse/SPARK-35334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-35334: --- Description: Internal K8s errors such as an etcdserver leader election is

[jira] [Created] (SPARK-35334) Spark should be more resilient to intermittent K8s flakiness

Attila Zsolt Piros created SPARK-35334: -- Summary: Spark should be more resilient to intermittent K8s flakiness Key: SPARK-35334 URL: https://issues.apache.org/jira/browse/SPARK-35334 Project:

[jira] [Resolved] (SPARK-35009) Avoid creating multiple Monitor threads for reused python workers for the same TaskContext

[ https://issues.apache.org/jira/browse/SPARK-35009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros resolved SPARK-35009. Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-35234) Reserve format of the stage failureMessage

[ https://issues.apache.org/jira/browse/SPARK-35234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros reassigned SPARK-35234: -- Assignee: wuyi > Reserve format of the stage failureMessage >

[jira] [Resolved] (SPARK-35234) Reserve format of the stage failureMessage

[ https://issues.apache.org/jira/browse/SPARK-35234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros resolved SPARK-35234. Fix Version/s: 3.2.0 Resolution: Fixed > Reserve format of the stage

[jira] [Comment Edited] (SPARK-34738) Upgrade Minikube and kubernetes and move to docker 'virtualization' layer

[ https://issues.apache.org/jira/browse/SPARK-34738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324167#comment-17324167 ] Attila Zsolt Piros edited comment on SPARK-34738 at 4/17/21, 4:27 AM:

[jira] [Commented] (SPARK-34738) Upgrade Minikube and kubernetes and move to docker 'virtualization' layer

[ https://issues.apache.org/jira/browse/SPARK-34738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324167#comment-17324167 ] Attila Zsolt Piros commented on SPARK-34738: My new commit:

[jira] [Comment Edited] (SPARK-34738) Upgrade Minikube and kubernetes and move to docker 'virtualization' layer

[ https://issues.apache.org/jira/browse/SPARK-34738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324165#comment-17324165 ] Attila Zsolt Piros edited comment on SPARK-34738 at 4/17/21, 3:56 AM:

[jira] [Commented] (SPARK-34738) Upgrade Minikube and kubernetes and move to docker 'virtualization' layer

[ https://issues.apache.org/jira/browse/SPARK-34738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324165#comment-17324165 ] Attila Zsolt Piros commented on SPARK-34738: @shane I can modify my PR to skip the PV tests

[jira] [Commented] (SPARK-35009) Avoid creating multiple Monitor threads for reused python workers for the same TaskContext

[ https://issues.apache.org/jira/browse/SPARK-35009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17318097#comment-17318097 ] Attila Zsolt Piros commented on SPARK-35009: I am working on this > Avoid creating multiple

[jira] [Created] (SPARK-35009) Avoid creating multiple Monitor threads for reused python workers for the same TaskContext

Attila Zsolt Piros created SPARK-35009: -- Summary: Avoid creating multiple Monitor threads for reused python workers for the same TaskContext Key: SPARK-35009 URL:

[jira] [Commented] (SPARK-34738) Upgrade Minikube and kubernetes cluster version on Jenkins

[ https://issues.apache.org/jira/browse/SPARK-34738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317779#comment-17317779 ] Attila Zsolt Piros commented on SPARK-34738: For sure `withValues` is needed. This is

[jira] [Commented] (SPARK-34738) Upgrade Minikube and kubernetes cluster version on Jenkins

[ https://issues.apache.org/jira/browse/SPARK-34738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317426#comment-17317426 ] Attila Zsolt Piros commented on SPARK-34738: [~shaneknapp] the operator is IN: {noformat}

[jira] [Commented] (SPARK-34738) Upgrade Minikube and kubernetes cluster version on Jenkins

[ https://issues.apache.org/jira/browse/SPARK-34738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317391#comment-17317391 ] Attila Zsolt Piros commented on SPARK-34738: >From the log we can see the mount is not ok:

[jira] [Comment Edited] (SPARK-34738) Upgrade Minikube and kubernetes cluster version on Jenkins

[ https://issues.apache.org/jira/browse/SPARK-34738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317381#comment-17317381 ] Attila Zsolt Piros edited comment on SPARK-34738 at 4/8/21, 6:02 PM: -

[jira] [Commented] (SPARK-34738) Upgrade Minikube and kubernetes cluster version on Jenkins

[ https://issues.apache.org/jira/browse/SPARK-34738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317381#comment-17317381 ] Attila Zsolt Piros commented on SPARK-34738: No worries. I have guess. Check

[jira] [Commented] (SPARK-34738) Upgrade Minikube and kubernetes cluster version on Jenkins

[ https://issues.apache.org/jira/browse/SPARK-34738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317034#comment-17317034 ] Attila Zsolt Piros commented on SPARK-34738: hi [~shaneknapp]! Can you please share with

[jira] [Resolved] (SPARK-34779) ExecutorMetricsPoller should keep stage entry in stageTCMP until a heartbeat occurs

[ https://issues.apache.org/jira/browse/SPARK-34779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros resolved SPARK-34779. Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-34779) ExecutorMetricsPoller should keep stage entry in stageTCMP until a heartbeat occurs

[ https://issues.apache.org/jira/browse/SPARK-34779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros reassigned SPARK-34779: -- Assignee: Baohe Zhang > ExecutorMetricsPoller should keep stage entry in

[jira] [Commented] (SPARK-34738) Upgrade Minikube and kubernetes cluster version on Jenkins

[ https://issues.apache.org/jira/browse/SPARK-34738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17313144#comment-17313144 ] Attila Zsolt Piros commented on SPARK-34738: So I wanted to check whether I can help next

[jira] [Commented] (SPARK-34738) Upgrade Minikube and kubernetes cluster version on Jenkins

[ https://issues.apache.org/jira/browse/SPARK-34738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17312851#comment-17312851 ] Attila Zsolt Piros commented on SPARK-34738: Shane, as Docker is available on every platform

[jira] [Updated] (SPARK-34869) Extend k8s "EXTRA LOGS FOR THE FAILED TEST" section with describe pods output

[ https://issues.apache.org/jira/browse/SPARK-34869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-34869: --- Summary: Extend k8s "EXTRA LOGS FOR THE FAILED TEST" section with describe pods

[jira] [Updated] (SPARK-34869) Extend k8s "EXTRA LOGS FOR THE FAILED TEST" section with describe pods output

[ https://issues.apache.org/jira/browse/SPARK-34869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-34869: --- Description: As came up in

[jira] [Assigned] (SPARK-34869) Extend k8s "EXTRA LOGS FOR THE FAILED TEST" section with pod describe output

[ https://issues.apache.org/jira/browse/SPARK-34869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros reassigned SPARK-34869: -- Assignee: Attila Zsolt Piros > Extend k8s "EXTRA LOGS FOR THE FAILED TEST"

[jira] [Commented] (SPARK-34869) Extend k8s "EXTRA LOGS FOR THE FAILED TEST" section with pod describe output

[ https://issues.apache.org/jira/browse/SPARK-34869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308679#comment-17308679 ] Attila Zsolt Piros commented on SPARK-34869: I am working on this. > Extend k8s "EXTRA LOGS

[jira] [Updated] (SPARK-34869) Extend k8s "EXTRA LOGS FOR THE FAILED TEST" section with pod describe output

[ https://issues.apache.org/jira/browse/SPARK-34869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-34869: --- Description: As came up in

[jira] [Updated] (SPARK-34869) Extend k8s "EXTRA LOGS FOR THE FAILED TEST" section with pod describe output

[ https://issues.apache.org/jira/browse/SPARK-34869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-34869: --- Description: As came up in

[jira] [Created] (SPARK-34869) Extend k8s "EXTRA LOGS FOR THE FAILED TEST" section with pod describe output

Attila Zsolt Piros created SPARK-34869: -- Summary: Extend k8s "EXTRA LOGS FOR THE FAILED TEST" section with pod describe output Key: SPARK-34869 URL: https://issues.apache.org/jira/browse/SPARK-34869

[jira] [Commented] (SPARK-34684) Hadoop config could not be successfully serilized from driver pods to executor pods

[ https://issues.apache.org/jira/browse/SPARK-34684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307710#comment-17307710 ] Attila Zsolt Piros commented on SPARK-34684: I did not say there is no error but by

[jira] [Commented] (SPARK-34684) Hadoop config could not be successfully serilized from driver pods to executor pods

[ https://issues.apache.org/jira/browse/SPARK-34684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307392#comment-17307392 ] Attila Zsolt Piros commented on SPARK-34684: > Have you tried to create a POD from a simple

[jira] [Commented] (SPARK-34684) Hadoop config could not be successfully serilized from driver pods to executor pods

[ https://issues.apache.org/jira/browse/SPARK-34684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17306895#comment-17306895 ] Attila Zsolt Piros commented on SPARK-34684: > spark pi example job keeps failing where

[jira] [Updated] (SPARK-34789) Introduce Jetty based construct for integration tests where HTTP(S) is used

[ https://issues.apache.org/jira/browse/SPARK-34789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-34789: --- Description: This came up during

[jira] [Commented] (SPARK-34789) Introduce Jetty based construct for integration tests where HTTP(S) is used

[ https://issues.apache.org/jira/browse/SPARK-34789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304245#comment-17304245 ] Attila Zsolt Piros commented on SPARK-34789: I am working on this > Introduce Jetty based

[jira] [Created] (SPARK-34789) Introduce Jetty based construct for integration tests where HTTP(S) is used

Attila Zsolt Piros created SPARK-34789: -- Summary: Introduce Jetty based construct for integration tests where HTTP(S) is used Key: SPARK-34789 URL: https://issues.apache.org/jira/browse/SPARK-34789

[jira] [Commented] (SPARK-34738) Upgrade Minikube and kubernetes cluster version on Jenkins

[ https://issues.apache.org/jira/browse/SPARK-34738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17303287#comment-17303287 ] Attila Zsolt Piros commented on SPARK-34738: Thanks Shane! This sounds great! > Upgrade

[jira] [Created] (SPARK-34738) Upgrade Minikube and kubernetes cluster version on Jenkins

Attila Zsolt Piros created SPARK-34738: -- Summary: Upgrade Minikube and kubernetes cluster version on Jenkins Key: SPARK-34738 URL: https://issues.apache.org/jira/browse/SPARK-34738 Project: Spark

[jira] [Commented] (SPARK-34736) Kubernetes and Minikube version upgrade for integration tests

[ https://issues.apache.org/jira/browse/SPARK-34736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17301046#comment-17301046 ] Attila Zsolt Piros commented on SPARK-34736: Working on this > Kubernetes and Minikube

[jira] [Created] (SPARK-34736) Kubernetes and Minikube version upgrade for integration tests

Attila Zsolt Piros created SPARK-34736: -- Summary: Kubernetes and Minikube version upgrade for integration tests Key: SPARK-34736 URL: https://issues.apache.org/jira/browse/SPARK-34736 Project:

[jira] [Created] (SPARK-34732) The logForFailedTest throws an exception when driver is not started

Attila Zsolt Piros created SPARK-34732: -- Summary: The logForFailedTest throws an exception when driver is not started Key: SPARK-34732 URL: https://issues.apache.org/jira/browse/SPARK-34732

[jira] [Commented] (SPARK-34509) Make dynamic allocation upscaling more progressive on K8S

[ https://issues.apache.org/jira/browse/SPARK-34509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289158#comment-17289158 ] Attila Zsolt Piros commented on SPARK-34509: I am working on this. > Make dynamic

[jira] [Created] (SPARK-34509) Make dynamic allocation upscaling more progressive on K8S

Attila Zsolt Piros created SPARK-34509: -- Summary: Make dynamic allocation upscaling more progressive on K8S Key: SPARK-34509 URL: https://issues.apache.org/jira/browse/SPARK-34509 Project: Spark

[jira] [Created] (SPARK-34433) Lock jekyll version by Gemfile and Bundler

Attila Zsolt Piros created SPARK-34433: -- Summary: Lock jekyll version by Gemfile and Bundler Key: SPARK-34433 URL: https://issues.apache.org/jira/browse/SPARK-34433 Project: Spark Issue

[jira] [Commented] (SPARK-34433) Lock jekyll version by Gemfile and Bundler

[ https://issues.apache.org/jira/browse/SPARK-34433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284287#comment-17284287 ] Attila Zsolt Piros commented on SPARK-34433: I am working on this. > Lock jekyll version by

[jira] [Created] (SPARK-34426) Add driver and executors POD logs to integration-tests log when the test fails

Attila Zsolt Piros created SPARK-34426: -- Summary: Add driver and executors POD logs to integration-tests log when the test fails Key: SPARK-34426 URL: https://issues.apache.org/jira/browse/SPARK-34426

[jira] [Commented] (SPARK-34426) Add driver and executors POD logs to integration-tests log when the test fails

[ https://issues.apache.org/jira/browse/SPARK-34426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17283974#comment-17283974 ] Attila Zsolt Piros commented on SPARK-34426: I am working on this. > Add driver and

[jira] [Updated] (SPARK-34426) Add driver and executors POD logs to integration tests log when the test fails

[ https://issues.apache.org/jira/browse/SPARK-34426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-34426: --- Summary: Add driver and executors POD logs to integration tests log when the test

[jira] [Comment Edited] (SPARK-34389) Spark job on Kubernetes scheduled For Zero or less than minimum number of executors and Wait indefinitely under resource starvation

[ https://issues.apache.org/jira/browse/SPARK-34389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282614#comment-17282614 ] Attila Zsolt Piros edited comment on SPARK-34389 at 2/12/21, 8:36 AM:

[jira] [Commented] (SPARK-34389) Spark job on Kubernetes scheduled For Zero or less than minimum number of executors and Wait indefinitely under resource starvation

[ https://issues.apache.org/jira/browse/SPARK-34389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282994#comment-17282994 ] Attila Zsolt Piros commented on SPARK-34389: [~ranju] as I see your question is more about

[jira] [Commented] (SPARK-34389) Spark job on Kubernetes scheduled For Zero or less than minimum number of executors and Wait indefinitely under resource starvation

[ https://issues.apache.org/jira/browse/SPARK-34389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282871#comment-17282871 ] Attila Zsolt Piros commented on SPARK-34389: @Ranju from the logs you attached it is clear

[jira] [Commented] (SPARK-34389) Spark job on Kubernetes scheduled For Zero or less than minimum number of executors and Wait indefinitely under resource starvation

[ https://issues.apache.org/jira/browse/SPARK-34389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282614#comment-17282614 ] Attila Zsolt Piros commented on SPARK-34389: [~ranju] the description of this config is a

[jira] [Commented] (SPARK-33763) Add metrics for better tracking of dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-33763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282201#comment-17282201 ] Attila Zsolt Piros commented on SPARK-33763: I am not positive about the "stage re-submitted

[jira] [Comment Edited] (SPARK-33763) Add metrics for better tracking of dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-33763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282201#comment-17282201 ] Attila Zsolt Piros edited comment on SPARK-33763 at 2/10/21, 3:30 AM:

[jira] [Commented] (SPARK-34372) Speculation results in broken CSV files in Amazon S3

[ https://issues.apache.org/jira/browse/SPARK-34372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17281815#comment-17281815 ] Attila Zsolt Piros commented on SPARK-34372: A Direct output committer with speculation

[jira] [Commented] (SPARK-34389) Spark job on Kubernetes scheduled For Zero or less than minimum number of executors and Wait indefinitely under resource starvation

[ https://issues.apache.org/jira/browse/SPARK-34389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17281645#comment-17281645 ] Attila Zsolt Piros commented on SPARK-34389: > is it possible to get the available resources

[jira] [Commented] (SPARK-34389) Spark job on Kubernetes scheduled For Zero or less than minimum number of executors and Wait indefinitely under resource starvation

[ https://issues.apache.org/jira/browse/SPARK-34389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17281134#comment-17281134 ] Attila Zsolt Piros commented on SPARK-34389: [~ranju] I understand your concern but Spark

[jira] [Updated] (SPARK-34370) Supporting Avro schema evolution for partitioned Hive tables using "avro.schema.url"

[ https://issues.apache.org/jira/browse/SPARK-34370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-34370: --- Description: This came up in

[jira] [Updated] (SPARK-34370) Supporting Avro schema evolution for partitioned Hive tables using "avro.schema.url"

[ https://issues.apache.org/jira/browse/SPARK-34370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-34370: --- Description: This came up in

[jira] [Updated] (SPARK-34370) Supporting Avro schema evolution for partitioned Hive tables using "avro.schema.url"

[ https://issues.apache.org/jira/browse/SPARK-34370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-34370: --- Description: This came up in

[jira] [Updated] (SPARK-34370) Supporting Avro schema evolution for partitioned Hive tables using "avro.schema.url"

[ https://issues.apache.org/jira/browse/SPARK-34370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-34370: --- Description: This came up in

[jira] [Commented] (SPARK-34370) Supporting Avro schema evolution for partitioned Hive tables using "avro.schema.url"

[ https://issues.apache.org/jira/browse/SPARK-34370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17279156#comment-17279156 ] Attila Zsolt Piros commented on SPARK-34370: I am working on this. > Supporting Avro schema

[jira] [Updated] (SPARK-34370) Supporting Avro schema evolution for partitioned Hive tables using "avro.schema.url"

[ https://issues.apache.org/jira/browse/SPARK-34370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-34370: --- Description: This came up in

[jira] [Created] (SPARK-34370) Supporting Avro schema evolution for partitioned Hive tables using "avro.schema.url"

Attila Zsolt Piros created SPARK-34370: -- Summary: Supporting Avro schema evolution for partitioned Hive tables using "avro.schema.url" Key: SPARK-34370 URL: https://issues.apache.org/jira/browse/SPARK-34370

[jira] [Commented] (SPARK-34361) Dynamic allocation on K8s kills executors with running tasks

[ https://issues.apache.org/jira/browse/SPARK-34361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17278932#comment-17278932 ] Attila Zsolt Piros commented on SPARK-34361: I am working on this. > Dynamic allocation on

[jira] [Created] (SPARK-34361) Dynamic allocation on K8s kills executors with running tasks

Attila Zsolt Piros created SPARK-34361: -- Summary: Dynamic allocation on K8s kills executors with running tasks Key: SPARK-34361 URL: https://issues.apache.org/jira/browse/SPARK-34361 Project:

[jira] [Commented] (SPARK-27764) Feature Parity between PostgreSQL and Spark

[ https://issues.apache.org/jira/browse/SPARK-27764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17278065#comment-17278065 ] Attila Zsolt Piros commented on SPARK-27764: Sorry to chime in but we have (after

[jira] [Commented] (SPARK-29391) Default year-month units

[ https://issues.apache.org/jira/browse/SPARK-29391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17278015#comment-17278015 ] Attila Zsolt Piros commented on SPARK-29391: I found this:

[jira] [Commented] (SPARK-29391) Default year-month units

[ https://issues.apache.org/jira/browse/SPARK-29391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17278010#comment-17278010 ] Attila Zsolt Piros commented on SPARK-29391: This is fine but learning from this case we

[jira] [Comment Edited] (SPARK-29391) Default year-month units

[ https://issues.apache.org/jira/browse/SPARK-29391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17278006#comment-17278006 ] Attila Zsolt Piros edited comment on SPARK-29391 at 2/3/21, 1:35 PM: -

[jira] [Resolved] (SPARK-29391) Default year-month units

[ https://issues.apache.org/jira/browse/SPARK-29391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros resolved SPARK-29391. Resolution: Won't Fix As PostgreSQL feature parity is not a requirement anymore

[jira] [Commented] (SPARK-33763) Add metrics for better tracking of dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-33763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17277675#comment-17277675 ] Attila Zsolt Piros commented on SPARK-33763: I am ready with the executor removals (1 and 4

[jira] [Created] (SPARK-34334) ExecutorPodsAllocator fails to identify some excess requests during downscaling

Attila Zsolt Piros created SPARK-34334: -- Summary: ExecutorPodsAllocator fails to identify some excess requests during downscaling Key: SPARK-34334 URL: https://issues.apache.org/jira/browse/SPARK-34334

[jira] [Commented] (SPARK-34334) ExecutorPodsAllocator fails to identify some excess requests during downscaling

[ https://issues.apache.org/jira/browse/SPARK-34334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17277436#comment-17277436 ] Attila Zsolt Piros commented on SPARK-34334: I am working on the this. >

[jira] [Comment Edited] (SPARK-34194) Queries that only touch partition columns shouldn't scan through all files

[ https://issues.apache.org/jira/browse/SPARK-34194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276874#comment-17276874 ] Attila Zsolt Piros edited comment on SPARK-34194 at 2/2/21, 6:59 AM: -

[jira] [Commented] (SPARK-34194) Queries that only touch partition columns shouldn't scan through all files

[ https://issues.apache.org/jira/browse/SPARK-34194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276874#comment-17276874 ] Attila Zsolt Piros commented on SPARK-34194: Yes. > Queries that only touch partition

[jira] [Commented] (SPARK-34194) Queries that only touch partition columns shouldn't scan through all files

[ https://issues.apache.org/jira/browse/SPARK-34194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17275965#comment-17275965 ] Attila Zsolt Piros commented on SPARK-34194: [~nchammas] I suggest to check out this config

[jira] [Commented] (SPARK-29391) Default year-month units

[ https://issues.apache.org/jira/browse/SPARK-29391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17275952#comment-17275952 ] Attila Zsolt Piros commented on SPARK-29391: If this helps in decision: the change itself

[jira] [Commented] (SPARK-29391) Default year-month units

[ https://issues.apache.org/jira/browse/SPARK-29391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17275926#comment-17275926 ] Attila Zsolt Piros commented on SPARK-29391: I am working on this and within a few days a PR

[jira] [Commented] (SPARK-29220) Flaky test: org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]

[ https://issues.apache.org/jira/browse/SPARK-29220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17275786#comment-17275786 ] Attila Zsolt Piros commented on SPARK-29220: I think we can close this as with

[jira] [Commented] (SPARK-34280) Avoid migrating un-needed shuffle files

[ https://issues.apache.org/jira/browse/SPARK-34280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17275783#comment-17275783 ] Attila Zsolt Piros commented on SPARK-34280: It would be interesting to know whether the

[jira] [Comment Edited] (SPARK-34285) Implement Parquet StringEndsWith、StringContains Filter

[ https://issues.apache.org/jira/browse/SPARK-34285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17274458#comment-17274458 ] Attila Zsolt Piros edited comment on SPARK-34285 at 1/31/21, 4:50 AM:

[jira] [Commented] (SPARK-34285) Implement Parquet StringEndsWith、StringContains Filter

[ https://issues.apache.org/jira/browse/SPARK-34285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17274458#comment-17274458 ] Attila Zsolt Piros commented on SPARK-34285: [~Xudingyu] predicate pushdown is extremely

[jira] [Commented] (SPARK-33763) Add metrics for better tracking of dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-33763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273923#comment-17273923 ] Attila Zsolt Piros commented on SPARK-33763: I started working on this. > Add metrics for

[jira] [Commented] (SPARK-34126) SQL running error, spark does not exit, resulting in data quality problems

[ https://issues.apache.org/jira/browse/SPARK-34126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273761#comment-17273761 ] Attila Zsolt Piros commented on SPARK-34126: [~shikui] are you using the " -f " argument of

[jira] [Commented] (SPARK-34179) examples provided in https://spark.apache.org/docs/latest/api/sql/index.html link not working

[ https://issues.apache.org/jira/browse/SPARK-34179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273348#comment-17273348 ] Attila Zsolt Piros commented on SPARK-34179: [~chetdb] The

[jira] [Commented] (SPARK-34154) Flaky Test: LocalityPlacementStrategySuite.handle large number of containers and tasks (SPARK-18750)

[ https://issues.apache.org/jira/browse/SPARK-34154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272990#comment-17272990 ] Attila Zsolt Piros commented on SPARK-34154: In my PR the bug does not surfaced even after

[jira] [Commented] (SPARK-34154) Flaky Test: LocalityPlacementStrategySuite.handle large number of containers and tasks (SPARK-18750)

[ https://issues.apache.org/jira/browse/SPARK-34154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272867#comment-17272867 ] Attila Zsolt Piros commented on SPARK-34154: I tried to reproduce this issue by running it

[jira] [Comment Edited] (SPARK-34167) Reading parquet with Decimal(8,2) written as a Decimal64 blows up

[ https://issues.apache.org/jira/browse/SPARK-34167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269182#comment-17269182 ] Attila Zsolt Piros edited comment on SPARK-34167 at 1/21/21, 9:49 AM:

<    1   2   3   4   >