[jira] [Resolved] (SPARK-37690) Recursive view `df` detected (cycle: `df` -> `df`)

[ https://issues.apache.org/jira/browse/SPARK-37690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-37690. -- Fix Version/s: 3.2.2 3.3.0 Resolution: Duplicate > Recursive view

[jira] [Updated] (SPARK-32003) Shuffle files for lost executor are not unregistered if fetch failure occurs after executor is lost

[ https://issues.apache.org/jira/browse/SPARK-32003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-32003: - Fix Version/s: 2.4.7 > Shuffle files for lost executor are not unregistered if fetch failure

[jira] [Commented] (SPARK-32003) Shuffle files for lost executor are not unregistered if fetch failure occurs after executor is lost

[ https://issues.apache.org/jira/browse/SPARK-32003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171083#comment-17171083 ] Imran Rashid commented on SPARK-32003: -- Fixed in 2.4.7 by

[jira] [Commented] (SPARK-32003) Shuffle files for lost executor are not unregistered if fetch failure occurs after executor is lost

[ https://issues.apache.org/jira/browse/SPARK-32003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170945#comment-17170945 ] Imran Rashid commented on SPARK-32003: -- Fixed in 3.0.1 by

[jira] [Updated] (SPARK-32003) Shuffle files for lost executor are not unregistered if fetch failure occurs after executor is lost

[ https://issues.apache.org/jira/browse/SPARK-32003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-32003: - Fix Version/s: 3.0.1 > Shuffle files for lost executor are not unregistered if fetch failure

[jira] [Resolved] (SPARK-32003) Shuffle files for lost executor are not unregistered if fetch failure occurs after executor is lost

[ https://issues.apache.org/jira/browse/SPARK-32003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-32003. -- Fix Version/s: 3.1.0 Resolution: Fixed Fixed in master for 3.1 by

[jira] [Assigned] (SPARK-32003) Shuffle files for lost executor are not unregistered if fetch failure occurs after executor is lost

[ https://issues.apache.org/jira/browse/SPARK-32003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-32003: Assignee: Wing Yew Poon > Shuffle files for lost executor are not unregistered if fetch

[jira] [Updated] (SPARK-25561) HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql

[ https://issues.apache.org/jira/browse/SPARK-25561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-25561: - Description: In HiveShim.scala, the current

[jira] [Updated] (SPARK-25561) HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql

[ https://issues.apache.org/jira/browse/SPARK-25561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-25561: - Description: In HiveShim.scala, the current behavior is that if hive.metastore.try.direct.sql

[jira] [Resolved] (SPARK-27324) document configurations related to executor metrics

[ https://issues.apache.org/jira/browse/SPARK-27324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-27324. -- Fix Version/s: 3.0.0 Resolution: Fixed > document configurations related to executor

[jira] [Assigned] (SPARK-27324) document configurations related to executor metrics

[ https://issues.apache.org/jira/browse/SPARK-27324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-27324: Assignee: Wing Yew Poon > document configurations related to executor metrics >

[jira] [Commented] (SPARK-27324) document configurations related to executor metrics

[ https://issues.apache.org/jira/browse/SPARK-27324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027807#comment-17027807 ] Imran Rashid commented on SPARK-27324: -- Fixed by https://github.com/apache/spark/pull/27329 >

[jira] [Commented] (SPARK-6235) Address various 2G limits

[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998577#comment-16998577 ] Imran Rashid commented on SPARK-6235: - [~sammysheep] Spark's ML library uses the same jira project,

[jira] [Commented] (SPARK-6235) Address various 2G limits

[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997631#comment-16997631 ] Imran Rashid commented on SPARK-6235: - [~sammysheep] are you discussing the use case for task results

[jira] [Resolved] (SPARK-27189) Add Executor metrics and memory usage instrumentation to the metrics system

[ https://issues.apache.org/jira/browse/SPARK-27189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-27189. -- Fix Version/s: 3.0.0 Resolution: Fixed Fixed by

[jira] [Assigned] (SPARK-27189) Add Executor metrics and memory usage instrumentation to the metrics system

[ https://issues.apache.org/jira/browse/SPARK-27189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-27189: Assignee: Luca Canali > Add Executor metrics and memory usage instrumentation to the

[jira] [Updated] (SPARK-23485) Kubernetes should support node blacklist

[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-23485: - Labels: (was: bulk-closed) > Kubernetes should support node blacklist >

[jira] [Reopened] (SPARK-23485) Kubernetes should support node blacklist

[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reopened SPARK-23485: -- I think this issue is still valid for current versions, it was just opened against 2.3.0, so

[jira] [Updated] (SPARK-23485) Kubernetes should support node blacklist

[ https://issues.apache.org/jira/browse/SPARK-23485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-23485: - Affects Version/s: (was: 2.3.0) 3.0.0 2.4.0 >

[jira] [Commented] (SPARK-29762) GPU Scheduling - default task resource amount to 1

[ https://issues.apache.org/jira/browse/SPARK-29762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972703#comment-16972703 ] Imran Rashid commented on SPARK-29762: -- The scenario I was thinking is: application is submitted

[jira] [Commented] (SPARK-29762) GPU Scheduling - default task resource amount to 1

[ https://issues.apache.org/jira/browse/SPARK-29762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972631#comment-16972631 ] Imran Rashid commented on SPARK-29762: -- I don't really understand the complication. I know there

[jira] [Commented] (SPARK-22579) BlockManager.getRemoteValues and BlockManager.getRemoteBytes should be implemented using streaming

[ https://issues.apache.org/jira/browse/SPARK-22579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964281#comment-16964281 ] Imran Rashid commented on SPARK-22579: -- Sorry I had not noticed this issue before. I agree that

[jira] [Resolved] (SPARK-29398) Allow RPC endpoints to use dedicated thread pools

[ https://issues.apache.org/jira/browse/SPARK-29398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-29398. -- Fix Version/s: 3.0.0 Assignee: Marcelo Masiero Vanzin Resolution: Fixed

[jira] [Assigned] (SPARK-28211) Shuffle Storage API: Driver Lifecycle

[ https://issues.apache.org/jira/browse/SPARK-28211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-28211: Assignee: Yifei Huang > Shuffle Storage API: Driver Lifecycle >

[jira] [Resolved] (SPARK-28211) Shuffle Storage API: Driver Lifecycle

[ https://issues.apache.org/jira/browse/SPARK-28211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-28211. -- Fix Version/s: 3.0.0 Resolution: Fixed Resolved by

[jira] [Commented] (SPARK-28867) InMemoryStore checkpoint to speed up replay log file in HistoryServer

[ https://issues.apache.org/jira/browse/SPARK-28867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949767#comment-16949767 ] Imran Rashid commented on SPARK-28867: -- This is closely related to SPARK-20656. Its not *quite* a

[jira] [Commented] (SPARK-29396) Extend Spark plugin interface to driver

[ https://issues.apache.org/jira/browse/SPARK-29396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949669#comment-16949669 ] Imran Rashid commented on SPARK-29396: -- The memory monitor plugin I wrote (why I pushed the

[jira] [Commented] (SPARK-29396) Extend Spark plugin interface to driver

[ https://issues.apache.org/jira/browse/SPARK-29396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948720#comment-16948720 ] Imran Rashid commented on SPARK-29396: -- My hack to get around this in the past was to create a

[jira] [Resolved] (SPARK-27468) "Storage Level" in "RDD Storage Page" is not correct

[ https://issues.apache.org/jira/browse/SPARK-27468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-27468. -- Fix Version/s: 3.0.0 Resolution: Fixed Resolved by

[jira] [Assigned] (SPARK-27468) "Storage Level" in "RDD Storage Page" is not correct

[ https://issues.apache.org/jira/browse/SPARK-27468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-27468: Assignee: Marcelo Masiero Vanzin > "Storage Level" in "RDD Storage Page" is not correct

[jira] [Commented] (SPARK-29189) Add an option to ignore block locations when listing file

[ https://issues.apache.org/jira/browse/SPARK-29189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946176#comment-16946176 ] Imran Rashid commented on SPARK-29189: -- Fixed by pr https://github.com/apache/spark/pull/25869 >

[jira] [Assigned] (SPARK-29189) Add an option to ignore block locations when listing file

[ https://issues.apache.org/jira/browse/SPARK-29189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-29189: Assignee: Wang, Gang > Add an option to ignore block locations when listing file >

[jira] [Resolved] (SPARK-29189) Add an option to ignore block locations when listing file

[ https://issues.apache.org/jira/browse/SPARK-29189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-29189. -- Fix Version/s: 3.0.0 Resolution: Fixed > Add an option to ignore block locations when

[jira] [Updated] (SPARK-28917) Jobs can hang because of race of RDD.dependencies

[ https://issues.apache.org/jira/browse/SPARK-28917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-28917: - Description: {{RDD.dependencies}} stores the precomputed cache value, but it is not

[jira] [Updated] (SPARK-28917) Jobs can hang because of race of RDD.dependencies

[ https://issues.apache.org/jira/browse/SPARK-28917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-28917: - Description: {{RDD.dependencies}} stores the precomputed cache value, but it is not

[jira] [Commented] (SPARK-28592) Mark new Shuffle apis as @Experimental (instead of @Private)

[ https://issues.apache.org/jira/browse/SPARK-28592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938028#comment-16938028 ] Imran Rashid commented on SPARK-28592: -- [~holden] no, I think this is intentionally *not* a

[jira] [Commented] (SPARK-28917) Jobs can hang because of race of RDD.dependencies

[ https://issues.apache.org/jira/browse/SPARK-28917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930616#comment-16930616 ] Imran Rashid commented on SPARK-28917: -- I finally got some more info about this case -- they are

[jira] [Resolved] (SPARK-29072) Properly track shuffle write time with refactor

[ https://issues.apache.org/jira/browse/SPARK-29072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-29072. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25780

[jira] [Assigned] (SPARK-29072) Properly track shuffle write time with refactor

[ https://issues.apache.org/jira/browse/SPARK-29072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-29072: Assignee: Matt Cheah > Properly track shuffle write time with refactor >

[jira] [Resolved] (SPARK-29017) JobGroup and LocalProperty not respected by PySpark

[ https://issues.apache.org/jira/browse/SPARK-29017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-29017. -- Resolution: Duplicate > JobGroup and LocalProperty not respected by PySpark >

[jira] [Commented] (SPARK-29017) JobGroup and LocalProperty not respected by PySpark

[ https://issues.apache.org/jira/browse/SPARK-29017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924975#comment-16924975 ] Imran Rashid commented on SPARK-29017: -- [~hyukjin.kwon][~holdensmagicalunicorn] [~bryanc] any

[jira] [Updated] (SPARK-29017) JobGroup and LocalProperty not respected by PySpark

[ https://issues.apache.org/jira/browse/SPARK-29017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-29017: - Description: Pyspark has {{setJobGroup}} and {{setLocalProperty}} methods, which are intended

[jira] [Created] (SPARK-29017) JobGroup and LocalProperty not respected by PySpark

Imran Rashid created SPARK-29017: Summary: JobGroup and LocalProperty not respected by PySpark Key: SPARK-29017 URL: https://issues.apache.org/jira/browse/SPARK-29017 Project: Spark Issue

[jira] [Resolved] (SPARK-28770) Flaky Tests: Test ReplayListenerSuite.End-to-end replay with compression failed

[ https://issues.apache.org/jira/browse/SPARK-28770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-28770. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25673

[jira] [Assigned] (SPARK-28770) Flaky Tests: Test ReplayListenerSuite.End-to-end replay with compression failed

[ https://issues.apache.org/jira/browse/SPARK-28770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-28770: Assignee: Wing Yew Poon > Flaky Tests: Test ReplayListenerSuite.End-to-end replay with

[jira] [Commented] (SPARK-28770) Flaky Tests: Test ReplayListenerSuite.End-to-end replay with compression failed

[ https://issues.apache.org/jira/browse/SPARK-28770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922825#comment-16922825 ] Imran Rashid commented on SPARK-28770: -- Wing yew and I chatted about this offline a bit. I follow

[jira] [Commented] (SPARK-28770) Flaky Tests: Test ReplayListenerSuite.End-to-end replay with compression failed

[ https://issues.apache.org/jira/browse/SPARK-28770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922771#comment-16922771 ] Imran Rashid commented on SPARK-28770: -- I'm not sure I follow ... the metrics from the driver will

[jira] [Created] (SPARK-28917) Jobs can hang because of race of RDD.dependencies

Imran Rashid created SPARK-28917: Summary: Jobs can hang because of race of RDD.dependencies Key: SPARK-28917 URL: https://issues.apache.org/jira/browse/SPARK-28917 Project: Spark Issue

[jira] [Commented] (SPARK-28917) Jobs can hang because of race of RDD.dependencies

[ https://issues.apache.org/jira/browse/SPARK-28917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918764#comment-16918764 ] Imran Rashid commented on SPARK-28917: -- [~markhamstra] [~jiangxb1987] [~tgraves] [~Ngone51] would

[jira] [Updated] (SPARK-28626) Spark leaves unencrypted data on local disk, even with encryption turned on (CVE-2019-10099)

[ https://issues.apache.org/jira/browse/SPARK-28626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-28626: - Description: Severity: Important   Vendor: The Apache Software Foundation   Versions

[jira] [Updated] (SPARK-28626) Spark leaves unencrypted data on local disk, even with encryption turned on (CVE-2019-10099)

[ https://issues.apache.org/jira/browse/SPARK-28626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-28626: - Description: Severity: Important   Vendor: The Apache Software Foundation   Versions

[jira] [Created] (SPARK-28626) Spark leaves unencrypted data on local disk, even with encryption turned on (CVE-2019-10099)

Imran Rashid created SPARK-28626: Summary: Spark leaves unencrypted data on local disk, even with encryption turned on (CVE-2019-10099) Key: SPARK-28626 URL: https://issues.apache.org/jira/browse/SPARK-28626

[jira] [Resolved] (SPARK-28626) Spark leaves unencrypted data on local disk, even with encryption turned on (CVE-2019-10099)

[ https://issues.apache.org/jira/browse/SPARK-28626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-28626. -- Resolution: Fixed > Spark leaves unencrypted data on local disk, even with encryption turned

[jira] [Updated] (SPARK-28592) Mark new Shuffle apis as @Experimental (instead of @Private)

[ https://issues.apache.org/jira/browse/SPARK-28592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-28592: - Description: The new Shuffle api is initially marked as {{@Private}}, just to discourage anyone

[jira] [Updated] (SPARK-28592) Mark new Shuffle apis as @Experimental (instead of @Private)

[ https://issues.apache.org/jira/browse/SPARK-28592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-28592: - Summary: Mark new Shuffle apis as @Experimental (instead of @Private) (was: Mark apis as

[jira] [Resolved] (SPARK-26329) ExecutorMetrics should poll faster than heartbeats

[ https://issues.apache.org/jira/browse/SPARK-26329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-26329. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23767

[jira] [Assigned] (SPARK-26329) ExecutorMetrics should poll faster than heartbeats

[ https://issues.apache.org/jira/browse/SPARK-26329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-26329: Assignee: Wing Yew Poon > ExecutorMetrics should poll faster than heartbeats >

[jira] [Created] (SPARK-28592) Mark apis as @Experimental (instead of @Private)

Imran Rashid created SPARK-28592: Summary: Mark apis as @Experimental (instead of @Private) Key: SPARK-28592 URL: https://issues.apache.org/jira/browse/SPARK-28592 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-26755) Optimize Spark Scheduler to dequeue speculative tasks more efficiently

[ https://issues.apache.org/jira/browse/SPARK-26755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-26755: Assignee: Parth Gandhi > Optimize Spark Scheduler to dequeue speculative tasks more

[jira] [Resolved] (SPARK-26755) Optimize Spark Scheduler to dequeue speculative tasks more efficiently

[ https://issues.apache.org/jira/browse/SPARK-26755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-26755. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23677

[jira] [Commented] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.

[ https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886359#comment-16886359 ] Imran Rashid commented on SPARK-27812: -- as mentioned elsewhere, installing an uncaught exception

[jira] [Commented] (SPARK-28165) SHS does not delete old inprogress files until cleaner.maxAge after SHS start time

[ https://issues.apache.org/jira/browse/SPARK-28165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873614#comment-16873614 ] Imran Rashid commented on SPARK-28165: -- btw if anybody wants to investigate this more, here's a

[jira] [Assigned] (SPARK-28005) SparkRackResolver should not log for resolving empty list

[ https://issues.apache.org/jira/browse/SPARK-28005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-28005: Assignee: Gabor Somogyi > SparkRackResolver should not log for resolving empty list >

[jira] [Resolved] (SPARK-28005) SparkRackResolver should not log for resolving empty list

[ https://issues.apache.org/jira/browse/SPARK-28005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-28005. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24935

[jira] [Resolved] (SPARK-27630) Stage retry causes totalRunningTasks calculation to be negative

[ https://issues.apache.org/jira/browse/SPARK-27630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-27630. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24497

[jira] [Assigned] (SPARK-27630) Stage retry causes totalRunningTasks calculation to be negative

[ https://issues.apache.org/jira/browse/SPARK-27630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-27630: Assignee: dzcxzl > Stage retry causes totalRunningTasks calculation to be negative >

[jira] [Created] (SPARK-28165) SHS does not delete old inprogress files until cleaner.maxAge after SHS start time

Imran Rashid created SPARK-28165: Summary: SHS does not delete old inprogress files until cleaner.maxAge after SHS start time Key: SPARK-28165 URL: https://issues.apache.org/jira/browse/SPARK-28165

[jira] [Commented] (SPARK-28005) SparkRackResolver should not log for resolving empty list

[ https://issues.apache.org/jira/browse/SPARK-28005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861410#comment-16861410 ] Imran Rashid commented on SPARK-28005: -- cc [~cltlfcjin] > SparkRackResolver should not log for

[jira] [Created] (SPARK-28005) SparkRackResolver should not log for resolving empty list

Imran Rashid created SPARK-28005: Summary: SparkRackResolver should not log for resolving empty list Key: SPARK-28005 URL: https://issues.apache.org/jira/browse/SPARK-28005 Project: Spark

[jira] [Resolved] (SPARK-20286) dynamicAllocation.executorIdleTimeout is ignored after unpersist

[ https://issues.apache.org/jira/browse/SPARK-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-20286. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24704

[jira] [Assigned] (SPARK-20286) dynamicAllocation.executorIdleTimeout is ignored after unpersist

[ https://issues.apache.org/jira/browse/SPARK-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-20286: Assignee: Marcelo Vanzin > dynamicAllocation.executorIdleTimeout is ignored after

[jira] [Created] (SPARK-27780) Shuffle server & client should be versioned to enable smoother upgrade

Imran Rashid created SPARK-27780: Summary: Shuffle server & client should be versioned to enable smoother upgrade Key: SPARK-27780 URL: https://issues.apache.org/jira/browse/SPARK-27780 Project:

[jira] [Commented] (SPARK-25888) Service requests for persist() blocks via external service after dynamic deallocation

[ https://issues.apache.org/jira/browse/SPARK-25888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837632#comment-16837632 ] Imran Rashid commented on SPARK-25888: -- There is a lot of stuff listed here, so I think it makes

[jira] [Created] (SPARK-27677) Disk-persisted RDD blocks served by shuffle service, and ignored for Dynamic Allocation

Imran Rashid created SPARK-27677: Summary: Disk-persisted RDD blocks served by shuffle service, and ignored for Dynamic Allocation Key: SPARK-27677 URL: https://issues.apache.org/jira/browse/SPARK-27677

[jira] [Commented] (SPARK-27677) Disk-persisted RDD blocks served by shuffle service, and ignored for Dynamic Allocation

[ https://issues.apache.org/jira/browse/SPARK-27677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837629#comment-16837629 ] Imran Rashid commented on SPARK-27677: -- I'm breaking this out of SPARK-25888. [~attilapiros] is

[jira] [Resolved] (SPARK-27590) do not consider skipped tasks when scheduling speculative tasks

[ https://issues.apache.org/jira/browse/SPARK-27590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-27590. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24485

[jira] [Commented] (SPARK-27367) Faster RoaringBitmap Serialization with v0.8.0

[ https://issues.apache.org/jira/browse/SPARK-27367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822045#comment-16822045 ] Imran Rashid commented on SPARK-27367: -- Did you change spark code as well, to use the new suggested

[jira] [Commented] (SPARK-25422) flaky test: org.apache.spark.DistributedSuite.caching on disk, replicated (encryption = on) (with replication as stream)

[ https://issues.apache.org/jira/browse/SPARK-25422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821178#comment-16821178 ] Imran Rashid commented on SPARK-25422: -- [~magnusfa] handling blocks > 2GB was broken in many ways

[jira] [Commented] (SPARK-25088) Rest Server default & doc updates

[ https://issues.apache.org/jira/browse/SPARK-25088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820321#comment-16820321 ] Imran Rashid commented on SPARK-25088: -- if you're allowing unauthed rest, what is the point of auth

[jira] [Commented] (SPARK-6235) Address various 2G limits

[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820309#comment-16820309 ] Imran Rashid commented on SPARK-6235: - [~glal14] actually this was fixed in 2.4. There was one open

[jira] [Resolved] (SPARK-6235) Address various 2G limits

[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-6235. - Resolution: Fixed Fix Version/s: 2.4.0 > Address various 2G limits >

[jira] [Resolved] (SPARK-24936) Better error message when trying a shuffle fetch over 2 GB

[ https://issues.apache.org/jira/browse/SPARK-24936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-24936. -- Resolution: Won't Fix As we've already shipped 2.4, I think its unlikely we're going to fix

[jira] [Commented] (SPARK-25250) Race condition with tasks running when new attempt for same stage is created leads to other task in the next attempt running on the same partition id retry multiple ti

[ https://issues.apache.org/jira/browse/SPARK-25250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818318#comment-16818318 ] Imran Rashid commented on SPARK-25250: -- I agree about opening a new jira. Wenchen discussed

[jira] [Assigned] (SPARK-13704) TaskSchedulerImpl.createTaskSetManager can be expensive, and result in lost executors due to blocked heartbeats

[ https://issues.apache.org/jira/browse/SPARK-13704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-13704: Assignee: Lantao Jin > TaskSchedulerImpl.createTaskSetManager can be expensive, and

[jira] [Resolved] (SPARK-13704) TaskSchedulerImpl.createTaskSetManager can be expensive, and result in lost executors due to blocked heartbeats

[ https://issues.apache.org/jira/browse/SPARK-13704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-13704. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24245

[jira] [Updated] (SPARK-27389) pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"

[ https://issues.apache.org/jira/browse/SPARK-27389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-27389: - Summary: pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'" (was: Odd failures

[jira] [Created] (SPARK-27389) Odd failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"

Imran Rashid created SPARK-27389: Summary: Odd failures w/ "UnknownTimeZoneError: 'US/Pacific-New'" Key: SPARK-27389 URL: https://issues.apache.org/jira/browse/SPARK-27389 Project: Spark

[jira] [Updated] (SPARK-27389) Odd failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"

[ https://issues.apache.org/jira/browse/SPARK-27389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-27389: - Component/s: PySpark > Odd failures w/ "UnknownTimeZoneError: 'US/Pacific-New'" >

[jira] [Resolved] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue

[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-27216. -- Resolution: Fixed Assignee: Lantao Jin > Upgrade RoaringBitmap to 0.7.45 to fix Kryo

[jira] [Commented] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue

[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809415#comment-16809415 ] Imran Rashid commented on SPARK-27216: -- Fixed by https://github.com/apache/spark/pull/24264 in

[jira] [Updated] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue

[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-27216: - Fix Version/s: 3.0.0 > Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue >

[jira] [Created] (SPARK-27367) Faster RoaringBitmap Serialization with v0.8.0

Imran Rashid created SPARK-27367: Summary: Faster RoaringBitmap Serialization with v0.8.0 Key: SPARK-27367 URL: https://issues.apache.org/jira/browse/SPARK-27367 Project: Spark Issue Type:

[jira] [Commented] (SPARK-27220) Remove Yarn specific leftover from CoarseGrainedSchedulerBackend

[ https://issues.apache.org/jira/browse/SPARK-27220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808000#comment-16808000 ] Imran Rashid commented on SPARK-27220: -- {quote} because we add something to executor id and it ends

[jira] [Commented] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue

[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807749#comment-16807749 ] Imran Rashid commented on SPARK-27216: -- I think its probably fine to upgrade it, I just wanted to

[jira] [Commented] (SPARK-27216) Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue

[ https://issues.apache.org/jira/browse/SPARK-27216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807181#comment-16807181 ] Imran Rashid commented on SPARK-27216: -- Any chance you know what the issue in roaring bitmap is?

[jira] [Commented] (SPARK-27121) Resolve Scala compiler failure for Java 9+ in REPL

[ https://issues.apache.org/jira/browse/SPARK-27121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804184#comment-16804184 ] Imran Rashid commented on SPARK-27121: -- I think this might just be an issue with the test setup.

[jira] [Commented] (SPARK-27220) Remove Yarn specific leftover from CoarseGrainedSchedulerBackend

[ https://issues.apache.org/jira/browse/SPARK-27220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16802202#comment-16802202 ] Imran Rashid commented on SPARK-27220: -- sorry for the delay. i don't have a really strong opinion

[jira] [Commented] (SPARK-21097) Dynamic allocation will preserve cached data

[ https://issues.apache.org/jira/browse/SPARK-21097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16801960#comment-16801960 ] Imran Rashid commented on SPARK-21097: -- {quote} I'm wondering if this is going to be subsumed by

[jira] [Commented] (SPARK-27112) Spark Scheduler encounters two independent Deadlocks when trying to kill executors either due to dynamic allocation or blacklisting

[ https://issues.apache.org/jira/browse/SPARK-27112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797321#comment-16797321 ] Imran Rashid commented on SPARK-27112: -- [~Dhruve Ashar] -- rc8 is already defined, there is nothing

[jira] [Updated] (SPARK-27112) Spark Scheduler encounters two independent Deadlocks when trying to kill executors either due to dynamic allocation or blacklisting

[ https://issues.apache.org/jira/browse/SPARK-27112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-27112: - Fix Version/s: (was: 2.4.1) 2.4.2 > Spark Scheduler encounters two

  1   2   3   4   5   6   7   8   9   10   >