[jira] [Updated] (SPARK-21656) spark dynamic allocation should not idle timeout executors when tasks still to run

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21656: -- Description: Right now spark lets go of executors when they are idle for the 60s (or configura

[jira] [Updated] (SPARK-21656) spark dynamic allocation should not idle timeout executors when there are enough tasks to run on them

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21656: -- Summary: spark dynamic allocation should not idle timeout executors when there are enough tasks

[jira] [Commented] (SPARK-21656) spark dynamic allocation should not idle timeout executors when tasks still to run

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117241#comment-16117241 ] Thomas Graves commented on SPARK-21656: --- As a said above it DOES help the applicati

[jira] [Commented] (SPARK-21656) spark dynamic allocation should not idle timeout executors when tasks still to run

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117204#comment-16117204 ] Thomas Graves commented on SPARK-21656: --- Another option would be just to add logic

[jira] [Commented] (SPARK-21656) spark dynamic allocation should not idle timeout executors when tasks still to run

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117200#comment-16117200 ] Thomas Graves commented on SPARK-21656: --- why not fix the bug in dynamic allocation?

[jira] [Commented] (SPARK-21656) spark dynamic allocation should not idle timeout executors when tasks still to run

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117159#comment-16117159 ] Thomas Graves commented on SPARK-21656: --- If given more time the scheduler would hav

[jira] [Commented] (SPARK-21656) spark dynamic allocation should not idle timeout executors when tasks still to run

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117086#comment-16117086 ] Thomas Graves commented on SPARK-21656: --- The executor can be idle if the scheduler

[jira] [Updated] (SPARK-21656) spark dynamic allocation should not idle timeout executors when tasks still to run

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21656: -- Issue Type: Bug (was: Improvement) > spark dynamic allocation should not idle timeout executor

[jira] [Updated] (SPARK-21656) spark dynamic allocation should not idle timeout executors when tasks still to run

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21656: -- Priority: Major (was: Minor) > spark dynamic allocation should not idle timeout executors when

[jira] [Commented] (SPARK-21655) Kill CLI for Yarn mode

[ https://issues.apache.org/jira/browse/SPARK-21655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116989#comment-16116989 ] Thomas Graves commented on SPARK-21655: --- The UI kill requests are acl protected. Y

[jira] [Commented] (SPARK-21655) Kill CLI for Yarn mode

[ https://issues.apache.org/jira/browse/SPARK-21655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116859#comment-16116859 ] Thomas Graves commented on SPARK-21655: --- the yarn kill does work, but it does a kil

[jira] [Assigned] (SPARK-20713) Speculative task that got CommitDenied exception shows up as failed

[ https://issues.apache.org/jira/browse/SPARK-20713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-20713: - Assignee: (was: Nuochen Lyu) > Speculative task that got CommitDenied exception show

[jira] [Assigned] (SPARK-20713) Speculative task that got CommitDenied exception shows up as failed

[ https://issues.apache.org/jira/browse/SPARK-20713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-20713: - Assignee: Nuochen Lyu > Speculative task that got CommitDenied exception shows up as fai

[jira] [Resolved] (SPARK-20713) Speculative task that got CommitDenied exception shows up as failed

[ https://issues.apache.org/jira/browse/SPARK-20713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-20713. --- Resolution: Fixed Assignee: Nuochen Lyu Fix Version/s: 2.3.0 > Speculative ta

[jira] [Commented] (SPARK-21585) Application Master marking application status as Failed for Client Mode

[ https://issues.apache.org/jira/browse/SPARK-21585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108913#comment-16108913 ] Thomas Graves commented on SPARK-21585: --- https://github.com/apache/spark/pull/18788

[jira] [Resolved] (SPARK-21585) Application Master marking application status as Failed for Client Mode

[ https://issues.apache.org/jira/browse/SPARK-21585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21585. --- Resolution: Fixed Assignee: Parth Gandhi Fix Version/s: 2.3.0 > Application M

[jira] [Commented] (SPARK-21585) Application Master marking application status as Failed for Client Mode

[ https://issues.apache.org/jira/browse/SPARK-21585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108901#comment-16108901 ] Thomas Graves commented on SPARK-21585: --- [~srowen]do you know if the github pull re

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105164#comment-16105164 ] Thomas Graves commented on SPARK-17321: --- Can you clarify? as stated above you sho

[jira] [Commented] (SPARK-21541) Spark Logs show incorrect job status for a job that does not create SparkContext

[ https://issues.apache.org/jira/browse/SPARK-21541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105133#comment-16105133 ] Thomas Graves commented on SPARK-21541: --- it was merged https://github.com/apache/s

[jira] [Resolved] (SPARK-21541) Spark Logs show incorrect job status for a job that does not create SparkContext

[ https://issues.apache.org/jira/browse/SPARK-21541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21541. --- Resolution: Fixed Assignee: Parth Gandhi Fix Version/s: 2.3.0 > Spark Logs sh

[jira] [Created] (SPARK-21530) Update description of spark.shuffle.maxChunksBeingTransferred

Thomas Graves created SPARK-21530: - Summary: Update description of spark.shuffle.maxChunksBeingTransferred Key: SPARK-21530 URL: https://issues.apache.org/jira/browse/SPARK-21530 Project: Spark

[jira] [Commented] (SPARK-21501) Spark shuffle index cache size should be memory based

[ https://issues.apache.org/jira/browse/SPARK-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1616#comment-1616 ] Thomas Graves commented on SPARK-21501: --- We want to change it from a # of entries t

[jira] [Commented] (SPARK-21501) Spark shuffle index cache size should be memory based

[ https://issues.apache.org/jira/browse/SPARK-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098907#comment-16098907 ] Thomas Graves commented on SPARK-21501: --- The issue was actually introduced with SPA

[jira] [Updated] (SPARK-21501) Spark shuffle index cache size should be memory based

[ https://issues.apache.org/jira/browse/SPARK-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21501: -- Affects Version/s: (was: 2.0.0) 2.1.0 > Spark shuffle index cache si

[jira] [Updated] (SPARK-21243) Limit the number of maps in a single shuffle fetch

[ https://issues.apache.org/jira/browse/SPARK-21243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21243: -- Fix Version/s: 2.2.1 > Limit the number of maps in a single shuffle fetch > ---

[jira] [Created] (SPARK-21501) Spark shuffle index cache size should be memory based

Thomas Graves created SPARK-21501: - Summary: Spark shuffle index cache size should be memory based Key: SPARK-21501 URL: https://issues.apache.org/jira/browse/SPARK-21501 Project: Spark Issue

[jira] [Commented] (SPARK-21460) Spark dynamic allocation breaks when ListenerBus event queue runs full

[ https://issues.apache.org/jira/browse/SPARK-21460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095546#comment-16095546 ] Thomas Graves commented on SPARK-21460: --- I didn't think that was the case, but took

[jira] [Resolved] (SPARK-21243) Limit the number of maps in a single shuffle fetch

[ https://issues.apache.org/jira/browse/SPARK-21243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21243. --- Resolution: Fixed Fix Version/s: 2.3.0 > Limit the number of maps in a single shuffle

[jira] [Assigned] (SPARK-21243) Limit the number of maps in a single shuffle fetch

[ https://issues.apache.org/jira/browse/SPARK-21243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-21243: - Assignee: Dhruve Ashar > Limit the number of maps in a single shuffle fetch > --

[jira] [Commented] (SPARK-15703) Make ListenerBus event queue size configurable

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091531#comment-16091531 ] Thomas Graves commented on SPARK-15703: --- It should not be breaking dynamic allocati

[jira] [Resolved] (SPARK-21321) Spark very verbose on shutdown confusing users

[ https://issues.apache.org/jira/browse/SPARK-21321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21321. --- Resolution: Fixed Assignee: Jong Yoon Lee Fix Version/s: 2.3.0

[jira] [Commented] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084458#comment-16084458 ] Thomas Graves commented on SPARK-21376: --- so you are referring to the org.apache.spa

[jira] [Commented] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084075#comment-16084075 ] Thomas Graves commented on SPARK-21376: --- Can you please clarify the title and descr

[jira] [Comment Edited] (SPARK-21383) YARN can allocate to many executors

[ https://issues.apache.org/jira/browse/SPARK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083321#comment-16083321 ] Thomas Graves edited comment on SPARK-21383 at 7/12/17 2:17 AM: ---

[jira] [Commented] (SPARK-21383) YARN can allocate to many executors

[ https://issues.apache.org/jira/browse/SPARK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083321#comment-16083321 ] Thomas Graves commented on SPARK-21383: --- Note we saw this with dynamic allocation o

[jira] [Updated] (SPARK-21383) YARN can allocate to many executors

[ https://issues.apache.org/jira/browse/SPARK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21383: -- Summary: YARN can allocate to many executors (was: YARN: can allocate to many containers) > Y

[jira] [Created] (SPARK-21383) YARN: can allocate to many containers

Thomas Graves created SPARK-21383: - Summary: YARN: can allocate to many containers Key: SPARK-21383 URL: https://issues.apache.org/jira/browse/SPARK-21383 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16080340#comment-16080340 ] Thomas Graves commented on SPARK-19659: --- can you be more specific here, do you mean

[jira] [Commented] (SPARK-21122) Address starvation issues when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-21122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16075147#comment-16075147 ] Thomas Graves commented on SPARK-21122: --- I agree with Sean on this. if you are aim

[jira] [Resolved] (SPARK-13669) Job will always fail in the external shuffle service unavailable situation

[ https://issues.apache.org/jira/browse/SPARK-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-13669. --- Resolution: Fixed Fix Version/s: 2.3.0 > Job will always fail in the external shuffle

[jira] [Assigned] (SPARK-13669) Job will always fail in the external shuffle service unavailable situation

[ https://issues.apache.org/jira/browse/SPARK-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-13669: - Assignee: Saisai Shao > Job will always fail in the external shuffle service unavailable

[jira] [Assigned] (SPARK-20898) spark.blacklist.killBlacklistedExecutors doesn't work in YARN

[ https://issues.apache.org/jira/browse/SPARK-20898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-20898: - Assignee: Saisai Shao > spark.blacklist.killBlacklistedExecutors doesn't work in YARN >

[jira] [Resolved] (SPARK-20898) spark.blacklist.killBlacklistedExecutors doesn't work in YARN

[ https://issues.apache.org/jira/browse/SPARK-20898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-20898. --- Resolution: Fixed Fix Version/s: 2.3.0 > spark.blacklist.killBlacklistedExecutors does

[jira] [Commented] (SPARK-21082) Consider Executor's memory usage when scheduling task

[ https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16056118#comment-16056118 ] Thomas Graves commented on SPARK-21082: --- Yes executor should not oom if you are try

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16049228#comment-16049228 ] Thomas Graves commented on SPARK-20589: --- [~Robin Shao] can you please clarify your

[jira] [Created] (SPARK-20970) Deprecate TaskMetrics._updatedBlockStatuses

Thomas Graves created SPARK-20970: - Summary: Deprecate TaskMetrics._updatedBlockStatuses Key: SPARK-20970 URL: https://issues.apache.org/jira/browse/SPARK-20970 Project: Spark Issue Type: Imp

[jira] [Commented] (SPARK-20923) TaskMetrics._updatedBlockStatuses uses a lot of memory

[ https://issues.apache.org/jira/browse/SPARK-20923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16030059#comment-16030059 ] Thomas Graves commented on SPARK-20923: --- taking a quick look at the history of the

[jira] [Commented] (SPARK-20923) TaskMetrics._updatedBlockStatuses uses a lot of memory

[ https://issues.apache.org/jira/browse/SPARK-20923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16029917#comment-16029917 ] Thomas Graves commented on SPARK-20923: --- [~joshrosen] [~zsxwing] [~eseyfe] I think

[jira] [Commented] (SPARK-20923) TaskMetrics._updatedBlockStatuses uses a lot of memory

[ https://issues.apache.org/jira/browse/SPARK-20923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16029701#comment-16029701 ] Thomas Graves commented on SPARK-20923: --- [~rdblue] with SPARK-20084, did you see a

[jira] [Created] (SPARK-20923) TaskMetrics._updatedBlockStatuses uses a lot of memory

Thomas Graves created SPARK-20923: - Summary: TaskMetrics._updatedBlockStatuses uses a lot of memory Key: SPARK-20923 URL: https://issues.apache.org/jira/browse/SPARK-20923 Project: Spark Issu

[jira] [Commented] (SPARK-20178) Improve Scheduler fetch failures

[ https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16029425#comment-16029425 ] Thomas Graves commented on SPARK-20178: --- Yeah I think we should do something here.

[jira] [Created] (SPARK-20898) spark.blacklist.killBlacklistedExecutors doesn't work in YARN

Thomas Graves created SPARK-20898: - Summary: spark.blacklist.killBlacklistedExecutors doesn't work in YARN Key: SPARK-20898 URL: https://issues.apache.org/jira/browse/SPARK-20898 Project: Spark

[jira] [Commented] (SPARK-20178) Improve Scheduler fetch failures

[ https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16021199#comment-16021199 ] Thomas Graves commented on SPARK-20178: --- | My understanding of today's code is that

[jira] [Commented] (SPARK-20178) Improve Scheduler fetch failures

[ https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16019650#comment-16019650 ] Thomas Graves commented on SPARK-20178: --- | when the DAGScheduler is notified of a F

[jira] [Commented] (SPARK-19354) Killed tasks are getting marked as FAILED

[ https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16006549#comment-16006549 ] Thomas Graves commented on SPARK-19354: --- just an fyi, filed https://issues.apache.o

[jira] [Created] (SPARK-20713) Speculative task that got CommitDenied exception shows up as failed

Thomas Graves created SPARK-20713: - Summary: Speculative task that got CommitDenied exception shows up as failed Key: SPARK-20713 URL: https://issues.apache.org/jira/browse/SPARK-20713 Project: Spark

[jira] [Closed] (SPARK-19354) Killed tasks are getting marked as FAILED

[ https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves closed SPARK-19354. - Resolution: Duplicate > Killed tasks are getting marked as FAILED > -

[jira] [Commented] (SPARK-19354) Killed tasks are getting marked as FAILED

[ https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16006464#comment-16006464 ] Thomas Graves commented on SPARK-19354: --- thanks for pointing those out, that does f

[jira] [Commented] (SPARK-19354) Killed tasks are getting marked as FAILED

[ https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005817#comment-16005817 ] Thomas Graves commented on SPARK-19354: --- Right from what I've seen not a blacklisti

[jira] [Comment Edited] (SPARK-19354) Killed tasks are getting marked as FAILED

[ https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005189#comment-16005189 ] Thomas Graves edited comment on SPARK-19354 at 5/10/17 6:37 PM: ---

[jira] [Updated] (SPARK-19354) Killed tasks are getting marked as FAILED

[ https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-19354: -- Priority: Major (was: Minor) > Killed tasks are getting marked as FAILED > ---

[jira] [Commented] (SPARK-19354) Killed tasks are getting marked as FAILED

[ https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005189#comment-16005189 ] Thomas Graves commented on SPARK-19354: --- [~squito] wondering if you have seen the i

[jira] [Updated] (SPARK-19354) Killed tasks are getting marked as FAILED

[ https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-19354: -- Issue Type: Bug (was: Improvement) > Killed tasks are getting marked as FAILED > -

[jira] [Commented] (SPARK-19354) Killed tasks are getting marked as FAILED

[ https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005182#comment-16005182 ] Thomas Graves commented on SPARK-19354: --- This is definitely causing issues with bla

[jira] [Resolved] (SPARK-20355) Display Spark version on history page

[ https://issues.apache.org/jira/browse/SPARK-20355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-20355. --- Resolution: Fixed Assignee: Sanket Reddy Fix Version/s: 2.3.0 > Display Spark

[jira] [Commented] (SPARK-19112) add codec for ZStandard

[ https://issues.apache.org/jira/browse/SPARK-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000705#comment-16000705 ] Thomas Graves commented on SPARK-19112: --- also can you put some more details about t

[jira] [Commented] (SPARK-19112) add codec for ZStandard

[ https://issues.apache.org/jira/browse/SPARK-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000703#comment-16000703 ] Thomas Graves commented on SPARK-19112: --- Execution time is definitely worse, did yo

[jira] [Commented] (SPARK-18971) Netty issue may cause the shuffle client hang

[ https://issues.apache.org/jira/browse/SPARK-18971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998531#comment-15998531 ] Thomas Graves commented on SPARK-18971: --- [~zsxwing]have you seen any issues with th

[jira] [Comment Edited] (SPARK-18971) Netty issue may cause the shuffle client hang

[ https://issues.apache.org/jira/browse/SPARK-18971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998531#comment-15998531 ] Thomas Graves edited comment on SPARK-18971 at 5/5/17 4:31 PM:

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995650#comment-15995650 ] Thomas Graves commented on SPARK-20589: --- thanks for the suggestions [~mridulm80].

[jira] [Created] (SPARK-20589) Allow limiting task concurrency per stage

Thomas Graves created SPARK-20589: - Summary: Allow limiting task concurrency per stage Key: SPARK-20589 URL: https://issues.apache.org/jira/browse/SPARK-20589 Project: Spark Issue Type: Impro

[jira] [Updated] (SPARK-20426) OneForOneStreamManager occupies too much memory.

[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-20426: -- Fix Version/s: 2.2.1 > OneForOneStreamManager occupies too much memory. > -

[jira] [Updated] (SPARK-20426) OneForOneStreamManager occupies too much memory.

[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-20426: -- Fix Version/s: (was: 2.2.1) > OneForOneStreamManager occupies too much memory. > --

[jira] [Assigned] (SPARK-20426) OneForOneStreamManager occupies too much memory.

[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-20426: - Assignee: jin xing > OneForOneStreamManager occupies too much memory. >

[jira] [Resolved] (SPARK-20426) OneForOneStreamManager occupies too much memory.

[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-20426. --- Resolution: Fixed Fix Version/s: 2.3.0 2.2.0 > OneForOneStreamManag

[jira] [Closed] (SPARK-20480) FileFormatWriter hides FetchFailedException from scheduler

[ https://issues.apache.org/jira/browse/SPARK-20480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves closed SPARK-20480. - Resolution: Duplicate > FileFormatWriter hides FetchFailedException from scheduler >

[jira] [Commented] (SPARK-20480) FileFormatWriter hides FetchFailedException from scheduler

[ https://issues.apache.org/jira/browse/SPARK-20480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15985590#comment-15985590 ] Thomas Graves commented on SPARK-20480: --- ah it looks like it should, hadn't seen th

[jira] [Commented] (SPARK-20480) FileFormatWriter hides FetchFailedException from scheduler

[ https://issues.apache.org/jira/browse/SPARK-20480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15985571#comment-15985571 ] Thomas Graves commented on SPARK-20480: --- Note with blacklisting on this caused the

[jira] [Commented] (SPARK-20480) FileFormatWriter hides FetchFailedException from scheduler

[ https://issues.apache.org/jira/browse/SPARK-20480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15985567#comment-15985567 ] Thomas Graves commented on SPARK-20480: --- exception in task manager looks like: 17/0

[jira] [Updated] (SPARK-20480) FileFormatWriter hides FetchFailedException from scheduler

[ https://issues.apache.org/jira/browse/SPARK-20480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-20480: -- Description: I was running a large job where it was getting faiures, noticed they were listed

[jira] [Created] (SPARK-20480) FileFormatWriter hides FetchFailedException from scheduler

Thomas Graves created SPARK-20480: - Summary: FileFormatWriter hides FetchFailedException from scheduler Key: SPARK-20480 URL: https://issues.apache.org/jira/browse/SPARK-20480 Project: Spark

[jira] [Commented] (SPARK-20178) Improve Scheduler fetch failures

[ https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15985447#comment-15985447 ] Thomas Graves commented on SPARK-20178: --- Another thing we should tie in here is han

[jira] [Resolved] (SPARK-19812) YARN shuffle service fails to relocate recovery DB across NFS directories

[ https://issues.apache.org/jira/browse/SPARK-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-19812. --- Resolution: Fixed Fix Version/s: 2.3.0 2.2.0 > YARN shuffle service

[jira] [Updated] (SPARK-19812) YARN shuffle service fails to relocate recovery DB across NFS directories

[ https://issues.apache.org/jira/browse/SPARK-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-19812: -- Summary: YARN shuffle service fails to relocate recovery DB across NFS directories (was: YARN

[jira] [Commented] (SPARK-19812) YARN shuffle service fails to relocate recovery DB directories

[ https://issues.apache.org/jira/browse/SPARK-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979349#comment-15979349 ] Thomas Graves commented on SPARK-19812: --- Sorry wasn't clear in the original descrip

[jira] [Commented] (SPARK-20391) Properly rename the memory related fields in ExecutorSummary REST API

[ https://issues.apache.org/jira/browse/SPARK-20391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975115#comment-15975115 ] Thomas Graves commented on SPARK-20391: --- > My proposal was to add 2 extra fields wh

[jira] [Commented] (SPARK-20391) Properly rename the memory related fields in ExecutorSummary REST API

[ https://issues.apache.org/jira/browse/SPARK-20391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975029#comment-15975029 ] Thomas Graves commented on SPARK-20391: --- I agree that if its been released we can't

[jira] [Commented] (SPARK-14245) webUI should display the user

[ https://issues.apache.org/jira/browse/SPARK-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15971365#comment-15971365 ] Thomas Graves commented on SPARK-14245: --- see the commend in the PR, I think there w

[jira] [Commented] (SPARK-20340) Size estimate very wrong in ExternalAppendOnlyMap from CoGroupedRDD, cause OOM

[ https://issues.apache.org/jira/browse/SPARK-20340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15971071#comment-15971071 ] Thomas Graves commented on SPARK-20340: --- Right, I figured it was probably for perfo

[jira] [Created] (SPARK-20340) Size estimate very wrong in ExternalAppendOnlyMap from CoGroupedRDD, cause OOM

Thomas Graves created SPARK-20340: - Summary: Size estimate very wrong in ExternalAppendOnlyMap from CoGroupedRDD, cause OOM Key: SPARK-20340 URL: https://issues.apache.org/jira/browse/SPARK-20340 Proj

[jira] [Commented] (SPARK-20178) Improve Scheduler fetch failures

[ https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15969388#comment-15969388 ] Thomas Graves commented on SPARK-20178: --- One thing I ran into today which is somewh

[jira] [Comment Edited] (SPARK-20178) Improve Scheduler fetch failures

[ https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950917#comment-15950917 ] Thomas Graves edited comment on SPARK-20178 at 3/31/17 1:53 PM: ---

[jira] [Commented] (SPARK-20178) Improve Scheduler fetch failures

[ https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950917#comment-15950917 ] Thomas Graves commented on SPARK-20178: --- Overall what I would like to accomplish is

[jira] [Created] (SPARK-20178) Improve Scheduler fetch failures

Thomas Graves created SPARK-20178: - Summary: Improve Scheduler fetch failures Key: SPARK-20178 URL: https://issues.apache.org/jira/browse/SPARK-20178 Project: Spark Issue Type: Epic

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944046#comment-15944046 ] Thomas Graves commented on SPARK-19143: --- Yes I can be Shephard. > API in Spark for

[jira] [Commented] (SPARK-19904) SPIP Add Spark Project Improvement Proposal doc to website

[ https://issues.apache.org/jira/browse/SPARK-19904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943799#comment-15943799 ] Thomas Graves commented on SPARK-19904: --- Is this done or what is this waiting on?

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943284#comment-15943284 ] Thomas Graves commented on SPARK-19143: --- I assume this needs to go through the new

[jira] [Comment Edited] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905247#comment-15905247 ] Thomas Graves edited comment on SPARK-19143 at 3/10/17 3:20 PM: ---

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905247#comment-15905247 ] Thomas Graves commented on SPARK-19143: --- Made some comments in the design doc. My

[jira] [Commented] (SPARK-19812) YARN shuffle service fails to relocate recovery DB directories

[ https://issues.apache.org/jira/browse/SPARK-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894584#comment-15894584 ] Thomas Graves commented on SPARK-19812: --- note that it will go ahead and start using

<    5   6   7   8   9   10   11   12   13   14   >