[jira] [Resolved] (SPARK-21243) Limit the number of maps in a single shuffle fetch

2017-07-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21243. --- Resolution: Fixed Fix Version/s: 2.3.0 > Limit the number of maps in a single shuffle

[jira] [Assigned] (SPARK-21243) Limit the number of maps in a single shuffle fetch

2017-07-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-21243: - Assignee: Dhruve Ashar > Limit the number of maps in a single shuffle fetch >

[jira] [Commented] (SPARK-15703) Make ListenerBus event queue size configurable

2017-07-18 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091531#comment-16091531 ] Thomas Graves commented on SPARK-15703: --- It should not be breaking dynamic allocation, if it is,

[jira] [Resolved] (SPARK-21321) Spark very verbose on shutdown confusing users

2017-07-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21321. --- Resolution: Fixed Assignee: Jong Yoon Lee Fix Version/s: 2.3.0

[jira] [Commented] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

2017-07-12 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084458#comment-16084458 ] Thomas Graves commented on SPARK-21376: --- so you are referring to the

[jira] [Commented] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

2017-07-12 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084075#comment-16084075 ] Thomas Graves commented on SPARK-21376: --- Can you please clarify the title and description? What do

[jira] [Comment Edited] (SPARK-21383) YARN can allocate to many executors

2017-07-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083321#comment-16083321 ] Thomas Graves edited comment on SPARK-21383 at 7/12/17 2:17 AM: Note we

[jira] [Commented] (SPARK-21383) YARN can allocate to many executors

2017-07-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083321#comment-16083321 ] Thomas Graves commented on SPARK-21383: --- Note we saw this with dynamic allocation off. Its easy

[jira] [Updated] (SPARK-21383) YARN can allocate to many executors

2017-07-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21383: -- Summary: YARN can allocate to many executors (was: YARN: can allocate to many containers) >

[jira] [Created] (SPARK-21383) YARN: can allocate to many containers

2017-07-11 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-21383: - Summary: YARN: can allocate to many containers Key: SPARK-21383 URL: https://issues.apache.org/jira/browse/SPARK-21383 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-07-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080340#comment-16080340 ] Thomas Graves commented on SPARK-19659: --- can you be more specific here, do you mean that if you

[jira] [Commented] (SPARK-21122) Address starvation issues when dynamic allocation is enabled

2017-07-05 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075147#comment-16075147 ] Thomas Graves commented on SPARK-21122: --- I agree with Sean on this. if you are aiming this at

[jira] [Resolved] (SPARK-13669) Job will always fail in the external shuffle service unavailable situation

2017-06-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-13669. --- Resolution: Fixed Fix Version/s: 2.3.0 > Job will always fail in the external shuffle

[jira] [Assigned] (SPARK-13669) Job will always fail in the external shuffle service unavailable situation

2017-06-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-13669: - Assignee: Saisai Shao > Job will always fail in the external shuffle service

[jira] [Assigned] (SPARK-20898) spark.blacklist.killBlacklistedExecutors doesn't work in YARN

2017-06-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-20898: - Assignee: Saisai Shao > spark.blacklist.killBlacklistedExecutors doesn't work in YARN >

[jira] [Resolved] (SPARK-20898) spark.blacklist.killBlacklistedExecutors doesn't work in YARN

2017-06-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-20898. --- Resolution: Fixed Fix Version/s: 2.3.0 > spark.blacklist.killBlacklistedExecutors

[jira] [Commented] (SPARK-21082) Consider Executor's memory usage when scheduling task

2017-06-20 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056118#comment-16056118 ] Thomas Graves commented on SPARK-21082: --- Yes executor should not oom if you are trying to cache to

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

2017-06-14 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049228#comment-16049228 ] Thomas Graves commented on SPARK-20589: --- [~Robin Shao] can you please clarify your comment? What

[jira] [Created] (SPARK-20970) Deprecate TaskMetrics._updatedBlockStatuses

2017-06-02 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-20970: - Summary: Deprecate TaskMetrics._updatedBlockStatuses Key: SPARK-20970 URL: https://issues.apache.org/jira/browse/SPARK-20970 Project: Spark Issue Type:

[jira] [Commented] (SPARK-20923) TaskMetrics._updatedBlockStatuses uses a lot of memory

2017-05-30 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030059#comment-16030059 ] Thomas Graves commented on SPARK-20923: --- taking a quick look at the history of the

[jira] [Commented] (SPARK-20923) TaskMetrics._updatedBlockStatuses uses a lot of memory

2017-05-30 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029917#comment-16029917 ] Thomas Graves commented on SPARK-20923: --- [~joshrosen] [~zsxwing] [~eseyfe] I think you have looked

[jira] [Commented] (SPARK-20923) TaskMetrics._updatedBlockStatuses uses a lot of memory

2017-05-30 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029701#comment-16029701 ] Thomas Graves commented on SPARK-20923: --- [~rdblue] with SPARK-20084, did you see anything using

[jira] [Created] (SPARK-20923) TaskMetrics._updatedBlockStatuses uses a lot of memory

2017-05-30 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-20923: - Summary: TaskMetrics._updatedBlockStatuses uses a lot of memory Key: SPARK-20923 URL: https://issues.apache.org/jira/browse/SPARK-20923 Project: Spark

[jira] [Commented] (SPARK-20178) Improve Scheduler fetch failures

2017-05-30 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029425#comment-16029425 ] Thomas Graves commented on SPARK-20178: --- Yeah I think we should do something here. I never looked

[jira] [Created] (SPARK-20898) spark.blacklist.killBlacklistedExecutors doesn't work in YARN

2017-05-26 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-20898: - Summary: spark.blacklist.killBlacklistedExecutors doesn't work in YARN Key: SPARK-20898 URL: https://issues.apache.org/jira/browse/SPARK-20898 Project: Spark

[jira] [Commented] (SPARK-20178) Improve Scheduler fetch failures

2017-05-23 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021199#comment-16021199 ] Thomas Graves commented on SPARK-20178: --- | My understanding of today's code is that a single

[jira] [Commented] (SPARK-20178) Improve Scheduler fetch failures

2017-05-22 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019650#comment-16019650 ] Thomas Graves commented on SPARK-20178: --- | when the DAGScheduler is notified of a FetchFailure from

[jira] [Commented] (SPARK-19354) Killed tasks are getting marked as FAILED

2017-05-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16006549#comment-16006549 ] Thomas Graves commented on SPARK-19354: --- just an fyi, filed

[jira] [Created] (SPARK-20713) Speculative task that got CommitDenied exception shows up as failed

2017-05-11 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-20713: - Summary: Speculative task that got CommitDenied exception shows up as failed Key: SPARK-20713 URL: https://issues.apache.org/jira/browse/SPARK-20713 Project: Spark

[jira] [Closed] (SPARK-19354) Killed tasks are getting marked as FAILED

2017-05-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves closed SPARK-19354. - Resolution: Duplicate > Killed tasks are getting marked as FAILED >

[jira] [Commented] (SPARK-19354) Killed tasks are getting marked as FAILED

2017-05-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16006464#comment-16006464 ] Thomas Graves commented on SPARK-19354: --- thanks for pointing those out, that does fix this issue, I

[jira] [Commented] (SPARK-19354) Killed tasks are getting marked as FAILED

2017-05-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005817#comment-16005817 ] Thomas Graves commented on SPARK-19354: --- Right from what I've seen not a blacklisting bug. Bug with

[jira] [Comment Edited] (SPARK-19354) Killed tasks are getting marked as FAILED

2017-05-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005189#comment-16005189 ] Thomas Graves edited comment on SPARK-19354 at 5/10/17 6:37 PM:

[jira] [Updated] (SPARK-19354) Killed tasks are getting marked as FAILED

2017-05-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-19354: -- Priority: Major (was: Minor) > Killed tasks are getting marked as FAILED >

[jira] [Commented] (SPARK-19354) Killed tasks are getting marked as FAILED

2017-05-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005189#comment-16005189 ] Thomas Graves commented on SPARK-19354: --- [~squito] wondering if you have seen the issue with

[jira] [Updated] (SPARK-19354) Killed tasks are getting marked as FAILED

2017-05-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-19354: -- Issue Type: Bug (was: Improvement) > Killed tasks are getting marked as FAILED >

[jira] [Commented] (SPARK-19354) Killed tasks are getting marked as FAILED

2017-05-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005182#comment-16005182 ] Thomas Graves commented on SPARK-19354: --- This is definitely causing issues with blacklisting.

[jira] [Resolved] (SPARK-20355) Display Spark version on history page

2017-05-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-20355. --- Resolution: Fixed Assignee: Sanket Reddy Fix Version/s: 2.3.0 > Display

[jira] [Commented] (SPARK-19112) add codec for ZStandard

2017-05-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000705#comment-16000705 ] Thomas Graves commented on SPARK-19112: --- also can you put some more details about the benchmark

[jira] [Commented] (SPARK-19112) add codec for ZStandard

2017-05-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000703#comment-16000703 ] Thomas Graves commented on SPARK-19112: --- Execution time is definitely worse, did you get the

[jira] [Commented] (SPARK-18971) Netty issue may cause the shuffle client hang

2017-05-05 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998531#comment-15998531 ] Thomas Graves commented on SPARK-18971: --- [~zsxwing]have you seen any issues with the new netty

[jira] [Comment Edited] (SPARK-18971) Netty issue may cause the shuffle client hang

2017-05-05 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998531#comment-15998531 ] Thomas Graves edited comment on SPARK-18971 at 5/5/17 4:31 PM: ---

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

2017-05-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995650#comment-15995650 ] Thomas Graves commented on SPARK-20589: --- thanks for the suggestions [~mridulm80]. Note just for

[jira] [Created] (SPARK-20589) Allow limiting task concurrency per stage

2017-05-03 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-20589: - Summary: Allow limiting task concurrency per stage Key: SPARK-20589 URL: https://issues.apache.org/jira/browse/SPARK-20589 Project: Spark Issue Type:

[jira] [Updated] (SPARK-20426) OneForOneStreamManager occupies too much memory.

2017-04-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-20426: -- Fix Version/s: 2.2.1 > OneForOneStreamManager occupies too much memory. >

[jira] [Updated] (SPARK-20426) OneForOneStreamManager occupies too much memory.

2017-04-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-20426: -- Fix Version/s: (was: 2.2.1) > OneForOneStreamManager occupies too much memory. >

[jira] [Assigned] (SPARK-20426) OneForOneStreamManager occupies too much memory.

2017-04-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-20426: - Assignee: jin xing > OneForOneStreamManager occupies too much memory. >

[jira] [Resolved] (SPARK-20426) OneForOneStreamManager occupies too much memory.

2017-04-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-20426. --- Resolution: Fixed Fix Version/s: 2.3.0 2.2.0 >

[jira] [Closed] (SPARK-20480) FileFormatWriter hides FetchFailedException from scheduler

2017-04-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves closed SPARK-20480. - Resolution: Duplicate > FileFormatWriter hides FetchFailedException from scheduler >

[jira] [Commented] (SPARK-20480) FileFormatWriter hides FetchFailedException from scheduler

2017-04-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985590#comment-15985590 ] Thomas Graves commented on SPARK-20480: --- ah it looks like it should, hadn't seen that jira, I'll

[jira] [Commented] (SPARK-20480) FileFormatWriter hides FetchFailedException from scheduler

2017-04-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985571#comment-15985571 ] Thomas Graves commented on SPARK-20480: --- Note with blacklisting on this caused the job to fail: Job

[jira] [Commented] (SPARK-20480) FileFormatWriter hides FetchFailedException from scheduler

2017-04-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985567#comment-15985567 ] Thomas Graves commented on SPARK-20480: --- exception in task manager looks like: 17/04/26 20:09:21

[jira] [Updated] (SPARK-20480) FileFormatWriter hides FetchFailedException from scheduler

2017-04-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-20480: -- Description: I was running a large job where it was getting faiures, noticed they were listed

[jira] [Created] (SPARK-20480) FileFormatWriter hides FetchFailedException from scheduler

2017-04-26 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-20480: - Summary: FileFormatWriter hides FetchFailedException from scheduler Key: SPARK-20480 URL: https://issues.apache.org/jira/browse/SPARK-20480 Project: Spark

[jira] [Commented] (SPARK-20178) Improve Scheduler fetch failures

2017-04-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985447#comment-15985447 ] Thomas Graves commented on SPARK-20178: --- Another thing we should tie in here is handling preempted

[jira] [Resolved] (SPARK-19812) YARN shuffle service fails to relocate recovery DB across NFS directories

2017-04-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-19812. --- Resolution: Fixed Fix Version/s: 2.3.0 2.2.0 > YARN shuffle

[jira] [Updated] (SPARK-19812) YARN shuffle service fails to relocate recovery DB across NFS directories

2017-04-24 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-19812: -- Summary: YARN shuffle service fails to relocate recovery DB across NFS directories (was: YARN

[jira] [Commented] (SPARK-19812) YARN shuffle service fails to relocate recovery DB directories

2017-04-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979349#comment-15979349 ] Thomas Graves commented on SPARK-19812: --- Sorry wasn't clear in the original description, it errors

[jira] [Commented] (SPARK-20391) Properly rename the memory related fields in ExecutorSummary REST API

2017-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975115#comment-15975115 ] Thomas Graves commented on SPARK-20391: --- > My proposal was to add 2 extra fields which duplicate

[jira] [Commented] (SPARK-20391) Properly rename the memory related fields in ExecutorSummary REST API

2017-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975029#comment-15975029 ] Thomas Graves commented on SPARK-20391: --- I agree that if its been released we can't change it, the

[jira] [Commented] (SPARK-14245) webUI should display the user

2017-04-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971365#comment-15971365 ] Thomas Graves commented on SPARK-14245: --- see the commend in the PR, I think there was a race with

[jira] [Commented] (SPARK-20340) Size estimate very wrong in ExternalAppendOnlyMap from CoGroupedRDD, cause OOM

2017-04-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971071#comment-15971071 ] Thomas Graves commented on SPARK-20340: --- Right, I figured it was probably for performance, the

[jira] [Created] (SPARK-20340) Size estimate very wrong in ExternalAppendOnlyMap from CoGroupedRDD, cause OOM

2017-04-14 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-20340: - Summary: Size estimate very wrong in ExternalAppendOnlyMap from CoGroupedRDD, cause OOM Key: SPARK-20340 URL: https://issues.apache.org/jira/browse/SPARK-20340

[jira] [Commented] (SPARK-20178) Improve Scheduler fetch failures

2017-04-14 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969388#comment-15969388 ] Thomas Graves commented on SPARK-20178: --- One thing I ran into today which is somewhat related to

[jira] [Comment Edited] (SPARK-20178) Improve Scheduler fetch failures

2017-03-31 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15950917#comment-15950917 ] Thomas Graves edited comment on SPARK-20178 at 3/31/17 1:53 PM: Overall

[jira] [Commented] (SPARK-20178) Improve Scheduler fetch failures

2017-03-31 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15950917#comment-15950917 ] Thomas Graves commented on SPARK-20178: --- Overall what I would like to accomplish is not throwing

[jira] [Created] (SPARK-20178) Improve Scheduler fetch failures

2017-03-31 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-20178: - Summary: Improve Scheduler fetch failures Key: SPARK-20178 URL: https://issues.apache.org/jira/browse/SPARK-20178 Project: Spark Issue Type: Epic

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2017-03-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944046#comment-15944046 ] Thomas Graves commented on SPARK-19143: --- Yes I can be Shephard. > API in Spark for distributing

[jira] [Commented] (SPARK-19904) SPIP Add Spark Project Improvement Proposal doc to website

2017-03-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943799#comment-15943799 ] Thomas Graves commented on SPARK-19904: --- Is this done or what is this waiting on? > SPIP Add

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2017-03-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943284#comment-15943284 ] Thomas Graves commented on SPARK-19143: --- I assume this needs to go through the new spip process:

[jira] [Comment Edited] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2017-03-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905247#comment-15905247 ] Thomas Graves edited comment on SPARK-19143 at 3/10/17 3:20 PM: Made some

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2017-03-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905247#comment-15905247 ] Thomas Graves commented on SPARK-19143: --- Made some comments in the design doc. My original idea

[jira] [Commented] (SPARK-19812) YARN shuffle service fails to relocate recovery DB directories

2017-03-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894584#comment-15894584 ] Thomas Graves commented on SPARK-19812: --- note that it will go ahead and start using the recovery

[jira] [Updated] (SPARK-19812) YARN shuffle service fails to relocate recovery DB directories

2017-03-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-19812: -- Summary: YARN shuffle service fails to relocate recovery DB directories (was: YARN shuffle

[jira] [Created] (SPARK-19812) YARN shuffle service fix moving recovery DB directories

2017-03-03 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-19812: - Summary: YARN shuffle service fix moving recovery DB directories Key: SPARK-19812 URL: https://issues.apache.org/jira/browse/SPARK-19812 Project: Spark

[jira] [Commented] (SPARK-18769) Spark to be smarter about what the upper bound is and to restrict number of executor when dynamic allocation is enabled

2017-03-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892318#comment-15892318 ] Thomas Graves commented on SPARK-18769: --- I definitely understand there is an actual problem here,

[jira] [Commented] (SPARK-18769) Spark to be smarter about what the upper bound is and to restrict number of executor when dynamic allocation is enabled

2017-03-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890629#comment-15890629 ] Thomas Graves commented on SPARK-18769: --- [~yuming] I already made a comment on that, I don't think

[jira] [Commented] (SPARK-18769) Spark to be smarter about what the upper bound is and to restrict number of executor when dynamic allocation is enabled

2017-02-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15888430#comment-15888430 ] Thomas Graves commented on SPARK-18769: --- {quote} A little clarification in case the summary is not

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

2017-02-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849965#comment-15849965 ] Thomas Graves commented on SPARK-18085: --- Are the ui updates just updates to use new api or are you

[jira] (SPARK-16742) Kerberos support for Spark on Mesos

2017-01-30 Thread Thomas Graves (JIRA)
Title: Message Title Thomas Graves commented on SPARK-16742

[jira] [Resolved] (SPARK-18750) spark should be able to control the number of executor and should not throw stack overslow

2017-01-25 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-18750. --- Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.2.0

[jira] [Resolved] (SPARK-19179) spark.yarn.access.namenodes description is wrong

2017-01-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-19179. --- Resolution: Fixed Fix Version/s: 2.2.0 > spark.yarn.access.namenodes description is

[jira] [Updated] (SPARK-19179) spark.yarn.access.namenodes description is wrong

2017-01-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-19179: -- Assignee: Saisai Shao > spark.yarn.access.namenodes description is wrong >

[jira] [Created] (SPARK-19179) spark.yarn.access.namenodes description is wrong

2017-01-11 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-19179: - Summary: spark.yarn.access.namenodes description is wrong Key: SPARK-19179 URL: https://issues.apache.org/jira/browse/SPARK-19179 Project: Spark Issue

[jira] [Resolved] (SPARK-19021) Generailize HDFSCredentialProvider to support non HDFS security FS

2017-01-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-19021. --- Resolution: Fixed Assignee: Saisai Shao Fix Version/s: 2.2.0 > Generailize

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2017-01-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815078#comment-15815078 ] Thomas Graves commented on SPARK-19143: --- [~mridulm80] You say "we added", you are saying you have

[jira] [Commented] (SPARK-19112) add codec for ZStandard

2017-01-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15811935#comment-15811935 ] Thomas Graves commented on SPARK-19112: --- For any sort of Hadoop output formats we get it for free

[jira] [Created] (SPARK-19112) add codec for ZStandard

2017-01-06 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-19112: - Summary: add codec for ZStandard Key: SPARK-19112 URL: https://issues.apache.org/jira/browse/SPARK-19112 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-19033) HistoryServer still uses old ACLs even if ACLs are updated

2017-01-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-19033. --- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > HistoryServer still

[jira] [Updated] (SPARK-19033) HistoryServer still uses old ACLs even if ACLs are updated

2017-01-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-19033: -- Assignee: Saisai Shao > HistoryServer still uses old ACLs even if ACLs are updated >

[jira] [Commented] (SPARK-19033) HistoryServer still uses old ACLs even if ACLs are updated

2017-01-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795863#comment-15795863 ] Thomas Graves commented on SPARK-19033: --- this should actually be an improvement to allow admin acls

[jira] [Commented] (SPARK-18733) Spark history server file cleaner excludes in-progress files

2016-12-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726701#comment-15726701 ] Thomas Graves commented on SPARK-18733: --- oh nevermind its looking at lastupdated time. > Spark

[jira] [Commented] (SPARK-18733) Spark history server file cleaner excludes in-progress files

2016-12-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726671#comment-15726671 ] Thomas Graves commented on SPARK-18733: --- yes looks like a dup but I'm not sure on current solution.

[jira] [Updated] (SPARK-17843) History Server gives no feedback about the application list being incomplete

2016-11-30 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-17843: -- Fix Version/s: 2.0.3 > History Server gives no feedback about the application list being

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

2016-11-15 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15667226#comment-15667226 ] Thomas Graves commented on SPARK-18085: --- I think at a high level its fine. You mention using a /v2

[jira] [Resolved] (SPARK-17843) History Server gives no feedback about the application list being incomplete

2016-11-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-17843. --- Resolution: Fixed Assignee: Vinayak Joshi Fix Version/s: 2.2.0

[jira] [Resolved] (SPARK-18357) YARN --files/--archives broke

2016-11-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-18357. --- Resolution: Fixed Fix Version/s: 2.2.0 2.1.0 > YARN

[jira] [Updated] (SPARK-18357) YARN --files/--archives broke

2016-11-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-18357: -- Assignee: Kishor Patil > YARN --files/--archives broke > - > >

[jira] [Commented] (SPARK-18357) YARN --files/--archives broke

2016-11-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15647941#comment-15647941 ] Thomas Graves commented on SPARK-18357: --- [~kishorvpatil] can you fix please > YARN

[jira] [Updated] (SPARK-18357) YARN --files/--archives broke

2016-11-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-18357: -- Priority: Blocker (was: Major) > YARN --files/--archives broke >

<    5   6   7   8   9   10   11   12   13   14   >