[jira] [Created] (SPARK-27725) GPU Scheduling - add an example discovery Script

2019-05-15 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-27725: - Summary: GPU Scheduling - add an example discovery Script Key: SPARK-27725 URL: https://issues.apache.org/jira/browse/SPARK-27725 Project: Spark Issue

[jira] [Resolved] (SPARK-27024) Executor interface for cluster managers to support GPU resources

2019-05-14 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27024. --- Resolution: Fixed > Executor interface for cluster managers to support GPU resources >

[jira] [Commented] (SPARK-27488) Driver interface to support GPU resources

2019-05-14 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839402#comment-16839402 ] Thomas Graves commented on SPARK-27488: --- With this Jira we need to move the function that does the

[jira] [Commented] (SPARK-27520) Introduce a global config system to replace hadoopConfiguration

2019-05-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836730#comment-16836730 ] Thomas Graves commented on SPARK-27520: --- Can we add more to the description to explain why we are

[jira] [Updated] (SPARK-27492) GPU scheduling - High level user documentation

2019-05-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27492: -- Description: For the SPIP - Accelerator-aware task scheduling for Spark, 

[jira] [Commented] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

2019-05-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16835014#comment-16835014 ] Thomas Graves commented on SPARK-27396: --- I just started a vote thread on this SPIP please take a

[jira] [Commented] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

2019-04-22 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823097#comment-16823097 ] Thomas Graves commented on SPARK-27396: --- thanks for the questions and commenting, please also vote

[jira] [Comment Edited] (SPARK-27495) Support Stage level resource configuration and scheduling

2019-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822166#comment-16822166 ] Thomas Graves edited comment on SPARK-27495 at 4/19/19 8:29 PM:

[jira] [Comment Edited] (SPARK-27495) Support Stage level resource configuration and scheduling

2019-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822166#comment-16822166 ] Thomas Graves edited comment on SPARK-27495 at 4/19/19 8:28 PM:

[jira] [Commented] (SPARK-27495) Support Stage level resource configuration and scheduling

2019-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822166#comment-16822166 ] Thomas Graves commented on SPARK-27495: --- Unfortunately the link to the original design doc was

[jira] [Updated] (SPARK-27492) GPU scheduling - High level user documentation

2019-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27492: -- Description: For the SPIP - Accelerator-aware task scheduling for Spark, 

[jira] [Updated] (SPARK-27492) GPU scheduling - High level user documentation

2019-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27492: -- Summary: GPU scheduling - High level user documentation (was: High level user documentation)

[jira] [Commented] (SPARK-27492) High level user documentation

2019-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821900#comment-16821900 ] Thomas Graves commented on SPARK-27492: --- Sorry, it is under the epic and didn't realize it didn't

[jira] [Commented] (SPARK-24655) [K8S] Custom Docker Image Expectations and Documentation

2019-04-18 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821270#comment-16821270 ] Thomas Graves commented on SPARK-24655: --- >From the linked issues it seems the goals would be: *

[jira] [Created] (SPARK-27495) Support Stage level resource scheduling

2019-04-17 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-27495: - Summary: Support Stage level resource scheduling Key: SPARK-27495 URL: https://issues.apache.org/jira/browse/SPARK-27495 Project: Spark Issue Type: Story

[jira] [Updated] (SPARK-27495) Support Stage level resource configuration and scheduling

2019-04-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27495: -- Summary: Support Stage level resource configuration and scheduling (was: Support Stage level

[jira] [Created] (SPARK-27492) High level user documentation

2019-04-17 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-27492: - Summary: High level user documentation Key: SPARK-27492 URL: https://issues.apache.org/jira/browse/SPARK-27492 Project: Spark Issue Type: Story

[jira] [Created] (SPARK-27489) UI updates to show executor resource information

2019-04-17 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-27489: - Summary: UI updates to show executor resource information Key: SPARK-27489 URL: https://issues.apache.org/jira/browse/SPARK-27489 Project: Spark Issue

[jira] [Reopened] (SPARK-27364) User-facing APIs for GPU-aware scheduling

2019-04-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reopened SPARK-27364: --- reopening since it has a subtask > User-facing APIs for GPU-aware scheduling >

[jira] [Commented] (SPARK-27364) User-facing APIs for GPU-aware scheduling

2019-04-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820120#comment-16820120 ] Thomas Graves commented on SPARK-27364: --- based on no comments on this I'm going to resolve this

[jira] [Resolved] (SPARK-27364) User-facing APIs for GPU-aware scheduling

2019-04-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27364. --- Resolution: Fixed > User-facing APIs for GPU-aware scheduling >

[jira] [Commented] (SPARK-27488) Driver interface to support GPU resources

2019-04-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820119#comment-16820119 ] Thomas Graves commented on SPARK-27488: --- Note, the api design is here:

[jira] [Created] (SPARK-27488) Driver interface to support GPU resources

2019-04-17 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-27488: - Summary: Driver interface to support GPU resources Key: SPARK-27488 URL: https://issues.apache.org/jira/browse/SPARK-27488 Project: Spark Issue Type:

[jira] [Commented] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

2019-04-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16819153#comment-16819153 ] Thomas Graves commented on SPARK-27396: --- Since I don't hear any strong objections against the

[jira] [Commented] (SPARK-25250) Race condition with tasks running when new attempt for same stage is created leads to other task in the next attempt running on the same partition id retry multiple ti

2019-04-15 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818053#comment-16818053 ] Thomas Graves commented on SPARK-25250: --- [~cloud_fan] can you please add details as to where and

[jira] [Comment Edited] (SPARK-27364) User-facing APIs for GPU-aware scheduling

2019-04-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815465#comment-16815465 ] Thomas Graves edited comment on SPARK-27364 at 4/11/19 2:37 PM: So there

[jira] [Commented] (SPARK-27364) User-facing APIs for GPU-aware scheduling

2019-04-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815465#comment-16815465 ] Thomas Graves commented on SPARK-27364: --- So there is actually another one we need for standalone

[jira] [Comment Edited] (SPARK-27364) User-facing APIs for GPU-aware scheduling

2019-04-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812488#comment-16812488 ] Thomas Graves edited comment on SPARK-27364 at 4/11/19 1:48 PM: There

[jira] [Commented] (SPARK-27176) Upgrade hadoop-3's built-in Hive maven dependencies to 2.3.4

2019-04-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815414#comment-16815414 ] Thomas Graves commented on SPARK-27176: --- looks like I see one:

[jira] [Commented] (SPARK-27176) Upgrade hadoop-3's built-in Hive maven dependencies to 2.3.4

2019-04-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815408#comment-16815408 ] Thomas Graves commented on SPARK-27176: --- It looks like the hadoop-3.2 profile no longer works, do

[jira] [Commented] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

2019-04-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815403#comment-16815403 ] Thomas Graves commented on SPARK-27396: --- I can shephard it. > SPIP: Public APIs for extended

[jira] [Assigned] (SPARK-27361) YARN support for GPU-aware scheduling

2019-04-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-27361: - Assignee: Thomas Graves > YARN support for GPU-aware scheduling >

[jira] [Comment Edited] (SPARK-27364) User-facing APIs for GPU-aware scheduling

2019-04-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812488#comment-16812488 ] Thomas Graves edited comment on SPARK-27364 at 4/8/19 3:10 PM: --- There are

[jira] [Comment Edited] (SPARK-27364) User-facing APIs for GPU-aware scheduling

2019-04-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812488#comment-16812488 ] Thomas Graves edited comment on SPARK-27364 at 4/8/19 3:01 PM: --- There are

[jira] [Commented] (SPARK-27364) User-facing APIs for GPU-aware scheduling

2019-04-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812488#comment-16812488 ] Thomas Graves commented on SPARK-27364: --- There are 3 main user facing impacts for the user for

[jira] [Commented] (SPARK-27364) User-facing APIs for GPU-aware scheduling

2019-04-04 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809893#comment-16809893 ] Thomas Graves commented on SPARK-27364: --- working on this, will post a basic design when I have

[jira] [Updated] (SPARK-27024) Executor interface for cluster managers to support GPU resources

2019-04-04 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27024: -- Description: The executor interface shall deal with the resources allocated to the executor

[jira] [Updated] (SPARK-27024) Executor interface for cluster managers to support GPU resources

2019-04-04 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27024: -- Description: The executor interface shall deal with the resources allocated to the executor

[jira] [Comment Edited] (SPARK-27005) Design sketch: Accelerator-aware scheduling

2019-03-05 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784555#comment-16784555 ] Thomas Graves edited comment on SPARK-27005 at 3/5/19 3:40 PM: --- so we have

[jira] [Commented] (SPARK-27005) Design sketch: Accelerator-aware scheduling

2019-03-05 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784555#comment-16784555 ] Thomas Graves commented on SPARK-27005: --- so we have both a google design doc and the comment

[jira] [Commented] (SPARK-27024) Design executor interface to support GPU resources

2019-03-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782167#comment-16782167 ] Thomas Graves commented on SPARK-27024: --- This and SPARK-27005 basically split the design of the

[jira] [Commented] (SPARK-27024) Design executor interface to support GPU resources

2019-03-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781734#comment-16781734 ] Thomas Graves commented on SPARK-27024: --- I will be looking at this and propose a design. > Design

[jira] [Commented] (SPARK-27005) Design sketch: Accelerator-aware scheduling

2019-03-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781728#comment-16781728 ] Thomas Graves commented on SPARK-27005: --- It seems like we are mixing gpu's as static resource vs

[jira] [Commented] (SPARK-26792) Apply custom log URL to Spark UI

2019-01-30 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756750#comment-16756750 ] Thomas Graves commented on SPARK-26792: --- don't see a problem with changing the default in 3.0, its

[jira] [Commented] (SPARK-22229) SPIP: RDMA Accelerated Shuffle Engine

2019-01-25 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752356#comment-16752356 ] Thomas Graves commented on SPARK-9: --- This is interesting, a few questions * I'm assuming all

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

2019-01-23 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750375#comment-16750375 ] Thomas Graves commented on SPARK-24615: --- [~jerryshao]  just curious where this is at, are you

[jira] [Commented] (SPARK-26413) SPIP: RDD Arrow Support in Spark Core and PySpark

2019-01-23 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750174#comment-16750174 ] Thomas Graves commented on SPARK-26413: --- Just a note I think this overlaps with 

[jira] [Commented] (SPARK-26689) Bad disk causing broadcast failure

2019-01-23 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750125#comment-16750125 ] Thomas Graves commented on SPARK-26689: --- Can you add more details about your setup?  Which

[jira] [Commented] (SPARK-24374) SPIP: Support Barrier Execution Mode in Apache Spark

2019-01-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738622#comment-16738622 ] Thomas Graves commented on SPARK-24374: --- [~luzengxiang] are you just saying when spark tries to

[jira] [Commented] (YARN-9116) Capacity Scheduler: add the default maximum-allocation-mb and maximum-allocation-vcores for the queues

2019-01-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737392#comment-16737392 ] Thomas Graves commented on YARN-9116: - Yes so you want to keep the behavior that the cluster level

[jira] [Updated] (SPARK-26269) YarnAllocator should have same blacklist behaviour with YARN to maxmize use of cluster resource

2019-01-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-26269: -- Fix Version/s: 2.4.1 > YarnAllocator should have same blacklist behaviour with YARN to

[jira] [Assigned] (SPARK-26285) Add a metric source for accumulators (aka AccumulatorSource)

2018-12-22 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-26285: - Assignee: Alessandro Bellina > Add a metric source for accumulators (aka

[jira] [Resolved] (SPARK-26285) Add a metric source for accumulators (aka AccumulatorSource)

2018-12-22 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-26285. --- Resolution: Fixed Fix Version/s: 3.0.0 > Add a metric source for accumulators (aka

[jira] [Updated] (SPARK-26269) YarnAllocator should have same blacklist behaviour with YARN to maxmize use of cluster resource

2018-12-22 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-26269: -- Issue Type: Bug (was: Improvement) > YarnAllocator should have same blacklist behaviour with

[jira] [Assigned] (SPARK-26269) YarnAllocator should have same blacklist behaviour with YARN to maxmize use of cluster resource

2018-12-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-26269: - Assignee: wuyi > YarnAllocator should have same blacklist behaviour with YARN to

[jira] [Resolved] (SPARK-26269) YarnAllocator should have same blacklist behaviour with YARN to maxmize use of cluster resource

2018-12-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-26269. --- Resolution: Fixed Fix Version/s: (was: 2.4.0) 3.0.0 >

[jira] [Resolved] (SPARK-26201) python broadcast.value on driver fails with disk encryption enabled

2018-11-30 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-26201. --- Resolution: Fixed Assignee: Sanket Chintapalli Fix Version/s: 3.0.0

[jira] [Commented] (SPARK-26201) python broadcast.value on driver fails with disk encryption enabled

2018-11-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702028#comment-16702028 ] Thomas Graves commented on SPARK-26201: --- the issue here seems to be that it isn't decrypting the

[jira] [Created] (SPARK-26201) python broadcast.value on driver fails with disk encryption enabled

2018-11-28 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-26201: - Summary: python broadcast.value on driver fails with disk encryption enabled Key: SPARK-26201 URL: https://issues.apache.org/jira/browse/SPARK-26201 Project: Spark

[jira] [Commented] (YARN-9055) Capacity Scheduler: allow larger queue level maximum-allocation-mb to override the cluster configuration

2018-11-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/YARN-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700511#comment-16700511 ] Thomas Graves commented on YARN-9055: - It would definitely be a change in behavior which could

[jira] [Commented] (SPARK-26089) Handle large corrupt shuffle blocks

2018-11-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700493#comment-16700493 ] Thomas Graves commented on SPARK-26089: --- Yeah that seems to make sense and I wouldn't think would

[jira] [Commented] (SPARK-26089) Handle large corrupt shuffle blocks

2018-11-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699639#comment-16699639 ] Thomas Graves commented on SPARK-26089: --- it would definitely be nice to improve blacklisting,

[jira] [Resolved] (SPARK-21809) Change Stage Page to use datatables to support sorting columns and searching

2018-11-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21809. --- Resolution: Fixed Assignee: Parth Gandhi Fix Version/s: 3.0.0 > Change

[jira] [Commented] (YARN-8991) nodemanager not cleaning blockmgr directories inside appcache

2018-11-12 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683946#comment-16683946 ] Thomas Graves commented on YARN-8991: - if its while its running then you should file this with Spark.

[jira] [Commented] (SPARK-25995) sparkR should ensure user args are after the argument used for the port

2018-11-12 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683876#comment-16683876 ] Thomas Graves commented on SPARK-25995: --- I haven't looked at the details but I would say whatever

[jira] [Created] (SPARK-25995) sparkR should ensure user args are after the argument used for the port

2018-11-09 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-25995: - Summary: sparkR should ensure user args are after the argument used for the port Key: SPARK-25995 URL: https://issues.apache.org/jira/browse/SPARK-25995 Project:

[jira] [Commented] (YARN-8991) nodemanager not cleaning blockmgr directories inside appcache

2018-11-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/YARN-8991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681525#comment-16681525 ] Thomas Graves commented on YARN-8991: - [~teonadi] can you clarify here.  Are you saying its not

[jira] [Resolved] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

2018-11-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-22148. --- Resolution: Fixed Fix Version/s: 3.0.0 2.4.1 >

[jira] [Assigned] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

2018-11-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-22148: - Assignee: Dhruve Ashar > TaskSetManager.abortIfCompletelyBlacklisted should not abort

[jira] [Resolved] (SPARK-15815) Hang while enable blacklistExecutor and DynamicExecutorAllocator

2018-11-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-15815. --- Resolution: Duplicate > Hang while enable blacklistExecutor and DynamicExecutorAllocator >

[jira] [Assigned] (SPARK-25023) Clarify Spark security documentation

2018-11-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-25023: - Assignee: Thomas Graves > Clarify Spark security documentation >

[jira] [Resolved] (SPARK-25023) Clarify Spark security documentation

2018-11-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-25023. --- Resolution: Fixed Fix Version/s: 3.0.0 2.4.1 > Clarify Spark

[jira] [Commented] (SPARK-25855) Don't use Erasure Coding for event log files

2018-10-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665690#comment-16665690 ] Thomas Graves commented on SPARK-25855: --- it seems like it depends on whether you care to see the

[jira] [Resolved] (SPARK-25753) binaryFiles broken for small files

2018-10-22 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-25753. --- Resolution: Fixed Fix Version/s: 3.0.0 > binaryFiles broken for small files >

[jira] [Assigned] (SPARK-25753) binaryFiles broken for small files

2018-10-22 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-25753: - Assignee: liuxian > binaryFiles broken for small files >

[jira] [Commented] (SPARK-25692) Flaky test: ChunkFetchIntegrationSuite

2018-10-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657318#comment-16657318 ] Thomas Graves commented on SPARK-25692: --- [~redsanket] can you please take a look at this > Flaky

[jira] [Comment Edited] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651840#comment-16651840 ] Thomas Graves edited comment on SPARK-25732 at 10/16/18 2:53 PM: - sorry

[jira] [Comment Edited] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651840#comment-16651840 ] Thomas Graves edited comment on SPARK-25732 at 10/16/18 2:49 PM: - sorry

[jira] [Commented] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651840#comment-16651840 ] Thomas Graves commented on SPARK-25732: --- sorry just realized I misread the second one.  why would

[jira] [Commented] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651827#comment-16651827 ] Thomas Graves commented on SPARK-25732: --- yeah I understand the concern, we don't want to confuse

[jira] [Commented] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651657#comment-16651657 ] Thomas Graves commented on SPARK-25732: --- So like Marcelo mentioned can't you re-use the

[jira] [Commented] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal

2018-10-15 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650560#comment-16650560 ] Thomas Graves commented on SPARK-25732: --- I would much rather see Spark start to push tokens and

[jira] [Reopened] (SPARK-21809) Change Stage Page to use datatables to support sorting columns and searching

2018-10-12 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reopened SPARK-21809: --- > Change Stage Page to use datatables to support sorting columns and searching >

[jira] [Resolved] (SPARK-24851) Map a Stage ID to it's Associated Job ID in UI

2018-10-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24851. --- Resolution: Fixed Fix Version/s: 3.0.0 > Map a Stage ID to it's Associated Job ID in

[jira] [Assigned] (SPARK-24851) Map a Stage ID to it's Associated Job ID in UI

2018-10-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-24851: - Assignee: Parth Gandhi > Map a Stage ID to it's Associated Job ID in UI >

[jira] [Resolved] (SPARK-25641) Change the spark.shuffle.server.chunkFetchHandlerThreadsPercent default to 100

2018-10-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-25641. --- Resolution: Fixed Fix Version/s: 3.0.0 > Change the

[jira] [Assigned] (SPARK-25641) Change the spark.shuffle.server.chunkFetchHandlerThreadsPercent default to 100

2018-10-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-25641: - Assignee: Sanket Reddy > Change the

[jira] [Commented] (SPARK-25501) Kafka delegation token support

2018-10-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637248#comment-16637248 ] Thomas Graves commented on SPARK-25501: --- the spip title has "Structured Streaming", is there some 

[jira] [Commented] (SPARK-25501) Kafka delegation token support

2018-10-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637240#comment-16637240 ] Thomas Graves commented on SPARK-25501: --- did you post SPIP to the dev list, I didn't see it go by

[jira] [Assigned] (SPARK-18364) Expose metrics for YarnShuffleService

2018-10-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-18364: - Assignee: Marek Simunek > Expose metrics for YarnShuffleService >

[jira] [Resolved] (SPARK-18364) Expose metrics for YarnShuffleService

2018-10-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-18364. --- Resolution: Fixed Fix Version/s: 2.5.0 > Expose metrics for YarnShuffleService >

[jira] [Updated] (SPARK-25538) incorrect row counts after distinct()

2018-10-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-25538: -- Priority: Blocker (was: Major) > incorrect row counts after distinct() >

[jira] [Assigned] (SPARK-24355) Improve Spark shuffle server responsiveness to non-ChunkFetch requests

2018-09-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-24355: - Assignee: Sanket Chintapalli > Improve Spark shuffle server responsiveness to

[jira] [Commented] (SPARK-24355) Improve Spark shuffle server responsiveness to non-ChunkFetch requests

2018-09-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623660#comment-16623660 ] Thomas Graves commented on SPARK-24355: --- pr that got merged didn't get linked properly: 

[jira] [Resolved] (SPARK-24355) Improve Spark shuffle server responsiveness to non-ChunkFetch requests

2018-09-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24355. --- Resolution: Fixed Fix Version/s: 2.5.0 > Improve Spark shuffle server responsiveness

[jira] [Updated] (SPARK-24415) Stage page aggregated executor metrics wrong when failures

2018-09-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24415: -- Fix Version/s: 2.3.2 > Stage page aggregated executor metrics wrong when failures >

[jira] [Resolved] (SPARK-25231) Running a Large Job with Speculation On Causes Executor Heartbeats to Time Out on Driver

2018-09-05 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-25231. --- Resolution: Fixed Assignee: Parth Gandhi Fix Version/s: 2.4.0

[jira] [Updated] (SPARK-24909) Spark scheduler can hang when fetch failures, executor lost, task running on lost executor, and multiple stage attempts

2018-08-30 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24909: -- Fix Version/s: 2.3.2 > Spark scheduler can hang when fetch failures, executor lost, task

[jira] [Created] (SPARK-25263) Add scheduler integration test for SPARK-24909

2018-08-28 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-25263: - Summary: Add scheduler integration test for SPARK-24909 Key: SPARK-25263 URL: https://issues.apache.org/jira/browse/SPARK-25263 Project: Spark Issue Type:

[jira] [Commented] (SPARK-25250) Race condition with tasks running when new attempt for same stage is created leads to other task in the next attempt running on the same partition id retry multiple ti

2018-08-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594281#comment-16594281 ] Thomas Graves commented on SPARK-25250: --- We are hitting a race condition here between the

<    1   2   3   4   5   6   7   8   9   10   >