[jira] [Resolved] (SPARK-27492) GPU scheduling - High level user documentation

2019-09-11 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27492. --- Fix Version/s: 3.0.0 Resolution: Fixed > GPU scheduling - High level user

[jira] [Commented] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-09-11 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927558#comment-16927558 ] Thomas Graves commented on SPARK-27495: --- [~felixcheung]  [~jiangxb1987]  I put this up for vote on

[jira] [Commented] (SPARK-27492) GPU scheduling - High level user documentation

2019-09-05 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923528#comment-16923528 ] Thomas Graves commented on SPARK-27492: --- I think we talked about this a big in the prs for the

[jira] [Assigned] (SPARK-27492) GPU scheduling - High level user documentation

2019-09-04 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-27492: - Assignee: Thomas Graves > GPU scheduling - High level user documentation >

[jira] [Updated] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-09-04 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27495: -- Description: *Q1.* What are you trying to do? Articulate your objectives using absolutely no

[jira] [Resolved] (SPARK-28577) Ensure executorMemoryHead requested value not less than MEMORY_OFFHEAP_SIZE when MEMORY_OFFHEAP_ENABLED is true

2019-09-04 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-28577. --- Fix Version/s: 3.0.0 Assignee: Yang Jie Resolution: Fixed > Ensure

[jira] [Resolved] (SPARK-28960) maven 3.6.1 gone from apache maven repo

2019-09-03 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-28960. --- Resolution: Duplicate > maven 3.6.1 gone from apache maven repo >

[jira] [Updated] (SPARK-28961) Upgrade Maven to 3.6.2

2019-09-03 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-28961: -- Description: It looks like maven 3.6.1 is missing from the apache maven repo:

[jira] [Commented] (SPARK-28960) maven 3.6.1 gone from apache maven repo

2019-09-03 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921611#comment-16921611 ] Thomas Graves commented on SPARK-28960: --- its a bit odd I don't see 3.6.2 on their page as released

[jira] [Commented] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-09-03 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921608#comment-16921608 ] Thomas Graves commented on SPARK-27495: --- Thanks for the feedback [~felixcheung] 1.   yes this is

[jira] [Updated] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-09-03 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27495: -- Description: *Q1.* What are you trying to do? Articulate your objectives using absolutely no

[jira] [Created] (SPARK-28960) maven 3.6.1 gone from apache maven repo

2019-09-03 Thread Thomas Graves (Jira)
Thomas Graves created SPARK-28960: - Summary: maven 3.6.1 gone from apache maven repo Key: SPARK-28960 URL: https://issues.apache.org/jira/browse/SPARK-28960 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-28917) Jobs can hang because of race of RDD.dependencies

2019-09-03 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921424#comment-16921424 ] Thomas Graves commented on SPARK-28917: --- So for this to happen, you essentially have to have that

[jira] [Resolved] (SPARK-27360) Standalone cluster mode support for GPU-aware scheduling

2019-08-27 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27360. --- Fix Version/s: 3.0.0 Resolution: Fixed all subtasks are finished so resolving >

[jira] [Resolved] (SPARK-28414) Standalone worker/master UI updates for Resource scheduling

2019-08-27 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-28414. --- Fix Version/s: 3.0.0 Assignee: wuyi Resolution: Fixed > Standalone

[jira] [Updated] (SPARK-28577) Ensure executorMemoryHead requested value not less than MEMORY_OFFHEAP_SIZE when MEMORY_OFFHEAP_ENABLED is true

2019-08-22 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-28577: -- Docs Text: On YARN, The off heap memory size is now separately included for the user in the

[jira] [Updated] (SPARK-28577) Ensure executorMemoryHead requested value not less than MEMORY_OFFHEAP_SIZE when MEMORY_OFFHEAP_ENABLED is true

2019-08-22 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-28577: -- Labels: release-notes (was: ) > Ensure executorMemoryHead requested value not less than

[jira] [Commented] (SPARK-27361) YARN support for GPU-aware scheduling

2019-08-14 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907431#comment-16907431 ] Thomas Graves commented on SPARK-27361: --- Since that was done prior to this feature, I think its ok

[jira] [Commented] (SPARK-23151) Provide a distribution of Spark with Hadoop 3.0

2019-08-13 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906218#comment-16906218 ] Thomas Graves commented on SPARK-23151: --- Hey so for the hadoop-3.2 profile, the dependencies are

[jira] [Created] (SPARK-28679) Spark Yarn ResourceRequestHelper shouldn't lookup setResourceInformation is no resources specified

2019-08-09 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-28679: - Summary: Spark Yarn ResourceRequestHelper shouldn't lookup setResourceInformation is no resources specified Key: SPARK-28679 URL:

[jira] [Assigned] (SPARK-27371) Standalone master receives resource info from worker and allocate driver/executor properly

2019-08-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-27371: - Assignee: wuyi > Standalone master receives resource info from worker and allocate >

[jira] [Resolved] (SPARK-27368) Design: Standalone supports GPU scheduling

2019-08-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27368. --- Resolution: Fixed resolving since code committed > Design: Standalone supports GPU

[jira] [Resolved] (SPARK-27371) Standalone master receives resource info from worker and allocate driver/executor properly

2019-08-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27371. --- Resolution: Fixed Fix Version/s: 3.0.0 > Standalone master receives resource info

[jira] [Updated] (SPARK-27492) GPU scheduling - High level user documentation

2019-08-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27492: -- Description: For the SPIP - Accelerator-aware task scheduling for Spark, 

[jira] [Updated] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-08-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27495: -- Description: *Q1.* What are you trying to do? Articulate your objectives using absolutely no

[jira] [Updated] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-08-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27495: -- Description: *Q1.* What are you trying to do? Articulate your objectives using absolutely no

[jira] [Commented] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-08-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902076#comment-16902076 ] Thomas Graves commented on SPARK-27495: --- Thanks for the comments. # I was trying to leave the API

[jira] [Assigned] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-08-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-27495: - Assignee: Thomas Graves > SPIP: Support Stage level resource configuration and

[jira] [Commented] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-08-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901253#comment-16901253 ] Thomas Graves commented on SPARK-27495: --- I updated the Jira to have the SPIP text in description.

[jira] [Updated] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-08-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27495: -- Description: *Q1.* What are you trying to do? Articulate your objectives using absolutely no

[jira] [Updated] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-08-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27495: -- Description: *Q1.* What are you trying to do? Articulate your objectives using absolutely no

[jira] [Updated] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-08-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27495: -- Issue Type: Epic (was: Story) > SPIP: Support Stage level resource configuration and

[jira] [Updated] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-07-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27495: -- Summary: SPIP: Support Stage level resource configuration and scheduling (was: Support Stage

[jira] [Created] (SPARK-28414) Standalone worker/master UI updates for Resource scheduling

2019-07-16 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-28414: - Summary: Standalone worker/master UI updates for Resource scheduling Key: SPARK-28414 URL: https://issues.apache.org/jira/browse/SPARK-28414 Project: Spark

[jira] [Created] (SPARK-28403) Executor Allocation Manager can add an extra executor when speculative tasks

2019-07-15 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-28403: - Summary: Executor Allocation Manager can add an extra executor when speculative tasks Key: SPARK-28403 URL: https://issues.apache.org/jira/browse/SPARK-28403

[jira] [Assigned] (SPARK-28213) Remove duplication between columnar and ColumnarBatchScan

2019-07-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-28213: - Assignee: Robert Joseph Evans > Remove duplication between columnar and

[jira] [Resolved] (SPARK-28213) Remove duplication between columnar and ColumnarBatchScan

2019-07-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-28213. --- Resolution: Fixed Fix Version/s: 3.0.0 > Remove duplication between columnar and

[jira] [Commented] (SPARK-24615) SPIP: Accelerator-aware task scheduling for Spark

2019-07-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882954#comment-16882954 ] Thomas Graves commented on SPARK-24615: --- Hey [~jomach], thanks for the offer, you can find SPIP

[jira] [Commented] (SPARK-28234) Spark Resources - add python support to get resources

2019-07-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882092#comment-16882092 ] Thomas Graves commented on SPARK-28234: --- Testing driver side: {code:java} >>>

[jira] [Assigned] (SPARK-28234) Spark Resources - add python support to get resources

2019-07-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-28234: - Assignee: Thomas Graves > Spark Resources - add python support to get resources >

[jira] [Created] (SPARK-28234) Spark Resources - add python support to get resources

2019-07-02 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-28234: - Summary: Spark Resources - add python support to get resources Key: SPARK-28234 URL: https://issues.apache.org/jira/browse/SPARK-28234 Project: Spark

[jira] [Commented] (SPARK-27360) Standalone cluster mode support for GPU-aware scheduling

2019-07-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877086#comment-16877086 ] Thomas Graves commented on SPARK-27360: --- Are you going to handle updating the master/worker UI to

[jira] [Assigned] (SPARK-27945) Make minimal changes to support columnar processing

2019-06-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-27945: - Assignee: Robert Joseph Evans > Make minimal changes to support columnar processing >

[jira] [Resolved] (SPARK-27945) Make minimal changes to support columnar processing

2019-06-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27945. --- Resolution: Fixed Fix Version/s: 3.0.0 > Make minimal changes to support columnar

[jira] [Commented] (SPARK-28116) Fix Flaky `SparkContextSuite.test resource scheduling under local-cluster mode`

2019-06-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869803#comment-16869803 ] Thomas Graves commented on SPARK-28116: --- Sorry for my delay on this, I'll take a look. haven't

[jira] [Assigned] (SPARK-27823) Add an abstraction layer for accelerator resource handling to avoid manipulating raw confs

2019-06-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-27823: - Assignee: Thomas Graves > Add an abstraction layer for accelerator resource handling

[jira] [Resolved] (SPARK-27760) Spark resources - user configs change .count to be .amount

2019-06-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27760. --- Resolution: Fixed Fix Version/s: 3.0.0 > Spark resources - user configs change

[jira] [Assigned] (SPARK-24615) SPIP: Accelerator-aware task scheduling for Spark

2019-06-05 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-24615: - Assignee: Thomas Graves (was: Xingbo Jiang) > SPIP: Accelerator-aware task scheduling

[jira] [Updated] (SPARK-27760) Spark resources - user configs change .count to be .amount

2019-06-05 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27760: -- Description: For the Spark resources, we created the config

[jira] [Updated] (SPARK-27760) Spark resources - user configs change .count to be .amount

2019-06-05 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27760: -- Description: For the Spark resources, we created the config

[jira] [Assigned] (SPARK-27760) Spark resources - user configs change .count to be .amount

2019-06-05 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-27760: - Assignee: Thomas Graves > Spark resources - user configs change .count to be .amount >

[jira] [Updated] (SPARK-27760) Spark resources - user configs change .count to be .amount

2019-06-05 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27760: -- Summary: Spark resources - user configs change .count to be .amount (was: Spark resources -

[jira] [Created] (SPARK-27959) Change YARN resource configs to use .amount

2019-06-05 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-27959: - Summary: Change YARN resource configs to use .amount Key: SPARK-27959 URL: https://issues.apache.org/jira/browse/SPARK-27959 Project: Spark Issue Type:

[jira] [Updated] (SPARK-27760) Spark resources - user configs change .count to be .amount, and yarn configs should match

2019-06-05 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27760: -- Description: For the Spark resources, we created the config

[jira] [Resolved] (SPARK-27364) User-facing APIs for GPU-aware scheduling

2019-06-05 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27364. --- Resolution: Fixed Fix Version/s: 3.0.0 > User-facing APIs for GPU-aware scheduling >

[jira] [Commented] (SPARK-27364) User-facing APIs for GPU-aware scheduling

2019-06-05 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16856895#comment-16856895 ] Thomas Graves commented on SPARK-27364: --- User facing changes are all committed so going to close

[jira] [Resolved] (SPARK-27374) Fetch assigned resources from TaskContext

2019-05-31 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27374. --- Resolution: Duplicate > Fetch assigned resources from TaskContext >

[jira] [Resolved] (SPARK-27362) Kubernetes support for GPU-aware scheduling

2019-05-31 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27362. --- Resolution: Fixed Fix Version/s: 3.0.0 > Kubernetes support for GPU-aware scheduling

[jira] [Resolved] (SPARK-27373) Design: Kubernetes support for GPU-aware scheduling

2019-05-31 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27373. --- Resolution: Fixed Fix Version/s: 3.0.0 > Design: Kubernetes support for GPU-aware

[jira] [Resolved] (SPARK-27897) GPU Scheduling - move example discovery Script to scripts directory

2019-05-31 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27897. --- Resolution: Fixed Fix Version/s: 3.0.0 > GPU Scheduling - move example discovery

[jira] [Created] (SPARK-27897) GPU Scheduling - move example discovery Script to scripts directory

2019-05-31 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-27897: - Summary: GPU Scheduling - move example discovery Script to scripts directory Key: SPARK-27897 URL: https://issues.apache.org/jira/browse/SPARK-27897 Project: Spark

[jira] [Resolved] (SPARK-27361) YARN support for GPU-aware scheduling

2019-05-30 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27361. --- Resolution: Fixed Fix Version/s: 3.0.0 all the subtasks are finished and the parts

[jira] [Resolved] (SPARK-27835) Resource Scheduling: change driver config from addresses to resourcesFile

2019-05-30 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27835. --- Resolution: Fixed Fix Version/s: 3.0.0 > Resource Scheduling: change driver config

[jira] [Commented] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

2019-05-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851445#comment-16851445 ] Thomas Graves commented on SPARK-27396: --- The vote passed. > SPIP: Public APIs for extended

[jira] [Resolved] (SPARK-27376) Design: YARN supports Spark GPU-aware scheduling

2019-05-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27376. --- Resolution: Fixed resolving this since seems no objections > Design: YARN supports Spark

[jira] [Commented] (SPARK-27361) YARN support for GPU-aware scheduling

2019-05-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851298#comment-16851298 ] Thomas Graves commented on SPARK-27361: --- Note the design Jira covers more details. > YARN support

[jira] [Updated] (SPARK-27361) YARN support for GPU-aware scheduling

2019-05-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27361: -- Description: Design and implement YARN support for GPU-aware scheduling: * User can request

[jira] [Assigned] (SPARK-27362) Kubernetes support for GPU-aware scheduling

2019-05-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-27362: - Assignee: Thomas Graves > Kubernetes support for GPU-aware scheduling >

[jira] [Assigned] (SPARK-27725) GPU Scheduling - add an example discovery Script

2019-05-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-27725: - Assignee: Thomas Graves > GPU Scheduling - add an example discovery Script >

[jira] [Assigned] (SPARK-27835) Resource Scheduling: change driver config from addresses to resourcesFile

2019-05-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-27835: - Assignee: Thomas Graves > Resource Scheduling: change driver config from addresses to

[jira] [Created] (SPARK-27835) Resource Scheduling: change driver config from addresses to resourcesFile

2019-05-24 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-27835: - Summary: Resource Scheduling: change driver config from addresses to resourcesFile Key: SPARK-27835 URL: https://issues.apache.org/jira/browse/SPARK-27835 Project:

[jira] [Commented] (SPARK-27495) Support Stage level resource configuration and scheduling

2019-05-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16845219#comment-16845219 ] Thomas Graves commented on SPARK-27495: --- I'm working though the design of this and there are

[jira] [Commented] (SPARK-26989) Flaky test:DAGSchedulerSuite.Barrier task failures from the same stage attempt don't trigger multiple stage retries

2019-05-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844845#comment-16844845 ] Thomas Graves commented on SPARK-26989: --- seeing the same error intermittently, the weird thing is

[jira] [Commented] (SPARK-27374) Fetch assigned resources from TaskContext

2019-05-20 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844067#comment-16844067 ] Thomas Graves commented on SPARK-27374: --- I believe this is being done under:

[jira] [Commented] (SPARK-27725) GPU Scheduling - add an example discovery Script

2019-05-20 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844066#comment-16844066 ] Thomas Graves commented on SPARK-27725: --- the above sed command doesn't quite handle it, here is a

[jira] [Commented] (SPARK-27376) Design: YARN supports Spark GPU-aware scheduling

2019-05-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842262#comment-16842262 ] Thomas Graves commented on SPARK-27376: --- I filed https://issues.apache.org/jira/browse/SPARK-27760

[jira] [Created] (SPARK-27760) Spark resources - user configs change .count to be .amount, and yarn configs should match

2019-05-17 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-27760: - Summary: Spark resources - user configs change .count to be .amount, and yarn configs should match Key: SPARK-27760 URL: https://issues.apache.org/jira/browse/SPARK-27760

[jira] [Commented] (SPARK-27736) Improve handling of FetchFailures caused by ExternalShuffleService losing track of executor registrations

2019-05-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841614#comment-16841614 ] Thomas Graves commented on SPARK-27736: --- to clarify my last suggestion, I mean each executor

[jira] [Commented] (SPARK-27736) Improve handling of FetchFailures caused by ExternalShuffleService losing track of executor registrations

2019-05-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841593#comment-16841593 ] Thomas Graves commented on SPARK-27736: --- Yeah we always ran yarn with node manager recover on, but

[jira] [Comment Edited] (SPARK-27373) Design: Kubernetes support for GPU-aware scheduling

2019-05-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841506#comment-16841506 ] Thomas Graves edited comment on SPARK-27373 at 5/16/19 4:21 PM: for the

[jira] [Commented] (SPARK-27373) Design: Kubernetes support for GPU-aware scheduling

2019-05-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841506#comment-16841506 ] Thomas Graves commented on SPARK-27373: --- for the kubernetes side, it has 2 options for requesting

[jira] [Assigned] (SPARK-27373) Design: Kubernetes support for GPU-aware scheduling

2019-05-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-27373: - Assignee: Thomas Graves > Design: Kubernetes support for GPU-aware scheduling >

[jira] [Resolved] (SPARK-27377) Upgrade YARN to 3.1.2+ to support GPU

2019-05-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27377. --- Resolution: Fixed > Upgrade YARN to 3.1.2+ to support GPU >

[jira] [Commented] (SPARK-27376) Design: YARN supports Spark GPU-aware scheduling

2019-05-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841404#comment-16841404 ] Thomas Graves commented on SPARK-27376: --- [~mengxr] [~jiangxb] Thoughts on my proposal above to

[jira] [Commented] (SPARK-27378) spark-submit requests GPUs in YARN mode

2019-05-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841412#comment-16841412 ] Thomas Graves commented on SPARK-27378: --- Spark 3.0 already added support for requesting any

[jira] [Commented] (SPARK-27379) YARN passes GPU info to Spark executor

2019-05-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841409#comment-16841409 ] Thomas Graves commented on SPARK-27379: --- The way yarn works is it actually doesn't tell the

[jira] [Resolved] (SPARK-27379) YARN passes GPU info to Spark executor

2019-05-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27379. --- Resolution: Invalid Assignee: Thomas Graves > YARN passes GPU info to Spark executor

[jira] [Commented] (SPARK-27377) Upgrade YARN to 3.1.2+ to support GPU

2019-05-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841407#comment-16841407 ] Thomas Graves commented on SPARK-27377: --- there are enough pieces in the hadoop 3.2 support

[jira] [Assigned] (SPARK-27376) Design: YARN supports Spark GPU-aware scheduling

2019-05-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-27376: - Assignee: Thomas Graves > Design: YARN supports Spark GPU-aware scheduling >

[jira] [Commented] (SPARK-27376) Design: YARN supports Spark GPU-aware scheduling

2019-05-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841402#comment-16841402 ] Thomas Graves commented on SPARK-27376: --- The design is pretty straight forward, there is really

[jira] [Created] (SPARK-27725) GPU Scheduling - add an example discovery Script

2019-05-15 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-27725: - Summary: GPU Scheduling - add an example discovery Script Key: SPARK-27725 URL: https://issues.apache.org/jira/browse/SPARK-27725 Project: Spark Issue

[jira] [Resolved] (SPARK-27024) Executor interface for cluster managers to support GPU resources

2019-05-14 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27024. --- Resolution: Fixed > Executor interface for cluster managers to support GPU resources >

[jira] [Commented] (SPARK-27488) Driver interface to support GPU resources

2019-05-14 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839402#comment-16839402 ] Thomas Graves commented on SPARK-27488: --- With this Jira we need to move the function that does the

[jira] [Commented] (SPARK-27520) Introduce a global config system to replace hadoopConfiguration

2019-05-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836730#comment-16836730 ] Thomas Graves commented on SPARK-27520: --- Can we add more to the description to explain why we are

[jira] [Updated] (SPARK-27492) GPU scheduling - High level user documentation

2019-05-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27492: -- Description: For the SPIP - Accelerator-aware task scheduling for Spark, 

[jira] [Commented] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

2019-05-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16835014#comment-16835014 ] Thomas Graves commented on SPARK-27396: --- I just started a vote thread on this SPIP please take a

[jira] [Commented] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

2019-04-22 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823097#comment-16823097 ] Thomas Graves commented on SPARK-27396: --- thanks for the questions and commenting, please also vote

[jira] [Comment Edited] (SPARK-27495) Support Stage level resource configuration and scheduling

2019-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822166#comment-16822166 ] Thomas Graves edited comment on SPARK-27495 at 4/19/19 8:29 PM:

[jira] [Comment Edited] (SPARK-27495) Support Stage level resource configuration and scheduling

2019-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822166#comment-16822166 ] Thomas Graves edited comment on SPARK-27495 at 4/19/19 8:28 PM:

[jira] [Commented] (SPARK-27495) Support Stage level resource configuration and scheduling

2019-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822166#comment-16822166 ] Thomas Graves commented on SPARK-27495: --- Unfortunately the link to the original design doc was

<    1   2   3   4   5   6   7   8   9   10   >