[jira] [Updated] (SPARK-30512) Use a dedicated boss event group loop in the netty pipeline for external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-30512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-30512: -- Fix Version/s: 2.4.5 > Use a dedicated boss event group loop in the netty pipeline for

[jira] [Assigned] (SPARK-30512) Use a dedicated boss event group loop in the netty pipeline for external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-30512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-30512: - Assignee: Chandni Singh > Use a dedicated boss event group loop in the netty pipeline

[jira] [Resolved] (SPARK-30512) Use a dedicated boss event group loop in the netty pipeline for external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-30512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-30512. --- Fix Version/s: 3.0.0 Resolution: Fixed this could be pulled back into branch-2.X as

[jira] [Created] (SPARK-30638) add resources as parameter to the PluginContext

Thomas Graves created SPARK-30638: - Summary: add resources as parameter to the PluginContext Key: SPARK-30638 URL: https://issues.apache.org/jira/browse/SPARK-30638 Project: Spark Issue

[jira] [Commented] (SPARK-28403) Executor Allocation Manager can add an extra executor when speculative tasks

[ https://issues.apache.org/jira/browse/SPARK-28403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022527#comment-17022527 ] Thomas Graves commented on SPARK-28403: --- so after looking at the pr for this, this logic may have

[jira] [Created] (SPARK-30623) Spark external shuffle allow disable of separate event loop group

Thomas Graves created SPARK-30623: - Summary: Spark external shuffle allow disable of separate event loop group Key: SPARK-30623 URL: https://issues.apache.org/jira/browse/SPARK-30623 Project: Spark

[jira] [Commented] (SPARK-27213) Unexpected results when filter is used after distinct

[ https://issues.apache.org/jira/browse/SPARK-27213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021532#comment-17021532 ] Thomas Graves commented on SPARK-27213: --- did anyone try this on the latest 2.4.x release? >

[jira] [Commented] (SPARK-29699) Different answers in nested aggregates with window functions

[ https://issues.apache.org/jira/browse/SPARK-29699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021404#comment-17021404 ] Thomas Graves commented on SPARK-29699: --- this seems to be in the context of feature parity with

[jira] [Commented] (SPARK-27082) Dynamic Allocation: we should consider the scenario that speculative task being killed and never resubmit

[ https://issues.apache.org/jira/browse/SPARK-27082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021284#comment-17021284 ] Thomas Graves commented on SPARK-27082: --- I ran into this case while testing, I think when you say

[jira] [Commented] (SPARK-27784) Alias ID reuse can break correctness when substituting foldable expressions

[ https://issues.apache.org/jira/browse/SPARK-27784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020563#comment-17020563 ] Thomas Graves commented on SPARK-27784: --- [~rdblue] can you confirm this doesn't exist in master?

[jira] [Commented] (SPARK-27282) Spark incorrect results when using UNION with GROUP BY clause

[ https://issues.apache.org/jira/browse/SPARK-27282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020560#comment-17020560 ] Thomas Graves commented on SPARK-27282: --- [~sofia] can you confirm this isn't fixed in the last

[jira] [Resolved] (SPARK-29306) Executors need to track what ResourceProfile they are created with

[ https://issues.apache.org/jira/browse/SPARK-29306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-29306. --- Fix Version/s: 3.0.0 Resolution: Fixed > Executors need to track what

[jira] [Created] (SPARK-30529) Improve error messages when Executor dies before registering with driver

Thomas Graves created SPARK-30529: - Summary: Improve error messages when Executor dies before registering with driver Key: SPARK-30529 URL: https://issues.apache.org/jira/browse/SPARK-30529 Project:

[jira] [Assigned] (SPARK-30448) accelerator aware scheduling enforce cores as limiting resource

[ https://issues.apache.org/jira/browse/SPARK-30448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-30448: - Assignee: Thomas Graves > accelerator aware scheduling enforce cores as limiting

[jira] [Resolved] (SPARK-30448) accelerator aware scheduling enforce cores as limiting resource

[ https://issues.apache.org/jira/browse/SPARK-30448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-30448. --- Fix Version/s: 3.0.0 Resolution: Fixed > accelerator aware scheduling enforce cores

[jira] [Resolved] (SPARK-30446) Accelerator aware scheduling checkResourcesPerTask doesn't cover all cases

[ https://issues.apache.org/jira/browse/SPARK-30446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-30446. --- Resolution: Duplicate this is superceded by SPARK-30448 > Accelerator aware scheduling

[jira] [Commented] (SPARK-30049) SQL fails to parse when comment contains an unmatched quote character

[ https://issues.apache.org/jira/browse/SPARK-30049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010698#comment-17010698 ] Thomas Graves commented on SPARK-30049: --- that Pr doesn't seem to be in yet. I'm not sure the

[jira] [Commented] (SPARK-30295) Remove Hive dependencies from SparkSQLCLI

[ https://issues.apache.org/jira/browse/SPARK-30295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010697#comment-17010697 ] Thomas Graves commented on SPARK-30295: --- So what do you mean by unnecessary dependencies?  I

[jira] [Commented] (SPARK-30417) SPARK-29976 calculation of slots wrong for Standalone Mode

[ https://issues.apache.org/jira/browse/SPARK-30417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010146#comment-17010146 ] Thomas Graves commented on SPARK-30417: --- The only way for standalone mode would be to look at what

[jira] [Created] (SPARK-30451) Stage level Sched: ExecutorResourceRequests/TaskResourceRequests should have functions to remove requests

Thomas Graves created SPARK-30451: - Summary: Stage level Sched: ExecutorResourceRequests/TaskResourceRequests should have functions to remove requests Key: SPARK-30451 URL:

[jira] [Commented] (SPARK-30448) accelerator aware scheduling enforce cores as limiting resource

[ https://issues.apache.org/jira/browse/SPARK-30448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009989#comment-17009989 ] Thomas Graves commented on SPARK-30448: --- Note this actually overlaps with

[jira] [Commented] (SPARK-30448) accelerator aware scheduling enforce cores as limiting resource

[ https://issues.apache.org/jira/browse/SPARK-30448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009901#comment-17009901 ] Thomas Graves commented on SPARK-30448: --- note there are other calculations throughout spark code

[jira] [Commented] (SPARK-30446) Accelerator aware scheduling checkResourcesPerTask doesn't cover all cases

[ https://issues.apache.org/jira/browse/SPARK-30446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009896#comment-17009896 ] Thomas Graves commented on SPARK-30446: --- working on this > Accelerator aware scheduling

[jira] [Created] (SPARK-30448) accelerator aware scheduling enforce cores as limiting resource

Thomas Graves created SPARK-30448: - Summary: accelerator aware scheduling enforce cores as limiting resource Key: SPARK-30448 URL: https://issues.apache.org/jira/browse/SPARK-30448 Project: Spark

[jira] [Commented] (SPARK-30446) Accelerator aware scheduling checkResourcesPerTask doesn't cover all cases

[ https://issues.apache.org/jira/browse/SPARK-30446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009830#comment-17009830 ] Thomas Graves commented on SPARK-30446: --- Yeah so running on standalone if you set the

[jira] [Commented] (SPARK-30446) Accelerator aware scheduling checkResourcesPerTask doesn't cover all cases

[ https://issues.apache.org/jira/browse/SPARK-30446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009742#comment-17009742 ] Thomas Graves commented on SPARK-30446: --- I think there may also be issues in it with standalone

[jira] [Created] (SPARK-30446) Accelerator aware scheduling checkResourcesPerTask doesn't cover all cases

Thomas Graves created SPARK-30446: - Summary: Accelerator aware scheduling checkResourcesPerTask doesn't cover all cases Key: SPARK-30446 URL: https://issues.apache.org/jira/browse/SPARK-30446

[jira] [Created] (SPARK-30445) Accelerator aware scheduling handle setting configs to 0 better

Thomas Graves created SPARK-30445: - Summary: Accelerator aware scheduling handle setting configs to 0 better Key: SPARK-30445 URL: https://issues.apache.org/jira/browse/SPARK-30445 Project: Spark

[jira] [Commented] (SPARK-30417) SPARK-29976 calculation of slots wrong for Standalone Mode

[ https://issues.apache.org/jira/browse/SPARK-30417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007694#comment-17007694 ] Thomas Graves commented on SPARK-30417: --- [~yuchen.huo] is this something you could work on? >

[jira] [Created] (SPARK-30417) SPARK-29976 calculation of slots wrong for Standalone Mode

Thomas Graves created SPARK-30417: - Summary: SPARK-29976 calculation of slots wrong for Standalone Mode Key: SPARK-30417 URL: https://issues.apache.org/jira/browse/SPARK-30417 Project: Spark

[jira] [Updated] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27495: -- Target Version/s: 3.0.0 > SPIP: Support Stage level resource configuration and scheduling >

[jira] [Created] (SPARK-30391) ExecutorAllocationManager requestTotalExecutors in removeExecutors may not be needed

Thomas Graves created SPARK-30391: - Summary: ExecutorAllocationManager requestTotalExecutors in removeExecutors may not be needed Key: SPARK-30391 URL: https://issues.apache.org/jira/browse/SPARK-30391

[jira] [Created] (SPARK-30322) Add stage level scheduling docs

Thomas Graves created SPARK-30322: - Summary: Add stage level scheduling docs Key: SPARK-30322 URL: https://issues.apache.org/jira/browse/SPARK-30322 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-30300) Update correct string in UI for metrics when driver updates same metrics id as tasks.

[ https://issues.apache.org/jira/browse/SPARK-30300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-30300. --- Fix Version/s: 3.0.0 Assignee: Niranjan Artal Resolution: Fixed > Update

[jira] [Created] (SPARK-30299) Dynamic allocation with Standalone mode calculates to many executors needed

Thomas Graves created SPARK-30299: - Summary: Dynamic allocation with Standalone mode calculates to many executors needed Key: SPARK-30299 URL: https://issues.apache.org/jira/browse/SPARK-30299

[jira] [Assigned] (SPARK-30209) Display stageId, attemptId, taskId with SQL max metric in UI

[ https://issues.apache.org/jira/browse/SPARK-30209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-30209: - Assignee: Niranjan Artal > Display stageId, attemptId, taskId with SQL max metric in

[jira] [Resolved] (SPARK-30209) Display stageId, attemptId, taskId with SQL max metric in UI

[ https://issues.apache.org/jira/browse/SPARK-30209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-30209. --- Fix Version/s: 3.0.0 Resolution: Fixed > Display stageId, attemptId, taskId with SQL

[jira] [Resolved] (SPARK-29976) Allow speculation even if there is only one task

[ https://issues.apache.org/jira/browse/SPARK-29976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-29976. --- Fix Version/s: 3.0.0 Assignee: Yuchen Huo Resolution: Fixed > Allow

[jira] [Commented] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987037#comment-16987037 ] Thomas Graves commented on SPARK-18886: --- Note there is discussion on this subject on prs:

[jira] [Resolved] (SPARK-29415) Stage Level Sched: Add base ResourceProfile and Request classes

[ https://issues.apache.org/jira/browse/SPARK-29415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-29415. --- Fix Version/s: 3.0.0 Resolution: Fixed > Stage Level Sched: Add base ResourceProfile

[jira] [Assigned] (SPARK-29149) Update YARN cluster manager For Stage Level Scheduling

[ https://issues.apache.org/jira/browse/SPARK-29149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-29149: - Assignee: Thomas Graves > Update YARN cluster manager For Stage Level Scheduling >

[jira] [Commented] (SPARK-29762) GPU Scheduling - default task resource amount to 1

[ https://issues.apache.org/jira/browse/SPARK-29762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972755#comment-16972755 ] Thomas Graves commented on SPARK-29762: --- But like you mention, if we were to support something

[jira] [Commented] (SPARK-29762) GPU Scheduling - default task resource amount to 1

[ https://issues.apache.org/jira/browse/SPARK-29762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972750#comment-16972750 ] Thomas Graves commented on SPARK-29762: --- Ok, in the scenario above you would need 2 different

[jira] [Commented] (SPARK-29762) GPU Scheduling - default task resource amount to 1

[ https://issues.apache.org/jira/browse/SPARK-29762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972661#comment-16972661 ] Thomas Graves commented on SPARK-29762: --- The code that gets the task requirements is generic and

[jira] [Commented] (SPARK-29762) GPU Scheduling - default task resource amount to 1

[ https://issues.apache.org/jira/browse/SPARK-29762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971798#comment-16971798 ] Thomas Graves commented on SPARK-29762: --- this is actually more complex then you might think

[jira] [Assigned] (SPARK-29763) Stage UI Page not showing all accumulators in Task Table

[ https://issues.apache.org/jira/browse/SPARK-29763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-29763: - Assignee: Thomas Graves > Stage UI Page not showing all accumulators in Task Table >

[jira] [Created] (SPARK-29763) Stage UI Page not showing all accumulators in Task Table

Thomas Graves created SPARK-29763: - Summary: Stage UI Page not showing all accumulators in Task Table Key: SPARK-29763 URL: https://issues.apache.org/jira/browse/SPARK-29763 Project: Spark

[jira] [Resolved] (SPARK-29151) Support fraction resources for task resource scheduling

[ https://issues.apache.org/jira/browse/SPARK-29151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-29151. --- Fix Version/s: 3.0.0 Resolution: Fixed > Support fraction resources for task

[jira] [Assigned] (SPARK-29151) Support fraction resources for task resource scheduling

[ https://issues.apache.org/jira/browse/SPARK-29151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-29151: - Assignee: Alessandro Bellina > Support fraction resources for task resource scheduling

[jira] [Created] (SPARK-29762) GPU Scheduling - default task resource amount to 1

Thomas Graves created SPARK-29762: - Summary: GPU Scheduling - default task resource amount to 1 Key: SPARK-29762 URL: https://issues.apache.org/jira/browse/SPARK-29762 Project: Spark Issue

[jira] [Comment Edited] (SPARK-29415) Stage Level Sched: Add base ResourceProfile and Request classes

[ https://issues.apache.org/jira/browse/SPARK-29415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964360#comment-16964360 ] Thomas Graves edited comment on SPARK-29415 at 10/31/19 8:03 PM: - More

[jira] [Comment Edited] (SPARK-29415) Stage Level Sched: Add base ResourceProfile and Request classes

[ https://issues.apache.org/jira/browse/SPARK-29415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964360#comment-16964360 ] Thomas Graves edited comment on SPARK-29415 at 10/31/19 7:59 PM: - More

[jira] [Commented] (SPARK-29415) Stage Level Sched: Add base ResourceProfile and Request classes

[ https://issues.apache.org/jira/browse/SPARK-29415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964360#comment-16964360 ] Thomas Graves commented on SPARK-29415: --- More details: TaskResourceRequest - this supports taking

[jira] [Created] (SPARK-29641) Stage Level Sched: Add python api's and test

Thomas Graves created SPARK-29641: - Summary: Stage Level Sched: Add python api's and test Key: SPARK-29641 URL: https://issues.apache.org/jira/browse/SPARK-29641 Project: Spark Issue Type:

[jira] [Commented] (SPARK-29415) Stage Level Sched: Add base ResourceProfile and Request classes

[ https://issues.apache.org/jira/browse/SPARK-29415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16957974#comment-16957974 ] Thomas Graves commented on SPARK-29415: --- >From a high level design point, this is the base classes

[jira] [Commented] (SPARK-29329) maven incremental builds not working

[ https://issues.apache.org/jira/browse/SPARK-29329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956362#comment-16956362 ] Thomas Graves commented on SPARK-29329: --- from the linked issue above it sounds like the zinc 

[jira] [Updated] (SPARK-26154) Stream-stream joins - left outer join gives inconsistent output

[ https://issues.apache.org/jira/browse/SPARK-26154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-26154: -- Affects Version/s: 3.0.0 > Stream-stream joins - left outer join gives inconsistent output >

[jira] [Commented] (SPARK-29465) Unable to configure SPARK UI (spark.ui.port) in spark yarn cluster mode.

[ https://issues.apache.org/jira/browse/SPARK-29465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952617#comment-16952617 ] Thomas Graves commented on SPARK-29465: --- So I'd be fine with a range of ports. I suggest updating

[jira] [Commented] (SPARK-29465) Unable to configure SPARK UI (spark.ui.port) in spark yarn cluster mode.

[ https://issues.apache.org/jira/browse/SPARK-29465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952603#comment-16952603 ] Thomas Graves commented on SPARK-29465: --- Note the problem I see with just using the port if user

[jira] [Commented] (SPARK-29465) Unable to configure SPARK UI (spark.ui.port) in spark yarn cluster mode.

[ https://issues.apache.org/jira/browse/SPARK-29465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952542#comment-16952542 ] Thomas Graves commented on SPARK-29465: --- Thanks for copying me, yes agree this would be

[jira] [Assigned] (SPARK-29148) Modify dynamic allocation manager for stage level scheduling

[ https://issues.apache.org/jira/browse/SPARK-29148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-29148: - Assignee: Thomas Graves > Modify dynamic allocation manager for stage level scheduling

[jira] [Assigned] (SPARK-29415) Stage Level Sched: Add base ResourceProfile and Request classes

[ https://issues.apache.org/jira/browse/SPARK-29415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-29415: - Assignee: Thomas Graves > Stage Level Sched: Add base ResourceProfile and Request

[jira] [Assigned] (SPARK-29306) Executors need to track what ResourceProfile they are created with

[ https://issues.apache.org/jira/browse/SPARK-29306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-29306: - Assignee: Thomas Graves > Executors need to track what ResourceProfile they are

[jira] [Commented] (SPARK-29396) Extend Spark plugin interface to driver

[ https://issues.apache.org/jira/browse/SPARK-29396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949460#comment-16949460 ] Thomas Graves commented on SPARK-29396: ---  While I do like the idea, do we have some concrete use

[jira] [Assigned] (SPARK-29396) Extend Spark plugin interface to driver

[ https://issues.apache.org/jira/browse/SPARK-29396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-29396: - Assignee: (was: Thomas Graves) > Extend Spark plugin interface to driver >

[jira] [Assigned] (SPARK-29396) Extend Spark plugin interface to driver

[ https://issues.apache.org/jira/browse/SPARK-29396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-29396: - Assignee: Thomas Graves > Extend Spark plugin interface to driver >

[jira] [Commented] (SPARK-28859) Remove value check of MEMORY_OFFHEAP_SIZE in declaration section

[ https://issues.apache.org/jira/browse/SPARK-28859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948652#comment-16948652 ] Thomas Graves commented on SPARK-28859: --- I wouldn't expect users to specify the size when enabled

[jira] [Created] (SPARK-29417) Resource Scheduling - add TaskContext.resource java api

Thomas Graves created SPARK-29417: - Summary: Resource Scheduling - add TaskContext.resource java api Key: SPARK-29417 URL: https://issues.apache.org/jira/browse/SPARK-29417 Project: Spark

[jira] [Commented] (SPARK-28859) Remove value check of MEMORY_OFFHEAP_SIZE in declaration section

[ https://issues.apache.org/jira/browse/SPARK-28859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16947908#comment-16947908 ] Thomas Graves commented on SPARK-28859: --- The comment came from the pr you linked to, but was

[jira] [Created] (SPARK-29415) Stage Level Sched: Add base ResourceProfile and Request classes

Thomas Graves created SPARK-29415: - Summary: Stage Level Sched: Add base ResourceProfile and Request classes Key: SPARK-29415 URL: https://issues.apache.org/jira/browse/SPARK-29415 Project: Spark

[jira] [Commented] (SPARK-29329) maven incremental builds not working

[ https://issues.apache.org/jira/browse/SPARK-29329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943839#comment-16943839 ] Thomas Graves commented on SPARK-29329: --- filed issue with scala-maven-plugin, we will see what

[jira] [Commented] (SPARK-29329) maven incremental builds not working

[ https://issues.apache.org/jira/browse/SPARK-29329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942790#comment-16942790 ] Thomas Graves commented on SPARK-29329: --- there are few comments on SPARK-28759 in regards to this,

[jira] [Updated] (SPARK-29329) maven incremental builds not working

[ https://issues.apache.org/jira/browse/SPARK-29329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-29329: -- Description: It looks like since we Upgraded scala-maven-plugin to 4.2.0

[jira] [Commented] (SPARK-28759) Upgrade scala-maven-plugin to 4.2.0

[ https://issues.apache.org/jira/browse/SPARK-28759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942787#comment-16942787 ] Thomas Graves commented on SPARK-28759: --- I rolled back this commit and the incremental compile now

[jira] [Created] (SPARK-29329) maven incremental builds not working

Thomas Graves created SPARK-29329: - Summary: maven incremental builds not working Key: SPARK-29329 URL: https://issues.apache.org/jira/browse/SPARK-29329 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-28759) Upgrade scala-maven-plugin to 4.2.0

[ https://issues.apache.org/jira/browse/SPARK-28759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942308#comment-16942308 ] Thomas Graves edited comment on SPARK-28759 at 10/1/19 9:05 PM:

[jira] [Commented] (SPARK-28759) Upgrade scala-maven-plugin to 4.2.0

[ https://issues.apache.org/jira/browse/SPARK-28759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942308#comment-16942308 ] Thomas Graves commented on SPARK-28759: --- [~dongjoon] [~hyukjin.kwon]  Did either of you noticed

[jira] [Created] (SPARK-29306) Executors need to track what ResourceProfile they are created with

Thomas Graves created SPARK-29306: - Summary: Executors need to track what ResourceProfile they are created with Key: SPARK-29306 URL: https://issues.apache.org/jira/browse/SPARK-29306 Project: Spark

[jira] [Resolved] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27396. --- Assignee: Robert Joseph Evans Resolution: Fixed > SPIP: Public APIs for extended

[jira] [Commented] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940974#comment-16940974 ] Thomas Graves commented on SPARK-27396: --- The main objectives have actually already been

[jira] [Created] (SPARK-29303) UI updates for stage level scheduling

Thomas Graves created SPARK-29303: - Summary: UI updates for stage level scheduling Key: SPARK-29303 URL: https://issues.apache.org/jira/browse/SPARK-29303 Project: Spark Issue Type: Story

[jira] [Commented] (SPARK-29206) Number of shuffle Netty server threads should be a multiple of number of chunk fetch handler threads

[ https://issues.apache.org/jira/browse/SPARK-29206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935968#comment-16935968 ] Thomas Graves commented on SPARK-29206: --- that is definitely not the behavior I was expecting from

[jira] [Updated] (SPARK-29151) Support fraction resources for task resource scheduling

[ https://issues.apache.org/jira/browse/SPARK-29151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-29151: -- Summary: Support fraction resources for task resource scheduling (was: Support fraction

[jira] [Updated] (SPARK-29151) Support fraction resources for task resource scheduling

[ https://issues.apache.org/jira/browse/SPARK-29151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-29151: -- Description: The current resource scheduling code for GPU/FPGA, etc only supports amounts as

[jira] [Commented] (SPARK-29151) Support fraction resources for resource scheduling

[ https://issues.apache.org/jira/browse/SPARK-29151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934419#comment-16934419 ] Thomas Graves commented on SPARK-29151: --- To keep this simple for a design, I think we change the

[jira] [Updated] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27495: -- Epic Name: Stage Level Scheduling > SPIP: Support Stage level resource configuration and

[jira] [Updated] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27495: -- Labels: SPIP (was: ) > SPIP: Support Stage level resource configuration and scheduling >

[jira] [Created] (SPARK-29154) Update Spark scheduler for stage level scheduling

Thomas Graves created SPARK-29154: - Summary: Update Spark scheduler for stage level scheduling Key: SPARK-29154 URL: https://issues.apache.org/jira/browse/SPARK-29154 Project: Spark Issue

[jira] [Updated] (SPARK-29150) Update RDD API for Stage level scheduling

[ https://issues.apache.org/jira/browse/SPARK-29150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-29150: -- Description: See the SPIP and design doc attached to SPARK-27495.  Note this is meant to be

[jira] [Created] (SPARK-29153) ResourceProfile conflict resolution stage level scheduling

Thomas Graves created SPARK-29153: - Summary: ResourceProfile conflict resolution stage level scheduling Key: SPARK-29153 URL: https://issues.apache.org/jira/browse/SPARK-29153 Project: Spark

[jira] [Commented] (SPARK-29151) Support fraction resources for resource scheduling

[ https://issues.apache.org/jira/browse/SPARK-29151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932422#comment-16932422 ] Thomas Graves commented on SPARK-29151: --- Depending on when things go in we could do this as part

[jira] [Created] (SPARK-29151) Support fraction resources for resource scheduling

Thomas Graves created SPARK-29151: - Summary: Support fraction resources for resource scheduling Key: SPARK-29151 URL: https://issues.apache.org/jira/browse/SPARK-29151 Project: Spark Issue

[jira] [Updated] (SPARK-29149) Update YARN cluster manager For Stage Level Scheduling

[ https://issues.apache.org/jira/browse/SPARK-29149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-29149: -- Issue Type: Story (was: Bug) > Update YARN cluster manager For Stage Level Scheduling >

[jira] [Updated] (SPARK-29150) Update RDD API for Stage level scheduling

[ https://issues.apache.org/jira/browse/SPARK-29150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-29150: -- Issue Type: Story (was: Improvement) > Update RDD API for Stage level scheduling >

[jira] [Updated] (SPARK-29148) Modify dynamic allocation manager for stage level scheduling

[ https://issues.apache.org/jira/browse/SPARK-29148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-29148: -- Issue Type: Story (was: Improvement) > Modify dynamic allocation manager for stage level

[jira] [Updated] (SPARK-29148) Modify dynamic allocation manager for stage level scheduling

[ https://issues.apache.org/jira/browse/SPARK-29148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-29148: -- Issue Type: Improvement (was: Bug) > Modify dynamic allocation manager for stage level

[jira] [Created] (SPARK-29150) Update RDD API for Stage level scheduling

Thomas Graves created SPARK-29150: - Summary: Update RDD API for Stage level scheduling Key: SPARK-29150 URL: https://issues.apache.org/jira/browse/SPARK-29150 Project: Spark Issue Type:

[jira] [Created] (SPARK-29149) Update YARN cluster manager For Stage Level Scheduling

Thomas Graves created SPARK-29149: - Summary: Update YARN cluster manager For Stage Level Scheduling Key: SPARK-29149 URL: https://issues.apache.org/jira/browse/SPARK-29149 Project: Spark

[jira] [Created] (SPARK-29148) Modify dynamic allocation manager for stage level scheduling

Thomas Graves created SPARK-29148: - Summary: Modify dynamic allocation manager for stage level scheduling Key: SPARK-29148 URL: https://issues.apache.org/jira/browse/SPARK-29148 Project: Spark

[jira] [Commented] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930600#comment-16930600 ] Thomas Graves commented on SPARK-27495: --- The SPIP vote passed - 

<    1   2   3   4   5   6   7   8   9   10   >