Re: planning & discussion for larger scheduler changes

filed [SPARK-20178] Improve Scheduler fetch failures - ASF JIRA | | | | || | | | | | [SPARK-20178] Improve Scheduler fetch failures - ASF JIRA | | | | Tom On Thursday, March 30, 2017 1:21 PM, Tom Graves wrote: If we are

Re: planning & discussion for larger scheduler changes

If we are worried about major changes destabilizing current code (which I can understand) only way around that is to make it pluggable or configurable.  For major changes it seems like making it pluggable is cleaner from a code being cluttered point of view. But it also means you may have to

Re: planning & discussion for larger scheduler changes

Thanks for the responses all. I may have worded my original email poorly -- I don't want to focus too much on SPARK-14649 and SPARK-13669 in particular, but more on how we should be approaching these changes. On Mon, Mar 27, 2017 at 9:01 PM, Kay Ousterhout wrote: > (1)

Re: planning & discussion for larger scheduler changes

(1) I'm pretty hesitant to merge these larger changes, even if they're feature flagged, because: (a) For some of these changes, it's not obvious that they'll always improve performance. e.g., for SPARK-14649, it's possible that the tasks that got re-started (and temporarily are running in two

Re: planning & discussion for larger scheduler changes

1) I think this depends on individual case by case jira.  I haven't looked in detail at spark-14649 seems much larger although more the way I think we want to go. While SPARK-13669 seems less risky and easily configurable. 2) I don't know whether it needs an entire rewrite but I think there need

Re: planning & discussion for larger scheduler changes

On Fri, Mar 24, 2017 at 4:41 PM, Imran Rashid wrote: > Kay and I were discussing some of the bigger scheduler changes getting > proposed lately, and realized there is a broader discussion to have with > the community, outside of any single jira. I'll start by sharing my >