[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14128357#comment-14128357 ] Hudson commented on YARN-1471: -- FAILURE: Integrated in Hadoop-Yarn-trunk #676 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/676/]) Add missing YARN-1471 to the CHANGES.txt (aw: rev 9b8104575444ed2de9b44fe902f86f7395f249ed) * hadoop-yarn-project/CHANGES.txt > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Minor > Fix For: 3.0.0 > > Attachments: SLSCapacityScheduler.java, YARN-1471.2.patch, > YARN-1471.patch, YARN-1471.patch > > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14128494#comment-14128494 ] Hudson commented on YARN-1471: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1867 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1867/]) Add missing YARN-1471 to the CHANGES.txt (aw: rev 9b8104575444ed2de9b44fe902f86f7395f249ed) * hadoop-yarn-project/CHANGES.txt > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Minor > Fix For: 3.0.0 > > Attachments: SLSCapacityScheduler.java, YARN-1471.2.patch, > YARN-1471.patch, YARN-1471.patch > > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14128915#comment-14128915 ] Hudson commented on YARN-1471: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1892 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1892/]) Add missing YARN-1471 to the CHANGES.txt (aw: rev 9b8104575444ed2de9b44fe902f86f7395f249ed) * hadoop-yarn-project/CHANGES.txt > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Minor > Fix For: 3.0.0 > > Attachments: SLSCapacityScheduler.java, YARN-1471.2.patch, > YARN-1471.patch, YARN-1471.patch > > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16037488#comment-16037488 ] Ye Zhou commented on YARN-1471: --- [~shv]Patches for branch 2.7, branch 2.8 and branch 2 build and pass unit tests locally. > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Minor > Labels: release-blocker > Fix For: 3.0.0-alpha1 > > Attachments: SLSCapacityScheduler.java, YARN-1471.2.patch, > YARN-1471-branch-2.7.4.patch, YARN-1471-branch-2.8.patch, > YARN-1471-branch-2.patch, YARN-1471.patch, YARN-1471.patch > > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16016887#comment-16016887 ] Ye Zhou commented on YARN-1471: --- [~shv] Attached patch for 2.7 branch. Build and Tests passed locally. Trigger Jenkins. > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Minor > Labels: release-blocker > Fix For: 3.0.0-alpha1 > > Attachments: SLSCapacityScheduler.java, YARN-1471.2.patch, > YARN-1471-branch-2.7.4.patch, YARN-1471.patch, YARN-1471.patch > > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16018960#comment-16018960 ] Konstantin Shvachko commented on YARN-1471: --- Hey [~zhouyejoe] or [~chris.douglas] do you mind updating path for branch-2 as well. I tried to merge but it didn't work. Cannot commit to 2.7 without committing to 2. > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Minor > Labels: release-blocker > Fix For: 3.0.0-alpha1 > > Attachments: SLSCapacityScheduler.java, YARN-1471.2.patch, > YARN-1471-branch-2.7.4.patch, YARN-1471.patch, YARN-1471.patch > > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023876#comment-16023876 ] Chris Douglas commented on YARN-1471: - [~curino] is this contained in YARN-6608? If so, maybe we should look into backporting that, instead of individual SLS patches. > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Minor > Labels: release-blocker > Fix For: 3.0.0-alpha1 > > Attachments: SLSCapacityScheduler.java, YARN-1471.2.patch, > YARN-1471-branch-2.7.4.patch, YARN-1471.patch, YARN-1471.patch > > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16031475#comment-16031475 ] Carlo Curino commented on YARN-1471: I believe it should be. YARN-6608 will need some more massaging, but it shouldn't be too much work. > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Minor > Labels: release-blocker > Fix For: 3.0.0-alpha1 > > Attachments: SLSCapacityScheduler.java, YARN-1471.2.patch, > YARN-1471-branch-2.7.4.patch, YARN-1471.patch, YARN-1471.patch > > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16032448#comment-16032448 ] Konstantin Shvachko commented on YARN-1471: --- Hey guys. Let's target full SLS feature for the next release. Would be too big change for a dot one. Here I just need a rebase of the branch-2 patch, which has been attached, but got stale, and which is just sufficient for our needs in 2.7. I am pretty confident about this one in particular as we were running it internally in production. Also since this is in trunk it may ease the backport effort of YARN-6608. > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Minor > Labels: release-blocker > Fix For: 3.0.0-alpha1 > > Attachments: SLSCapacityScheduler.java, YARN-1471.2.patch, > YARN-1471-branch-2.7.4.patch, YARN-1471.patch, YARN-1471.patch > > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035595#comment-16035595 ] Konstantin Shvachko commented on YARN-1471: --- I see a new patch in YARN-6608. So this is progressing. And if this change is incorporated there we just need to port into branch-2.7. Will do just that if there are no objections. > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Minor > Labels: release-blocker > Fix For: 3.0.0-alpha1 > > Attachments: SLSCapacityScheduler.java, YARN-1471.2.patch, > YARN-1471-branch-2.7.4.patch, YARN-1471.patch, YARN-1471.patch > > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13838488#comment-13838488 ] Carlo Curino commented on YARN-1471: As I raised during the discussion in https://issues.apache.org/jira/browse/YARN-1021?focusedCommentId=13745799&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13745799 the simulator is not currently compatible with the CapacityScheduler preemption mechanisms. More generally the "wrapping" done is likely to break interactions among components that are accessing not only the top-level interfaces, but leverage specifics of each scheduler. It would be nice to find some better wrapping mechanism. [~ywskycn], [~tucu00]... any idea? > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Wei Yan >Priority: Minor > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13838527#comment-13838527 ] Wei Yan commented on YARN-1471: --- Thanks, [~curino]. I'll take a look at ProportionalCapacityPreemptionPolicy firstly. > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Wei Yan >Priority: Minor > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13838579#comment-13838579
]
Alejandro Abdelnur commented on YARN-1471:
--
[~curino], I'm not familiar on how the ProportionalCapacityPreemptionPolicy
gets hold of data from the CapacityScheduler. The SLS simply wraps the
Scheduler implementation using a proxy pattern. Thus, the scheduler API is
fully exposed even if wrapped. If the ProportionalCapacityPreemptionPolicy cast
the Scheduler interface to the CapacityScheduler class, I would be inclined to
say that this is not a SLS issue but a ProportionalCapacityPreemptionPolicy
issue. Are the methods of the CapacityScheduler used by the
ProportionalCapacityPreemptionPolicy general purpose to qualify being in the
Scheduler API? If not, how about implementing a 'safety valve', something like
{{public T getComponent(Class klass)}} in the Scheduler API, and the
contract is to return NULL if such component is not implemented by the
scheduler. Would something like this work?
> The SLS simulator is not running the preemption policy for CapacityScheduler
>
>
> Key: YARN-1471
> URL: https://issues.apache.org/jira/browse/YARN-1471
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Carlo Curino
>Assignee: Wei Yan
>Priority: Minor
>
> The simulator does not run the ProportionalCapacityPreemptionPolicy monitor.
> This is because the policy needs to interact with a CapacityScheduler, and
> the wrapping done by the simulator breaks this.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13839483#comment-13839483 ] Carlo Curino commented on YARN-1471: The ProportionalCapacityPreemptionPolicy does indeed need to access methods that are specific to the CapacityScheduler. The three offending methods are: * scheduler.getResourceCalculator(); * scheduler.getRootQueue(); (which is a CSQueue) * scheduler.getClusterResources() However more generally this policy only makes sense when paired with the CapacityScheduler, so it needs somehow to check that the underlying scheduler is indeed a CapacityScheduler. This is coming up again in the work towards YARN-1051, as we have other "monitors" interacting with the underlying scheduler, possibly in ways that are specific to one specific scheduler. Speaking with [~chris.douglas] we seem to agree that the simulator would be much more useful if it allows to simulate specifics of each scheduler more than just exercise the general API (specifically I would not use it otherwise), and hacking the Scheduler API (and classes using it) to keep the abstraction in the simulator clean doesn't seem to be a solution. This seems a job for aspect programming... though it comes with its own set of headaches (see mumak). Maybe the simulator could also hijack the creation of monitors, and pass down the actual scheduler to those (assuming the rest of your metrics are collected in methods not used by the monitor). > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Wei Yan >Priority: Minor > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13839531#comment-13839531 ] Alejandro Abdelnur commented on YARN-1471: -- As the ProportionalCapacityPreemptionPolicy is CS specific, why does not live within the CS itself? Shouldn't be that way? > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Wei Yan >Priority: Minor > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13840745#comment-13840745 ] Carlo Curino commented on YARN-1471: We spoke with Alejandro, and a plausible solution goes as follows: The simulator instead of wrapping the scheduler sub-class it (and dynamically pick the right one, based on configs). The subclassing allows things like the ProportionalCapacityPreemptionPolicy (and future monitors) to work happily, and allows us to override only the methods relevant to the simulator. Wei, if you have nothing against it, I might try to give this a try. > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Wei Yan >Priority: Minor > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13841369#comment-13841369 ] Wei Yan commented on YARN-1471: --- Sure. Thanks, [~curino]. Just let me know if I can help. > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Minor > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13841541#comment-13841541 ] Carlo Curino commented on YARN-1471: Wei, I posted a patch that implements the change we discussed with Alejandro. Most of it is rather trivial, with one ugly exception. The subclassing of CapacityScheduler is basically a 1:1 copy of your ResourceSchedulerWrapper (changed to be a subclass instead of a wrapper). I don't have in mind a good alternative and I need to move foward with this somewhat quickly, if you have a better idea feel free to evolve our patch (or have a separate JIRA to prettify this). More importantly I tested it with both CapacityScheduler and FairScheduler and seems to work as intended, but I have limited understanding of all the metrics you collect and the functionalities of the simulator, as I just started using it, if you have a chance it would be great for you to test this more carefully. Thanks! > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Minor > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13841559#comment-13841559 ] Wei Yan commented on YARN-1471: --- Thanks, [~curino]. I'll test it locally. > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Minor > Attachments: YARN-1471.patch > > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13841766#comment-13841766 ] Carlo Curino commented on YARN-1471: Mmh... The problem is that you are creating a cycle passing "this" as the scheduler. E.g., calls to reinitialize for example will cycle between the two classes forever :-() I confirmed this integrating your code with the rest. > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Minor > Attachments: SLSCapacityScheduler.java, YARN-1471.patch > > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13841849#comment-13841849 ] Carlo Curino commented on YARN-1471: Alejandro, I tried the second version of what you proposed. It does not crash and seem to sort of work, but talking with Chris it has some very bad properties. If the inner CapacityScheduler (now or in the future) invokes from within one of its methods any of the methods you guys want to trap, this implementation would completely miss it. This is also true for the ResourceWrapper more generally I think, so probably the trick of subclassing instead of wrapping would be safer to propagate to the FairScheduler side as well. The maintenance cost here is that when a new scheduler is added, you have to create another one of this sub classes, but this seems rare enough of an event to be ok. > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Minor > Attachments: SLSCapacityScheduler.java, YARN-1471.patch, > YARN-1471.patch.2 > > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13851172#comment-13851172
]
Hadoop QA commented on YARN-1471:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12617451/SLSCapacityScheduler.java
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2686//console
This message is automatically generated.
> The SLS simulator is not running the preemption policy for CapacityScheduler
>
>
> Key: YARN-1471
> URL: https://issues.apache.org/jira/browse/YARN-1471
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Carlo Curino
>Assignee: Carlo Curino
>Priority: Minor
> Attachments: SLSCapacityScheduler.java, YARN-1471.patch
>
>
> The simulator does not run the ProportionalCapacityPreemptionPolicy monitor.
> This is because the policy needs to interact with a CapacityScheduler, and
> the wrapping done by the simulator breaks this.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13851214#comment-13851214
]
Hadoop QA commented on YARN-1471:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12619212/YARN-1471.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2687//console
This message is automatically generated.
> The SLS simulator is not running the preemption policy for CapacityScheduler
>
>
> Key: YARN-1471
> URL: https://issues.apache.org/jira/browse/YARN-1471
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Carlo Curino
>Assignee: Carlo Curino
>Priority: Minor
> Attachments: SLSCapacityScheduler.java, YARN-1471.patch,
> YARN-1471.patch
>
>
> The simulator does not run the ProportionalCapacityPreemptionPolicy monitor.
> This is because the policy needs to interact with a CapacityScheduler, and
> the wrapping done by the simulator breaks this.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13852298#comment-13852298
]
Hadoop QA commented on YARN-1471:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12619430/YARN-1471.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-sls.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/2694//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2694//console
This message is automatically generated.
> The SLS simulator is not running the preemption policy for CapacityScheduler
>
>
> Key: YARN-1471
> URL: https://issues.apache.org/jira/browse/YARN-1471
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Carlo Curino
>Assignee: Carlo Curino
>Priority: Minor
> Attachments: SLSCapacityScheduler.java, YARN-1471.2.patch,
> YARN-1471.patch, YARN-1471.patch
>
>
> The simulator does not run the ProportionalCapacityPreemptionPolicy monitor.
> This is because the policy needs to interact with a CapacityScheduler, and
> the wrapping done by the simulator breaks this.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13852373#comment-13852373 ] Chris Douglas commented on YARN-1471: - I committed this. Thanks Carlo > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Minor > Fix For: 3.0.0 > > Attachments: SLSCapacityScheduler.java, YARN-1471.2.patch, > YARN-1471.patch, YARN-1471.patch > > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13852374#comment-13852374 ] Hudson commented on YARN-1471: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4914 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4914/]) YARN-1471. Preserve scheduler typeinfo in simulator to work with resource monitors. Contributed by Carlo Curino. (cdouglas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1552173) * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerWrapper.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/web/SLSWebApp.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/capacity-scheduler.xml * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/yarn-site.xml > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Minor > Fix For: 3.0.0 > > Attachments: SLSCapacityScheduler.java, YARN-1471.2.patch, > YARN-1471.patch, YARN-1471.patch > > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853110#comment-13853110 ] Hudson commented on YARN-1471: -- FAILURE: Integrated in Hadoop-Yarn-trunk #426 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/426/]) YARN-1471. Preserve scheduler typeinfo in simulator to work with resource monitors. Contributed by Carlo Curino. (cdouglas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1552173) * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerWrapper.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/web/SLSWebApp.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/capacity-scheduler.xml * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/yarn-site.xml > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Minor > Fix For: 3.0.0 > > Attachments: SLSCapacityScheduler.java, YARN-1471.2.patch, > YARN-1471.patch, YARN-1471.patch > > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853205#comment-13853205 ] Hudson commented on YARN-1471: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1643 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1643/]) YARN-1471. Preserve scheduler typeinfo in simulator to work with resource monitors. Contributed by Carlo Curino. (cdouglas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1552173) * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerWrapper.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/web/SLSWebApp.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/capacity-scheduler.xml * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/yarn-site.xml > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Minor > Fix For: 3.0.0 > > Attachments: SLSCapacityScheduler.java, YARN-1471.2.patch, > YARN-1471.patch, YARN-1471.patch > > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853223#comment-13853223 ] Hudson commented on YARN-1471: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1617 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1617/]) YARN-1471. Preserve scheduler typeinfo in simulator to work with resource monitors. Contributed by Carlo Curino. (cdouglas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1552173) * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/appmaster/AMSimulator.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/ResourceSchedulerWrapper.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerWrapper.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/web/SLSWebApp.java * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/capacity-scheduler.xml * /hadoop/common/trunk/hadoop-tools/hadoop-sls/src/test/resources/yarn-site.xml > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Minor > Fix For: 3.0.0 > > Attachments: SLSCapacityScheduler.java, YARN-1471.2.patch, > YARN-1471.patch, YARN-1471.patch > > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (YARN-1471) The SLS simulator is not running the preemption policy for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853521#comment-13853521 ] Carlo Curino commented on YARN-1471: Thanks Chris! and thanks to Wei and Alejandro for useful conversations. > The SLS simulator is not running the preemption policy for CapacityScheduler > > > Key: YARN-1471 > URL: https://issues.apache.org/jira/browse/YARN-1471 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino >Priority: Minor > Fix For: 3.0.0 > > Attachments: SLSCapacityScheduler.java, YARN-1471.2.patch, > YARN-1471.patch, YARN-1471.patch > > > The simulator does not run the ProportionalCapacityPreemptionPolicy monitor. > This is because the policy needs to interact with a CapacityScheduler, and > the wrapping done by the simulator breaks this. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
